Best Practice for managing tables

@Alexandru suggestions are sound options but I feel you may be going against the flow here (pun intended!). Dataiku's strongest USP is that it can be used by "clickers" and "coders", meaning people that know Data Engineering / Machine Learning (aka Data Scientists) and those that don't. Having the flow persist every intermediate dataset supports that vision. It's easy to see the transformations visually when you can see what the output of each step is and breaking complex data pipelines in steps makes it much easier to understand. Visual recipes also provide a zero code approach that anyone can easily understand. If you use SQL Pipelines you are breaking that capability for little to no gain. The other thing you get by persisting the intermediate results is traveability and debug capability. Try to figure out the issue with a complex process with just the inputs and outputs is way more complex than debugging it step by step. So again you will remove this capability if don't have intermediate results.

At the end of the day storage is very cheap these days and virtually unlimited and available on demand, assuming you are on cloud (which you should). In most cases your bigger costs are people since the people that work on Dataiku projects will tend to be high value resources (Data Scientists, SMEs, BAs, Experts, PMs, Business Users). So making things more complex to understand and debug by saving some money on storage or because you think it looks messy doesn't seem like a reasonable trade off.

One thing you could do is to write output datasets to a separate Snowflake schema. That way you can separate intermediate tables from output tables and ignore the "noise". This could be done using a variable in the write schema in the connection which you can then customise in the recipe variables section. But you will have to maintain this manually so it's not silver bullet solution.