Do I need to create a dataset for every recipe?

The Dataiku design decision to persist every dataset at every step is one of the key unique selling points of the product. This allows less experienced users, aka clickers, to be able to see changes in the data pipeline making it much easier understand them. It also allows them to slowly build a very complicated data transformations in a visual step-by-step way along with assisting in debugging and enabling data quality checks at each step. Coders (the other user persona Dataiku tries to appeal to) may find this pattern wasteful of storage and too verbose. Having said that Dataiku does not force you to design the flow in a specific way. If you are a coder and prefer to bundle up multiple steps in a single recipe you are welcome to do so by using the available code recipes (Python, SQL, Shell, Spark, etc). However please note that your project will not easy to understand by other users that are not coders like you are (aka clickers: testers, data analysts, BAs, PMs, etc).

There is also a hybrid option called SQL pipelines. This option allows to execute a bunch of recipes as a single entity without having to persist the individual recipe outputs. There are several caveats to use this option, you have to have all your recipes run in SQL Engine mode which means that a lot of code recipes and some visual recipes will not be compatible with SQL Pipelines. You also have to have all the recipes use the same Dataiku connection. And finally the fact that you won't be persisting the intermediate outputs leads to harder to debug problems in your flow although you can temporarily turn SQL pipelines off in the datasets you need to check something. And of course SQL pipelines make the project harder to understand by clickers and prevents you from implementing data quality checks in your intermediate datasets as these are not persisted.

Ultimately this is an issue of having the cake and eat it. You can't have it both ways and you need to make a decision that works best for your use case. This again is one the Dataiku strengths. You can make this choice in a project by project basis and even mix and match these patterns on the same flow allowing you to use the solution that best fits the problem you are trying to solve.

One thing to note is that in these days data storage is very cheap and Data Engineers / Data Scientists are not. So in my view Dataiku's way of doing things ends up saving a lot more in human costs by allowing more users use their product than the extra storage costs it adds by the intermediate results being persisted. Finally whereas before there were some concerns around how much data you can store in a database these days there are plenty of database technologies like Snowflake, Databricks or Google BigQuery that can store and handle virtually unlimited amounts of data.