I'm currently trying to create a flow from datasets imported from external SQL database.
When running a simple join recipe on Dataiku using "SQL in base engine", the job takes 12 min.
When copying the generated SQL query and running it on my SQL manager tool (Dbeaver) it takes 1 second on the same database.
Where does this difference comes from ? It is the time Dataiku needs to write the joined table ? Is so, could I prevent it from writing it to only have a visual of the table ?
DSS only generates and issues SQL to the database. If your recipe output is a table of the database, then the 12 minutes are all 100% database work. The difference with interactive tools is usually that these tools stop when they gathered enough rows for the display, instead of processing all the data like a recipe does.