Speed up SQL calculations
Hello,
I'm currently trying to create a flow from datasets imported from external SQL database.
When running a simple join recipe on Dataiku using "SQL in base engine", the job takes 12 min.
When copying the generated SQL query and running it on my SQL manager tool (Dbeaver) it takes 1 second on the same database.
Where does this difference comes from ? It is the time Dataiku needs to write the joined table ? Is so, could I prevent it from writing it to only have a visual of the table ?
Answers
-
DSS only generates and issues SQL to the database. If your recipe output is a table of the database, then the 12 minutes are all 100% database work. The difference with interactive tools is usually that these tools stop when they gathered enough rows for the display, instead of processing all the data like a recipe does.
-
Well thx.
Then is there a way to work like these interactive tools in DSS ? Like try and see, then adjust without waiting that the whole database is proceded ?
-
you can run SQL commands directly against the database in the Notebooks section of DSS (the code icon in the menubar)
