Speed up SQL calculations

vmar
vmar Dataiku DSS Core Designer, Registered Posts: 9 ✭✭✭

Hello,

I'm currently trying to create a flow from datasets imported from external SQL database.

When running a simple join recipe on Dataiku using "SQL in base engine", the job takes 12 min.

When copying the generated SQL query and running it on my SQL manager tool (Dbeaver) it takes 1 second on the same database.

Where does this difference comes from ? It is the time Dataiku needs to write the joined table ? Is so, could I prevent it from writing it to only have a visual of the table ?

Answers

  • fchataigner2
    fchataigner2 Dataiker Posts: 355 Dataiker

    DSS only generates and issues SQL to the database. If your recipe output is a table of the database, then the 12 minutes are all 100% database work. The difference with interactive tools is usually that these tools stop when they gathered enough rows for the display, instead of processing all the data like a recipe does.

  • vmar
    vmar Dataiku DSS Core Designer, Registered Posts: 9 ✭✭✭

    Well thx.

    Then is there a way to work like these interactive tools in DSS ? Like try and see, then adjust without waiting that the whole database is proceded ?

  • fchataigner2
    fchataigner2 Dataiker Posts: 355 Dataiker

    you can run SQL commands directly against the database in the Notebooks section of DSS (the code icon in the menubar)

Setup Info
    Tags
      Help me…