Automating tables read and stacking from SQL database into Dataiku platform

kautuk
kautuk Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 5 Partner

Hello, I have a dataset in database which is updated every month. And I have imported the dataset into the DSS using SQL connection. Now, I am looking for the process which can read the new dataset (as soon as it is loaded in database) and stack it with the other dataset in Dataiku automatically. Is this possible in Dataiku?

Answers

  • Álvaro Andrés
    Álvaro Andrés Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 7 Dataiker

    Hello Kautuk,

    For this request you can use an scenario with the following parameters:

    • Add on SQL query change trigger, query should check changes in your SQL dataset (e.g. number of rows, check max in date columns, etc.)
    • Enable Auto-Triggers in Scenario-->Settings-->Auto-triggers-->ON
    • Add a step in your scenario which build your Stack recipe output dataset, in build mode please select "Build only this dataset"

    Best Regards,

    Álvaro

Setup Info
    Tags
      Help me…