Automating tables read and stacking from SQL database into Dataiku platform

kautuk
Level 2
Automating tables read and stacking from SQL database into Dataiku platform

Hello, I have a dataset in database which is updated every month. And I have imported the dataset into the DSS using SQL connection. Now, I am looking for the process which can read the new dataset (as soon as it is loaded in database) and stack it with the other dataset in Dataiku automatically. Is this possible in Dataiku? 

0 Kudos
1 Reply
anino
Dataiker

Hello Kautuk,

For this request you can use an scenario with the following parameters:

  • Add on SQL query change trigger, query should check changes in your SQL dataset (e.g. number of rows, check max in date columns, etc.)
  • Enable Auto-Triggers in Scenario-->Settings-->Auto-triggers-->ON
  • Add a step in your scenario which build your Stack recipe output dataset, in build mode please select "Build only this dataset"

Best Regards,

รlvaro

Technical Support Engineer