Automating tables read and stacking from SQL database into Dataiku platform
kautuk
Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 5 Partner
Hello, I have a dataset in database which is updated every month. And I have imported the dataset into the DSS using SQL connection. Now, I am looking for the process which can read the new dataset (as soon as it is loaded in database) and stack it with the other dataset in Dataiku automatically. Is this possible in Dataiku?
Answers
-
Álvaro Andrés Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 7 DataikerHello Kautuk,
For this request you can use an scenario with the following parameters:
- Add on SQL query change trigger, query should check changes in your SQL dataset (e.g. number of rows, check max in date columns, etc.)
- Enable Auto-Triggers in Scenario-->Settings-->Auto-triggers-->ON
- Add a step in your scenario which build your Stack recipe output dataset, in build mode please select "Build only this dataset"
Best Regards,
Álvaro