How to automate a recipe
Hi,
I have a recipe that syncs a sql server table to an oracle table. The sync copies the original table(SQL server) and creates a copy of that table in Oracle. How do you go about automating this process so that it keeps the Oracle table updated each day?
Thank you in advance.
Answers
-
RoyE Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 31 DataikerOptionsHello,
You can automate this process using Scenarios!
Scenarios allow you to define different types of triggers, and supports time-based triggers. Therefore, you can set up an end of day or start of day dataset rebuild which will rebuild your Oracle table.
https://doc.dataiku.com/dss/latest/scenarios/index.html
In the Scenarios page, create a new step-based scenario.
Add a time-based trigger in the Settings tab:
Create a "Build/Train" step in the Steps tab.
This Scenario will build this dataset at the specific time-based trigger that you have listed.
Roy
-
Hi Roy,
Thank you for responding back with this information,
I set up a trigger to run this morning with a build step. The step was set up to build the Oracle table.
The scenario ran at the correct time, but the table on the Oracle server did not update. Is there another step I am missing?When I manually run the recipe with the sync from one server/table to another it works perfectly.
Thank you in advance.
-
RoyE Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 31 DataikerOptions
Hi Stephen,
Just to clarify a few things,
Are you using the Sync Visual Recipe?
Does the Scenario show "Successful" in the "Last Runs" tab?
Does your Oracle dataset (in DSS) show the updated information?
Roy
-
It looks like it is a Sync Visual recipe. I am copying a table/dataset from one location to another.
The scenario does show successful for the past few days now.
It is hard to tell if the Oracle data set in DSS is updated. It is a large data set and it only shows a sample.
Thank you in advance
-
RoyE Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 31 DataikerOptions
Hello,
One idea is to use metrics within the Status tab of a dataset. This will provide easy to digest information about your dataset such as, row count, size, etc. If you know that the row count will most likely change, it it possible to use this metric to confirm the dataset has been built properly.
This can either be set to "Run after build" or can be triggered from the Scenario that you built earlier.
Roy