How to automate a recipe

Stephen_3

Hi,

I have a recipe that syncs a sql server table to an oracle table. The sync copies the original table(SQL server) and creates a copy of that table in Oracle. How do you go about automating this process so that it keeps the Oracle table updated each day?

Thank you in advance.

RoyE

Hello,

You can automate this process using Scenarios!

Scenarios allow you to define different types of triggers, and supports time-based triggers. Therefore, you can set up an end of day or start of day dataset rebuild which will rebuild your Oracle table.

https://doc.dataiku.com/dss/latest/scenarios/index.html

In the Scenarios page, create a new step-based scenario.

Add a time-based trigger in the Settings tab:

Create a "Build/Train" step in the Steps tab.

This Scenario will build this dataset at the specific time-based trigger that you have listed.

Roy

Stephen_3

Hi Roy,

Thank you for responding back with this information,

I set up a trigger to run this morning with a build step. The step was set up to build the Oracle table.
The scenario ran at the correct time, but the table on the Oracle server did not update. Is there another step I am missing?

When I manually run the recipe with the sync from one server/table to another it works perfectly.

Thank you in advance.

RoyE

Hi Stephen,

Just to clarify a few things,

Are you using the Sync Visual Recipe?

Does the Scenario show "Successful" in the "Last Runs" tab?

Does your Oracle dataset (in DSS) show the updated information?

Roy

Stephen_3

It looks like it is a Sync Visual recipe. I am copying a table/dataset from one location to another.

The scenario does show successful for the past few days now.

It is hard to tell if the Oracle data set in DSS is updated. It is a large data set and it only shows a sample.

Thank you in advance

RoyE

Hello,

One idea is to use metrics within the Status tab of a dataset. This will provide easy to digest information about your dataset such as, row count, size, etc. If you know that the row count will most likely change, it it possible to use this metric to confirm the dataset has been built properly.

This can either be set to "Run after build" or can be triggered from the Scenario that you built earlier.

Roy