How to automate a recipe

Stephen_3
Stephen_3 Registered Posts: 4 ✭✭✭

Hi,

I have a recipe that syncs a sql server table to an oracle table. The sync copies the original table(SQL server) and creates a copy of that table in Oracle. How do you go about automating this process so that it keeps the Oracle table updated each day?

Thank you in advance.

Answers

  • RoyE
    RoyE Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 31 Dataiker

    Hello,

    You can automate this process using Scenarios!

    Scenarios allow you to define different types of triggers, and supports time-based triggers. Therefore, you can set up an end of day or start of day dataset rebuild which will rebuild your Oracle table.

    https://doc.dataiku.com/dss/latest/scenarios/index.html

    In the Scenarios page, create a new step-based scenario.

    Add a time-based trigger in the Settings tab:

    Screen Shot 2021-07-14 at 10.09.27.png

    Create a "Build/Train" step in the Steps tab.

    Screen Shot 2021-07-14 at 10.08.52.png

    This Scenario will build this dataset at the specific time-based trigger that you have listed.

    Roy

  • Stephen_3
    Stephen_3 Registered Posts: 4 ✭✭✭

    Hi Roy,

    Thank you for responding back with this information,

    I set up a trigger to run this morning with a build step. The step was set up to build the Oracle table.
    The scenario ran at the correct time, but the table on the Oracle server did not update. Is there another step I am missing?

    When I manually run the recipe with the sync from one server/table to another it works perfectly.

    Thank you in advance.

  • RoyE
    RoyE Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 31 Dataiker

    Hi Stephen,

    Just to clarify a few things,

    Are you using the Sync Visual Recipe?

    Does the Scenario show "Successful" in the "Last Runs" tab?

    Does your Oracle dataset (in DSS) show the updated information?

    Roy

  • Stephen_3
    Stephen_3 Registered Posts: 4 ✭✭✭

    It looks like it is a Sync Visual recipe. I am copying a table/dataset from one location to another.

    The scenario does show successful for the past few days now.

    It is hard to tell if the Oracle data set in DSS is updated. It is a large data set and it only shows a sample.

    Thank you in advance

  • RoyE
    RoyE Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 31 Dataiker

    Hello,

    One idea is to use metrics within the Status tab of a dataset. This will provide easy to digest information about your dataset such as, row count, size, etc. If you know that the row count will most likely change, it it possible to use this metric to confirm the dataset has been built properly.

    This can either be set to "Run after build" or can be triggered from the Scenario that you built earlier.

    Roy

Setup Info
    Tags
      Help me…