How can I run two steps in parallel in a scenario

Options
yjagger
yjagger Registered Posts: 8

Hi,

I am trying to build a scenario in which I want two data sets to be built.

The datasets are built using sql code recipe already written.

I am wondering if there is any way in which these two datasets can be built parallelly instead of sequentially. Please guide

Also, I can see that there are options to define when does a step run (PFB attached screenshot).

will 'Run this step' to 'Always' achieve the same outcome ?

Thank you in advance


Operating system used: Windows


Operating system used: Windows

Tagged:

Best Answer

  • Miguel Angel
    Miguel Angel Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 118 Dataiker
    Answer ✓
    Options

    Hi,

    You can add more than one dataset to be built in the same Scenario step. However, this does not mean their build will be exactly in parallel. It will depend on the number of activities the job has to carry, e.g. due to a different amount of upstream datasets that each need to have built.

    The run steps condition the behaviour of the Scenario steps. The ‘Always’ condition ensures that a step will be run (or attempted to be run) regardless of the outcome of preceding steps.

    More information can be found in the documentation: https://doc.dataiku.com/dss/latest/scenarios/step_flow_control.html#run-step-conditionally

Setup Info
    Tags
      Help me…