How can I run two steps in parallel in a scenario

Solved!
yjagger
Level 2
How can I run two steps in parallel in a scenario

Hi,

I am trying to build a scenario in which I want two data sets to be built.

The datasets are built using sql code recipe already written.

I am wondering if there is any way in which these two datasets can be built parallelly instead of sequentially. Please guide

Also, I can see that there are options to define when does a step run (PFB attached screenshot).

will 'Run this step' to 'Always' achieve the same outcome ?

Thank you in advance


Operating system used: Windows


Operating system used: Windows

0 Kudos
1 Solution
MiguelangelC
Dataiker

Hi,

You can add more than one dataset to be built in the same Scenario step. However, this does not mean their build will be exactly in parallel. It will depend on the number of activities the job has to carry, e.g. due to a different amount of upstream datasets that each need to have built.

The run steps condition the behaviour of the Scenario steps. The โ€˜Alwaysโ€™ condition ensures that a step will be run (or attempted to be run) regardless of the outcome of preceding steps.

More information can be found in the documentation: https://doc.dataiku.com/dss/latest/scenarios/step_flow_control.html#run-step-conditionally

View solution in original post

0 Kudos
1 Reply
MiguelangelC
Dataiker

Hi,

You can add more than one dataset to be built in the same Scenario step. However, this does not mean their build will be exactly in parallel. It will depend on the number of activities the job has to carry, e.g. due to a different amount of upstream datasets that each need to have built.

The run steps condition the behaviour of the Scenario steps. The โ€˜Alwaysโ€™ condition ensures that a step will be run (or attempted to be run) regardless of the outcome of preceding steps.

More information can be found in the documentation: https://doc.dataiku.com/dss/latest/scenarios/step_flow_control.html#run-step-conditionally

0 Kudos