How can I run two steps in parallel in a scenario
Hi,
I am trying to build a scenario in which I want two data sets to be built.
The datasets are built using sql code recipe already written.
I am wondering if there is any way in which these two datasets can be built parallelly instead of sequentially. Please guide
Also, I can see that there are options to define when does a step run (PFB attached screenshot).
will 'Run this step' to 'Always' achieve the same outcome ?
Thank you in advance
Operating system used: Windows
Operating system used: Windows
Best Answer
-
Miguel Angel Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 118 DataikerHi,
You can add more than one dataset to be built in the same Scenario step. However, this does not mean their build will be exactly in parallel. It will depend on the number of activities the job has to carry, e.g. due to a different amount of upstream datasets that each need to have built.
The run steps condition the behaviour of the Scenario steps. The ‘Always’ condition ensures that a step will be run (or attempted to be run) regardless of the outcome of preceding steps.
More information can be found in the documentation: https://doc.dataiku.com/dss/latest/scenarios/step_flow_control.html#run-step-conditionally