If condition satisfied scenarios

pmunoz
pmunoz Registered Posts: 1 ✭✭✭✭

Hi!!

I am trying to add certain conditions in the executions of scenarios, but it seems that I am making some mistakes in the format or how I should do it correctly.

2020-10-19_16h42_40.png

If I may detail, the main idea is.

  1. Execution Step1.
  2. (parallel execution of step 2 and step3).
    1. Execute Step2 if Step1 finishes OK.
    2. Execution Step3 if Step1 finishes OK.
  3. Execute Step4 only if Step2 and Step3 finish OK.

These conditions should be placed in the execution on each step (2,3 and 4) like that

2020-10-19_16h53_44.png

So the questions are,

  1. What should be in the "if" section in order to recover the OK or Success value of a previous step?
  2. If the step 2 and step 3, are at the same time execution of other scenarios (from the same project), how should be written the "IF" condition as recovering scenarioX.outcome?

Thanks!!!

Answers

  • Ignacio_Toledo
    Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 415 Neuron

    Hi @pmunoz
    . The documentation on how to fill the 'if condition satisfied' (your "a" questions) can be found at:

    https://doc.dataiku.com/dss/latest/scenarios/step_flow_control.html#if-condition-satisfied

    As for "b", well, I wouldn't know. From the variables available in the previous documentation, apparently there are no variables associated to the outcome of another scenario, but maybe some dataiker can get back to you with that. Also, I'm no sure if it is possible to run two Steps in parallel.

    Cheers.

  • Liev
    Liev Dataiker Alumni Posts: 176 ✭✭✭✭✭✭✭✭

    Hi @pmunoz

    Another way of tackling this is by putting steps 2 and 3 in a separate scenario.

    In this case, in your first scenario your second step would be to run this secondary scenario, and upon successful completion, run step 4.

    Hope this helps!

  • tim-wright
    tim-wright Partner, L2 Designer, Snowflake Advanced, Neuron 2020, Registered, Neuron 2021, Neuron 2022 Posts: 77 Partner

    @pmunoz
    First off thanks for the good question. I'll admit that I also have not done this before so did a little testing on my own. I stumbled upon this link:

    https://doc.dataiku.com/dss/latest/scenarios/step_flow_control.html

    It appears that in the conditional you put:

    stepOutcome_<insert buildstep name (1)> == 'SUCCESS' (see my example below)

    where you replace the portion in between <> with the name of your step. It appeared to work for me in my brief testing. Note: It is not very clear how to handle whitespace in your step name, but I *think* you can replace it with an underscore character.

    b) I think you could create the same conditional for two separate steps to force them to be in "parallel". I suspect that you can add a final (4th) step to run if no other steps have failed OR to explicitly run when 2 and 3 were successful.

    conditional_dss_scenario.PNG

    Let us know how it turns out

  • sonal_18
    sonal_18 Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered, Frontrunner 2022 Participant Posts: 12 ✭✭✭✭

    Hi tim,

    This is very helpful, can we also get the error message once the condition is failed?

    Actually im creating scenarios for creating logs of the job build and in the table i want to insert the error message as well. Is there a way to do it?

  • elnurmdov
    elnurmdov Registered Posts: 5 ✭✭✭

    @tim-wright
    hello. I have a question regarding this case. I want my step to run if the previous custom python step returns or prints certain message. How can I do it? I did not get the difference between stepResult_stepName and stepOutput_stepName

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,024 Neuron
    edited July 17

    As the documentation says there are 3 three different kinds of outputs for each scenario step:

    stepOutcome: This is handled by Dataiku for you. Whatever the step does it will have a status, possible values are ‘SUCCESS’, ‘WARNING’, ‘FAILED’, ‘ABORTED’

    stepOutput: Some steps can produce output data. For instance you can have a step run a macro that lists datasets. The stepOutput will contain the output of the macro.

    stepResult: Is a JSON structure with basic information about the step and the outcome. See sample below:

    {"start":1693233602210,"end":1693233603166,"type":"STEP_DONE","outcome":"SUCCESS","target":{"projectKey":"CT_TEST","stepId":"runmacro_pyrunnable_builtin-macros_list-datasets-using-connection","type":"SCENARIO_STEP","scenarioId":"TESTSCENARIO"}}

    The easiest way to do your requirement is to have a Custom Python step define a scenario variable:

    from dataiku.scenario import Scenario
    Scenario().set_scenario_variables(VariableName="some_value")

    And then have a step that runs conditionally (if condition satisfied):

    VariableName == "some_value"

Setup Info
    Tags
      Help me…