Run a scenario when 3 other scenarios completed (in random order)

Options
Kok
Kok Registered Posts: 12

Hi,

I am trying to run a scenario in dataiku automation when 3 other scenarios have completed. But they can be completed in any order. Is there a way to accomplish this?

Thanks!

Answers

  • JordanB
    JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 293 Dataiker
    Options

    Hi @Kok
    ,

    Have you tried adding multiple triggers? For example, when any of the following scenarios runs successfully, this scenario will trigger:

    Screen Shot 2023-04-03 at 6.23.54 PM.png

    Thanks,

    Jordan

  • Kok
    Kok Registered Posts: 12
    Options

    Hi Jordan,

    Thank you for your reply.

    In my case, I need to run the scenario when all 3 other scenarios have completed successfully.

    For example, we have scenarios A, B, C, D, I need to have D triggered only when A, B, and C all completed, then D can be triggered. But scenario A, B, and C can finish running in any order. The problem is I don't know which one will run last. Basically, scenario A, B, and C are running in parallel.

    Any suggestion would be much appreciated.

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    Sounds like you want to run A, B, and C in parallel not sequentially. If you could ran them sequentially you could have a scenario that starts other Senarios and checks status. The Scenario would started Scenario A, check if it is successful and then Run Scenario B, and then Run C…

    Because you want to run them in parallel, knowing that you have a complete set that is internally consistent for all A, B, and C can be a bit challenging.

    One idea I have is to set project level variables at the end of each run of A, B, and C. And then in D look for those statuses by checking those variables.

    This can be a bit challenging.

    I’ve submitted an enhancement request on the subject.

    https://community.dataiku.com/t5/Product-Ideas/Improved-UX-for-Senario-Variables-Setup/idi-p/30707

    If you can tell by looking a the condition of data in the last dataset of each of the flows that the data has been written, you might be able to create a custom data set check and use this as your indication at the beginning of Scenario D, that the other Senarios have completed successfully.

    You could do other things if the data is in SQL accessible databases writing a SQL query that return true when everything is done. And then put a check at the beginning of D looking for this status to be true.

  • Kok
    Kok Registered Posts: 12
    Options

    Hi Tom,

    Thank you so much for your reply. I already tried what you suggested with project variable, but I guess it's because of concurrency issue, the timing for read and write of the status are not in sync and it didn't work.

    I also tried with using a status flag in database and it's the same result.

  • Kok
    Kok Registered Posts: 12
    Options

    Hi Tom,

    I tried asking ChatGPT for that question, and it suggested to use a "Parent Scenario" with a "Wait for scenario" plugin. But I can't find the "Wait for Scenario" plugin. Is there such a thing? Probably ChatGPT got the wrong information.

    Thanks!

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    @Kok
    ,

    As far as I know ChatGPT is hallucinating that plug-in.

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    @Kok

    Regarding Project variables are you sure that you actually go it set. I found this a challenge the first time I tried.

    Scenario step screen showing the setting Project level variable.

    Here are some further discussions on setting up project variables from a Senario.
    https://community.dataiku.com/t5/Using-Dataiku/Retrieving-the-value-of-a-metric-in-a-scenario-or-as-project/m-p/30706/highlight/true

    Triggering a Scenario when Project Variable Changes.
    https://community.dataiku.com/t5/Using-Dataiku/Trigger-a-scenario-when-project-variables-change/m-p/28828

  • Kok
    Kok Registered Posts: 12
    Options

    Hi Tom,

    I set the value of the project variable with the following code in python receipe:

    #Check and set scenarios completion
    project_handle = dataiku.api_client().get_project(dataiku.default_project_key())
    vars = project_handle.get_variables()
    vars['standard']['myVariable'] = "OK"
    project_handle.set_variables(vars)

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,726 Neuron
    Options

    Hahaha, just seen this post. Classic! ChatGPT making things up. Don't believe everything it says, sometimes it makes things up!!!

    https://arstechnica.com/tech-policy/2023/06/lawyers-have-real-bad-day-in-court-after-citing-fake-cases-made-up-by-chatgpt/

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,726 Neuron
    Options

    @Kok
    Have you solved this problem? I am thinking in writing some Python code to do this but no point in reinventing the wheel if you already have a solution for this.

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,726 Neuron
    Options

    @Kok
    By the way did you solve this problem if so, how? I am thinking in writing some Python code to do this as I have seen several questions on this subject but I rather not reinvent the wheel if a solution already exists.

Setup Info
    Tags
      Help me…