Submit your innovative use case or inspiring success story to the 2023 Dataiku Frontrunner Awards! LET'S GO

Run a scenario when 3 other scenarios completed (in random order)

Kok
Level 2
Run a scenario when 3 other scenarios completed (in random order)

Hi,

I am trying to run a scenario in dataiku automation when 3 other scenarios have completed. But they can be completed in any order. Is there a way to accomplish this?

Thanks!

0 Kudos
8 Replies
JordanB
Dataiker

Hi @Kok,

Have you tried adding multiple triggers? For example, when any of the following scenarios runs successfully, this scenario will trigger:

Screen Shot 2023-04-03 at 6.23.54 PM.png

Thanks,

Jordan

0 Kudos
Kok
Level 2
Author

Hi Jordan,

 

Thank you for your reply. 

In my case, I need to run the scenario when all 3 other scenarios have completed successfully.

For example, we have scenarios A, B, C, D, I need to have D triggered only when A, B, and C all completed, then D can be triggered. But scenario A, B, and C can finish running in any order. The problem is I don't know which one will run last. Basically, scenario A, B, and C are running in parallel.

Any suggestion would be much appreciated.

 

0 Kudos
tgb417

Sounds like you want to run A, B, and C in parallel not sequentially. If you could ran them sequentially you could have a scenario that starts other Senarios and checks status.  The Scenario would started Scenario A,  check if it is successful and then Run Scenario B, and then Run C… 

Because you want to run them in parallel, knowing that you have a complete set that is internally consistent for all A, B, and C can be a bit challenging.

One idea I have is to set project level variables at the end of each run of A, B, and C.  And then in D look for those statuses by checking those variables.  

This can be a bit challenging.

I’ve submitted an enhancement request on the subject.

https://community.dataiku.com/t5/Product-Ideas/Improved-UX-for-Senario-Variables-Setup/idi-p/30707

If you can tell by looking a the condition of data in the last dataset of each of the flows that the data has been written, you might be able to create a custom data set check and use this as your indication at the beginning of Scenario D, that the other Senarios have completed successfully.  

You could do other things if the data is in SQL accessible databases writing a SQL query that return true when everything is done.  And then put a check at the beginning of D looking for this status to be true.   

--Tom
0 Kudos
Kok
Level 2
Author

Hi Tom,

Thank you so much for your reply. I already tried what you suggested with project variable, but I guess it's because of concurrency issue, the timing for read and write of the status are not in sync and it didn't work.

I also tried with using a status flag in database and it's the same result.

 

 

0 Kudos
tgb417

@Kok 
Regarding Project variables are you sure that you actually go it set.  I found this a challenge the first time I tried.

Scenario step screen showing the setting Project level variable.Scenario step screen showing the setting Project level variable.

 

Here are some further discussions on setting up project variables from a Senario.
https://community.dataiku.com/t5/Using-Dataiku/Retrieving-the-value-of-a-metric-in-a-scenario-or-as-... 

Triggering a Scenario when Project Variable Changes.
https://community.dataiku.com/t5/Using-Dataiku/Trigger-a-scenario-when-project-variables-change/m-p/... 

--Tom
0 Kudos
Kok
Level 2
Author

Hi Tom,

I set the value of the project variable with the following code in python receipe:

#Check and set scenarios completion
project_handle = dataiku.api_client().get_project(dataiku.default_project_key())
vars = project_handle.get_variables()
vars['standard']['myVariable'] = "OK"
project_handle.set_variables(vars)

0 Kudos
Kok
Level 2
Author

Hi Tom,

I tried asking ChatGPT for that question, and it suggested to use a "Parent Scenario" with a "Wait for scenario" plugin. But I can't find the "Wait for Scenario" plugin. Is there such a thing? Probably ChatGPT got the wrong information.

Thanks!

0 Kudos
tgb417

@Kok ,

As far as I know ChatGPT is hallucinating that plug-in.

--Tom