Run a scenario when 3 other scenarios completed (in random order)

[Kok]

Hi,

I am trying to run a scenario in dataiku automation when 3 other scenarios have completed. But they can be completed in any order. Is there a way to accomplish this?

Thanks!

[JordanB]

Hi @Kok,

Have you tried adding multiple triggers? For example, when any of the following scenarios runs successfully, this scenario will trigger:

Thanks,

Jordan

[Kok]

Hi Jordan,

Thank you for your reply. 

In my case, I need to run the scenario when all 3 other scenarios have completed successfully.

For example, we have scenarios A, B, C, D, I need to have D triggered only when A, B, and C all completed, then D can be triggered. But scenario A, B, and C can finish running in any order. The problem is I don't know which one will run last. Basically, scenario A, B, and C are running in parallel.

Any suggestion would be much appreciated.

[tgb417]

Sounds like you want to run A, B, and C in parallel not sequentially. If you could ran them sequentially you could have a scenario that starts other Senarios and checks status.  The Scenario would started Scenario A,  check if it is successful and then Run Scenario B, and then Run C… 

Because you want to run them in parallel, knowing that you have a complete set that is internally consistent for all A, B, and C can be a bit challenging.

One idea I have is to set project level variables at the end of each run of A, B, and C.  And then in D look for those statuses by checking those variables.  

This can be a bit challenging.

I’ve submitted an enhancement request on the subject.

https://community.dataiku.com/t5/Product-Ideas/Improved-UX-for-Senario-Variables-Setup/idi-p/30707

If you can tell by looking a the condition of data in the last dataset of each of the flows that the data has been written, you might be able to create a custom data set check and use this as your indication at the beginning of Scenario D, that the other Senarios have completed successfully.  

You could do other things if the data is in SQL accessible databases writing a SQL query that return true when everything is done.  And then put a check at the beginning of D looking for this status to be true.   

[Kok]

Hi Tom,

Thank you so much for your reply. I already tried what you suggested with project variable, but I guess it's because of concurrency issue, the timing for read and write of the status are not in sync and it didn't work.

I also tried with using a status flag in database and it's the same result.

[tgb417]

@Kok 
Regarding Project variables are you sure that you actually go it set.  I found this a challenge the first time I tried.

Scenario step screen showing the setting Project level variable.

Here are some further discussions on setting up project variables from a Senario.
https://community.dataiku.com/t5/Using-Dataiku/Retrieving-the-value-of-a-metric-in-a-scenario-or-as-... 

Triggering a Scenario when Project Variable Changes.
https://community.dataiku.com/t5/Using-Dataiku/Trigger-a-scenario-when-project-variables-change/m-p/... 

[Kok]

Hi Tom,

I set the value of the project variable with the following code in python receipe:

#Check and set scenarios completion
project_handle = dataiku.api_client().get_project(dataiku.default_project_key())
vars = project_handle.get_variables()
vars['standard']['myVariable'] = "OK"
project_handle.set_variables(vars)

[Turribeach]

@Kok Have you solved this problem? I am thinking in writing some Python code to do this but no point in reinventing the wheel if you already have a solution for this. 

[Kok]

Hi Tom,

I tried asking ChatGPT for that question, and it suggested to use a "Parent Scenario" with a "Wait for scenario" plugin. But I can't find the "Wait for Scenario" plugin. Is there such a thing? Probably ChatGPT got the wrong information.

Thanks!

[tgb417]

@Kok ,

As far as I know ChatGPT is hallucinating that plug-in.

[Turribeach]

Hahaha, just seen this post. Classic! ChatGPT making things up. Don't believe everything it says, sometimes it makes things up!!!

https://arstechnica.com/tech-policy/2023/06/lawyers-have-real-bad-day-in-court-after-citing-fake-cas...

[Turribeach]

@Kok By the way did you solve this problem if so, how? I am thinking in writing some Python code to do this as I have seen several questions on this subject but I rather not reinvent the wheel if a solution already exists.