Run a scenario when 3 other scenarios completed (in random order)
Hi,
I am trying to run a scenario in dataiku automation when 3 other scenarios have completed. But they can be completed in any order. Is there a way to accomplish this?
Thanks!
Answers
-
Hi Jordan,
Thank you for your reply.
In my case, I need to run the scenario when all 3 other scenarios have completed successfully.
For example, we have scenarios A, B, C, D, I need to have D triggered only when A, B, and C all completed, then D can be triggered. But scenario A, B, and C can finish running in any order. The problem is I don't know which one will run last. Basically, scenario A, B, and C are running in parallel.
Any suggestion would be much appreciated.
-
tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron
Sounds like you want to run A, B, and C in parallel not sequentially. If you could ran them sequentially you could have a scenario that starts other Senarios and checks status. The Scenario would started Scenario A, check if it is successful and then Run Scenario B, and then Run C…
Because you want to run them in parallel, knowing that you have a complete set that is internally consistent for all A, B, and C can be a bit challenging.
One idea I have is to set project level variables at the end of each run of A, B, and C. And then in D look for those statuses by checking those variables.
This can be a bit challenging.
I’ve submitted an enhancement request on the subject.
https://community.dataiku.com/t5/Product-Ideas/Improved-UX-for-Senario-Variables-Setup/idi-p/30707
If you can tell by looking a the condition of data in the last dataset of each of the flows that the data has been written, you might be able to create a custom data set check and use this as your indication at the beginning of Scenario D, that the other Senarios have completed successfully.
You could do other things if the data is in SQL accessible databases writing a SQL query that return true when everything is done. And then put a check at the beginning of D looking for this status to be true.
-
Hi Tom,
Thank you so much for your reply. I already tried what you suggested with project variable, but I guess it's because of concurrency issue, the timing for read and write of the status are not in sync and it didn't work.
I also tried with using a status flag in database and it's the same result.
-
Hi Tom,
I tried asking ChatGPT for that question, and it suggested to use a "Parent Scenario" with a "Wait for scenario" plugin. But I can't find the "Wait for Scenario" plugin. Is there such a thing? Probably ChatGPT got the wrong information.
Thanks!
-
tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron
@Kok
,As far as I know ChatGPT is hallucinating that plug-in.
-
tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron
@Kok
Regarding Project variables are you sure that you actually go it set. I found this a challenge the first time I tried.Here are some further discussions on setting up project variables from a Senario.
https://community.dataiku.com/t5/Using-Dataiku/Retrieving-the-value-of-a-metric-in-a-scenario-or-as-project/m-p/30706/highlight/true
Triggering a Scenario when Project Variable Changes.
https://community.dataiku.com/t5/Using-Dataiku/Trigger-a-scenario-when-project-variables-change/m-p/28828 -
Hi Tom,
I set the value of the project variable with the following code in python receipe:
#Check and set scenarios completion
project_handle = dataiku.api_client().get_project(dataiku.default_project_key())
vars = project_handle.get_variables()
vars['standard']['myVariable'] = "OK"
project_handle.set_variables(vars) -
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,971 Neuron
Hahaha, just seen this post. Classic! ChatGPT making things up. Don't believe everything it says, sometimes it makes things up!!!
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,971 Neuron
@Kok
Have you solved this problem? I am thinking in writing some Python code to do this but no point in reinventing the wheel if you already have a solution for this. -
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,971 Neuron
@Kok
By the way did you solve this problem if so, how? I am thinking in writing some Python code to do this as I have seen several questions on this subject but I rather not reinvent the wheel if a solution already exists.