Asynchronous project variables recursive build
Hi,
We've seen that the project variables used by the recipes in a recursive build aren't synchronized when the recipe is ran.It seems the project variables are set at the begining of the recursive build and may cause some asynchronous events.
I explain the mechanic we've understood :
- I send a force recursive build of my dataset at the end of my flow
- The project variables seems to be send to an orchestrator
- The orchestrator computes the dependencies and run the recipes according to the schema
- If recipe 3 needs a project variable, the orchestrator will give it
- The orchestrator gives it, if recipe 2 updated it meanwhile, the value given isn't up to date.
- If the project variable has never been set, recipe 3 will fail.
Could you confirm this mechanic ?
Is there a place in the Dataiku documentation confirming this behavior ?
Greetings,
Steven
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,226 Dataiker
Hi @sseveur
,I assume you are using get_custom_variables() to read the project variable values? This would read the values at the start of the recipe/flow rebuild thus would have the behaviour you describe.
If you want the read the updated values set with set_variables() in the recipe you can read the variable using get_variables() instead.
vars = project.get_variables() current_variable_value = vars["standard"]["my_variable"]) #this would return the variables at the start of the variable run variable_at_start_of_recipe_run = dataiku.get_custom_variables()["my_variable"]
-
Hi,
No we are using the variables through the ${variable_name} in a SQL recipe that's the main problem.
Didn't know there was a difference between the API call and the get_custom_variables(). Thanks for the info.