Asynchronous project variables recursive build

sseveur

Hi,

We've seen that the project variables used by the recipes in a recursive build aren't synchronized when the recipe is ran.It seems the project variables are set at the begining of the recursive build and may cause some asynchronous events.

I explain the mechanic we've understood :

1. I send a force recursive build of my dataset at the end of my flow
2. The project variables seems to be send to an orchestrator
3. The orchestrator computes the dependencies and run the recipes according to the schema
4. If recipe 3 needs a project variable, the orchestrator will give it
   • The orchestrator gives it, if recipe 2 updated it meanwhile, the value given isn't up to date.
   • If the project variable has never been set, recipe 3 will fail.

Could you confirm this mechanic ?

Is there a place in the Dataiku documentation confirming this behavior ?

Greetings,

Steven

Alexandru

Hi @sseveur
,

I assume you are using get_custom_variables() to read the project variable values? This would read the values at the start of the recipe/flow rebuild thus would have the behaviour you describe.

If you want the read the updated values set with set_variables() in the recipe you can read the variable using get_variables() instead.

vars = project.get_variables()
current_variable_value = vars["standard"]["my_variable"])
#this would return the variables at the start of the variable run
variable_at_start_of_recipe_run = dataiku.get_custom_variables()["my_variable"]

sseveur

Hi,

No we are using the variables through the ${variable_name} in a SQL recipe that's the main problem.

Didn't know there was a difference between the API call and the get_custom_variables(). Thanks for the info.