Handling Project variables in Scenarios
Hi Team,
I have created scenario with consists of 5 steps.
step 1: sql query to check latest data
step 2: set project variable as execution start time using now() formula
step 3: Build the dataset
step 4: Run Ml model
step 5: set project variable as execution end time using now() formula.
Here question is step 2 and step 5 provide new values every re -run even if any other step fails. How can we maintain same values for step2 & 5 in next re-run when scenario failed in previous ran.
Please advice.
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,126 Neuron
Project variables are not a good place to store run time data. Scenario variables fit that use case much better. However I feel your post doesn't give enough information of what exactly you are trying to achieve and seems to be trying to reinvent the wheel. Why are you recording start and end times of some partial scenario steps? This information is already stored by Dataiku and available both from Dataiku API and in the Internal Stats dataset.