jobId from scenario execution
Hi. I created a scenario that executes a python recipe and builds a dataset. I tried to exact jobId within the recipe using get_custom_variables.get("jobId"). However, the approach is unreliable, as the code returns a value and times and sometimes it doesn't. The scenario is run using a time-based trigger. Could someone please help me with a reliable solution to derive jobId. Thank you.
Dataiku version used: 14
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023, Circle Member Posts: 2,718 NeuronWhat are you trying to achieve? What is your actual requirement?
-
A jobId variable is created during a scenario execution. I want to read it inside the recipe.
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023, Circle Member Posts: 2,718 Neuron
Why would you want to do that? What’s the end goal? There are a couple of ways of doing this but without understanding what are you trying to do I can’t suggest the best option.
-
I want to extract the jobId and get its associated log at the end of execution. As I stated, I have the setup in place. However, the method I have been using for tracing jobId isn't effective, so I am seeking alternate approaches. Thanks!
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023, Circle Member Posts: 2,718 Neuron
It is still unclear why you want to "extract the jobId and get its associated log at the end of execution". That's not a requirement, that a way to achieve your requirement. What do you want to do with the JobID and the log?
Scenarios can generate many JobIDs, depending on their steps. You can start with the Scenario Run ID which you can fetch in a Custom Python Scenario Step as follows:from dataiku.scenario import Scenario scenario = Scenario() all_scenario_variables = scenario.get_all_variables() print(f"Scenario Run ID: {all_scenario_variables['scenarioTriggerRunId']}")Then you will need to inspect the scenario object. However doing this while the scenario is running is not going to work. That is why you need to give a clear requirement of you are trying to do, not how you think you can do it.