Custom trigger to run itself?
Hi everyone,
I want to execute scenario again if its fail, so it can try 3 times, sometimes kubernetes fails or, spark fails could be fixed after run again so, i dont miss time range between fail and fix manually.
This is the code created with LLM, it used python for that. I changed project name to variable "project name", normally it written project name already, dont worry.
from dataiku import api_client
import time
def create_retry_trigger(scenario_name, project_key=None, max_retries=3, wait_time=300):
"""
Creates a custom trigger to retry failed scenarios in Dataiku using scenario name (or ID).
Parameters:
scenario_name (str): Scenario ID (or name, if same) to retry
project_key (str): Project key where the scenario is located. If None, uses the current project.
max_retries (int): Maximum number of retry attempts.
wait_time (int): Time to wait between retries in seconds.
Returns:
None
"""
client = api_client()
if project_key is None:
project = client.get_default_project()
else:
project = client.get_project(project_key)
try:
# Directly get scenario by ID (if name matches ID)
scenario = project.get_scenario(scenario_name)
latest_runs = scenario.get_last_runs(1)
if not latest_runs:
print(f"No previous runs found for scenario '{scenario_name}'.")
return
latest_run = latest_runs[0]
if latest_run['result']['outcome'] == 'FAILED':
metadata = scenario.get_custom_metadata()
retry_count = int(metadata.get('retry_count', 0))
if retry_count < max_retries:
scenario.set_custom_metadata({'retry_count': str(retry_count + 1)})
print(f"Retrying scenario '{scenario_name}'. Attempt {retry_count + 1} of {max_retries}")
time.sleep(wait_time)
scenario.run()
else:
print(f"Maximum retry attempts ({max_retries}) reached for scenario '{scenario_name}'")
scenario.set_custom_metadata({'retry_count': "0"})
else:
scenario.set_custom_metadata({'retry_count': "0"})
except Exception as e:
print(f"Error while processing scenario '{scenario_name}': {str(e)}")
# Example usage
create_retry_trigger(
scenario_name="Project_name",
project_key=None, # Use None for the current project or specify a project key
max_retries=3,
wait_time=300 # 5 minutes
)
Operating system used: Windows
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,329 NeuronSo what’s the problem here?
-
It didnt work.:) I just sent solution of llm to explain exactly what i want.
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,329 Neuron
It didnt work ⇒ Not really something I can work with. What's the error? What part does it fail on? What steps have you tried to fix it?
In my experience asking LLMs to write code for complex actions like this can easily lead to hallucinations. It's best to go step by step with simple actions and as you get the code you test it and fix each step as needed.
In any case this path seems wrong to me. Scenarios already have retry capabilities. You can do this in each scenario step. So set your retries there and you won't have to use API code. Finally if you want to write API code I suggest you look at the Dataiku Developer site. It's full of great and working examples and should have most of what you need. Here is the section for the Scenarios:
-
No error cause this code snippet didnt work at all(didnt triggered). I just created this to use quickly and provide it here to explain people, much simple way.
Thanks for your direction to link and guidance.
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,329 Neuron
Python code in a Scenario can be hard to debug. It's recommended you first test the code in a Jupyter Notebook, if possible.