SCENARIO : get the name of dataset that failed in a step

Options
GanNing69
GanNing69 Registered Posts: 2

Hello,

am currently trying to recover the name of the datasets which failed in a build step of a scenario. So far I have only found the way to recover the first step which fails, however I specifically want to know the name of all the dataSets which fail in order to transfer them to a separate dataset. Do you have any ideas for doing this in this python step ?

For example (if the construction of one of these datasets fails), I want to be able to recover its name

Answers

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,727 Neuron
    Options

    Can you please post the Python code you got so far in code block (the </> icon on the toolbar)?

  • GanNing69
    GanNing69 Registered Posts: 2
    edited July 17
    Options

    Of course, here is my current python code which aims to recover the logs in the event of an error. The rest is just a succession of steps preconstructed by dataiku.

    import dataiku
    from dataiku import pandasutils as pdu
    import pandas as pd
    from dataiku.scenario import Scenario
    
    scenario = Scenario()
    
    current_scenario = Scenario()
    client = dataiku.api_client()
    p = client.get_default_project()
    
    #permet de récupérer la variable de scenario pour avoir l'heure de début du run
    project = dataiku.api_client().get_project(dataiku.default_project_key())
    variables = project.get_variables()
    DebScen = dataiku.get_custom_variables()['DebScen']
    
    #récupère le scenario et le run acutel avec ses détails
    scenario_id = current_scenario.scenario_trigger["scenarioId"]
    run = p.get_scenario(scenario_id).get_current_run()
    run_details = run.get_details()
    
    errors = []
    
    #PARCOURS CHAQUE STEP ET LEURS DETAILS
    for step_run in run_details.steps:
        step_name = step_run["step"]["name"]
        error_msg = None
        
    #RECUPERE LES LOGS TAIL (fin des logs) ET ASSIGNE LES MSG D'ERREUR A LA LISTE ERRORS
    
        result_log_tail = step_run.get("result", {}).get("logTail", {}).get("lines")
        if result_log_tail:
            error_msg = "\n".join(result_log_tail)
            
        for report_item in step_run["additionalReportItems"]:
            report_type = report_item["type"]
            
            report_log_tail = report_item.get("logTail", {}).get("lines")
            if report_log_tail:
                if error_msg : 
                    error_msg += "\n" + "\n".join(report_log_tail)
                else :
                    error_msg = "\n".join(report_log_tail)
                    
                    
    #CREATION DU DF CONTENANT : LA DATE DE DEBUT DU SCENARIO, LE NOM DU SCENARIO, NOM DE LA STEP QUI A FAIL ET SON MSG D'ERREUR
    
        if error_msg : 
            errors.append({"scenario_name" : scenario_id, "step_name" : step_name, "error_message" : error_msg, "deb_scenario" : DebScen})
    
    #CREATION DU DATASET FINAL AVEC L'HISTORIQUE DES DONNEES
    dfErrors = pd.DataFrame(errors)
    dataset = dataiku.Dataset('scenario_logs')
    existing_data = dataset.get_dataframe()
    upadating_ds = pd.concat([existing_data,dfErrors], ignore_index = True)
    
    #OUTPUT
    scenario_logs = dataiku.Dataset("scenario_logs")
    scenario_logs.write_with_schema(upadating_ds)
    
Setup Info
    Tags
      Help me…