Log table to capture job status and steps for every dataset build

Hi @Curious_Analyst
,
This data should already be most available in the "internal stats datasets" https://doc.dataiku.com/dss/latest/connecting/internal-stats.html
, a DSS admin can create internal stats datasets from +Dataset - Internal - Select :Scenario Runs" and/or Job
click preview and you will be able to create a dataset with the following columns.

You can of course later calculate the duration in prepare as you have the start/end time.

Let us know if that helps. Additionally, if you need even more granular compute information you can CRU solution: https://knowledge.dataiku.com/latest/solutions/governance/solution-cru.html

Thanks

So indeed, if the data in the table from the runtime db is not see below.
You need to extract the information you are looking within each and write to a dataset.

You can do this as follows :
Create variables you want via code or use the built variables available from the scenarion

To print all scenario variables available you can use this python step in the scenario to see if the information you are looking is already available ( in most cases it should be)

from dataiku.scenario import Scenario
import json
print(json.dumps(Scenario().get_all_variables(), indent=2))

To obtain more granular information around failed steps and their summary you can use code :

import json
import dataiku
from dataiku.scenario import Scenario
# this works only with code steps
#def get_failed_steps(step_results):
  #  return [f"{step}: {data['thrown']['message']}" for step, data in step_results.items() if data["outcome"] == "FAILED"]

#variables = Scenario().get_all_variables()
#step_results = variables["stepResults"]
#failed_steps = get_failed_steps(step_results)
#failed_steps_str = ", ".join(failed_steps)

#scenario = Scenario()
#variable_param = {"failed_steps": failed_steps_str}
#scenario.set_scenario_variables(**variable_param)
#print(failed_steps_str)



p = dataiku.api_client().get_default_project()
s = p.get_scenario('TEST')  # the scenario id, from the location bar
r = s.get_current_run()
id_to_report_items = {}

for sr in r.get_details()["stepRuns"]:
    id_to_report_items[sr.get("step", {}).get("id", "n/a")] = sr.get('additionalReportItems')

failed_steps = []

for key in id_to_report_items.keys():
    step = id_to_report_items[key]
    for i in step:
        if i['type'] == 'JOB_EXECUTED' and i['outcome'] == 'FAILED':
            failed_steps.append(str(i['thrown']))

scenario = Scenario()
variable_param = {"failed_steps": '\n'.join(failed_steps)}
scenario.set_scenario_variables(**variable_param)

Now that you set the new "failed steps" variables, you can append to a dataset using the reporter step from the scenario or code directly e.g SQL Executor or Python code.

The UI option is to use the reporter ( first you first manually create a dataset e.g upload an empty csv with the column names you need note you need the timestamp column to match and have the correct type. As explained here : https://community.dataiku.com/t5/Using-Dataiku/How-to-use-available-variables-in-reporters-section-Send-to/m-p/21538



Hope that helps!