Setup Record Count Check in Scenario

khairulfathi

Hi,

I have created a check 'record_count' for my dataset 'avs_delta_stg'. I've compute metrics step 'Metrics' and run check step named 'Delta' as part of initial steps. How do I retrieve the result of this check to set as condition for next step? Tried following this URL (https://doc.dataiku.com/dss/latest/scenarios/variables.html#retrieving-the-message-of-a-check) but still doesn't product expected output.

In the subsequent step, I've set the following as condition:

Failure during evaluation of'parseJson(stepOutput_delta)['PROJNAME.avs_delta_stg'].results == 'OK'' as formula: ExpressionError: Cannot retrieve field from null

but getting the following error from log

parseJson(stepOutput_delta)['PROJNAME.avs_delta_stg'].results == 'OK'

Why I put .results == 'OK' is because I notice the result could be OK, ERROR or EMPTY. Appreaciate your feedback.

Thanks.

fchataigner2

Hi,

for such tasks, it's often simpler to use a 'Execute python step' to take a look at the variables available (by dumping them to the log). If after your 'Run checks' step you run:

from dataiku.scenario import Scenario
import json
print(json.dumps(Scenario().get_all_variables(), indent=2))

then you'll see in the logs the structure of the data produced by the 'Run checks' step.

Rather than use parseJson(), you could do:

strval("stepOutput_THE STEP NAME").getPath('.["PROJNAME.avs_delta_stg"].results[0].value.outcome').join('') == 'WARNING'

where "THE STEP NAME" is the name of the 'Run checks' step (possibly with whitespace in it, in your case if should be stepOutput_Delta). This will retrieve the outcome of the first check of the dataset. Note the ".join('')" at the end, needed because getPath() actually returns an array even if there is only one match.