Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi,
I have created a check 'record_count' for my dataset 'avs_delta_stg'. I've compute metrics step 'Metrics' and run check step named 'Delta' as part of initial steps. How do I retrieve the result of this check to set as condition for next step? Tried following this URL (https://doc.dataiku.com/dss/latest/scenarios/variables.html#retrieving-the-message-of-a-check) but still doesn't product expected output.
In the subsequent step, I've set the following as condition:
parseJson(stepOutput_delta)['PROJNAME.avs_delta_stg'].results == 'OK'
but getting the following error from log
Failure during evaluation of'parseJson(stepOutput_delta)['PROJNAME.avs_delta_stg'].results == 'OK'' as formula: ExpressionError: Cannot retrieve field from null
Why I put .results == 'OK' is because I notice the result could be OK, ERROR or EMPTY. Appreaciate your feedback.
Thanks.
Hi,
for such tasks, it's often simpler to use a 'Execute python step' to take a look at the variables available (by dumping them to the log). If after your 'Run checks' step you run:
from dataiku.scenario import Scenario
import json
print(json.dumps(Scenario().get_all_variables(), indent=2))
then you'll see in the logs the structure of the data produced by the 'Run checks' step.
Rather than use parseJson(), you could do:
strval("stepOutput_THE STEP NAME").getPath('.["PROJNAME.avs_delta_stg"].results[0].value.outcome').join('') == 'WARNING'
where "THE STEP NAME" is the name of the 'Run checks' step (possibly with whitespace in it, in your case if should be stepOutput_Delta). This will retrieve the outcome of the first check of the dataset. Note the ".join('')" at the end, needed because getPath() actually returns an array even if there is only one match.
Hi,
for such tasks, it's often simpler to use a 'Execute python step' to take a look at the variables available (by dumping them to the log). If after your 'Run checks' step you run:
from dataiku.scenario import Scenario
import json
print(json.dumps(Scenario().get_all_variables(), indent=2))
then you'll see in the logs the structure of the data produced by the 'Run checks' step.
Rather than use parseJson(), you could do:
strval("stepOutput_THE STEP NAME").getPath('.["PROJNAME.avs_delta_stg"].results[0].value.outcome').join('') == 'WARNING'
where "THE STEP NAME" is the name of the 'Run checks' step (possibly with whitespace in it, in your case if should be stepOutput_Delta). This will retrieve the outcome of the first check of the dataset. Note the ".join('')" at the end, needed because getPath() actually returns an array even if there is only one match.