Setup Record Count Check in Scenario

Options
khairulfathi
khairulfathi Registered Posts: 11 ✭✭✭✭
edited July 16 in Using Dataiku

Hi,

I have created a check 'record_count' for my dataset 'avs_delta_stg'. I've compute metrics step 'Metrics' and run check step named 'Delta' as part of initial steps. How do I retrieve the result of this check to set as condition for next step? Tried following this URL (https://doc.dataiku.com/dss/latest/scenarios/variables.html#retrieving-the-message-of-a-check) but still doesn't product expected output.

In the subsequent step, I've set the following as condition:

Failure during evaluation of'parseJson(stepOutput_delta)['PROJNAME.avs_delta_stg'].results == 'OK'' as formula: ExpressionError: Cannot retrieve field from null

but getting the following error from log

parseJson(stepOutput_delta)['PROJNAME.avs_delta_stg'].results == 'OK'

Why I put .results == 'OK' is because I notice the result could be OK, ERROR or EMPTY. Appreaciate your feedback.

Thanks.

Best Answer

  • fchataigner2
    fchataigner2 Dataiker Posts: 355 Dataiker
    edited July 17 Answer ✓
    Options

    Hi,

    for such tasks, it's often simpler to use a 'Execute python step' to take a look at the variables available (by dumping them to the log). If after your 'Run checks' step you run:

    from dataiku.scenario import Scenario
    import json
    print(json.dumps(Scenario().get_all_variables(), indent=2))

    then you'll see in the logs the structure of the data produced by the 'Run checks' step.

    Rather than use parseJson(), you could do:

    strval("stepOutput_THE STEP NAME").getPath('.["PROJNAME.avs_delta_stg"].results[0].value.outcome').join('') == 'WARNING'

    where "THE STEP NAME" is the name of the 'Run checks' step (possibly with whitespace in it, in your case if should be stepOutput_Delta). This will retrieve the outcome of the first check of the dataset. Note the ".join('')" at the end, needed because getPath() actually returns an array even if there is only one match.

Setup Info
    Tags
      Help me…