Sending Custom Email Notification for Empty Datasets in Dataiku Scenario

Devlina

Hi everyone,

I have a scenario in Dataiku where I am running step by step one flow and there I need to check four datasets. If any of them are empty, I want to send a custom email notification to users mentioning which datasets are empty. Additionally, the process should continue only if at least two datasets have data—otherwise, it should stop.

I want to implement this using a Custom Python step instead of the built-in “Send message” step. Is this possible? If yes, how can I achieve it?

Previously I have tried passing a quality checks for those dataset but as my motive is to run the process even if one dataset has data in it, that method is failing.

Any help would be appreciated!

Thanks!

Turribeach

A custom Python step would work but it's way too much code when this can be done with visual steps in a much more clear way. I covered the reasons for not wanting to use code for a similar requirement on this post. So the way to do this requirement is as follows:

Create a new Compute Metrics scenario step and give it a name without spaces or strange characters. Add the dataset you want to count the rows to it as an item:

Now add a new Define variables scenario step as follows:

toNumber(filter(parseJson(stepOutput_Compute_Metrics)['PROJECT_KEY.datasetid_NP']['computed'], x, x["metricId"]=="records:COUNT_RECORDS")[0].value)

"Compute_Metrics" must the name of your compute metrics step. Replace PROJECT_KEY and datasetid as appropiate (note datasetid is not the same as dataset name, get the datasetid from the URL when you visit your dataset).

Create a new send message scenario step, set "Run this step" = "If condition satisfied" and use this criteria:

Dataset_Record_Count > 0 && (outcome == 'SUCCESS' || outcome == 'WARNING')

Finally modify any subsequent scenario steps as well setting "Run this step" = "If condition satisfied" and use the same criteria as the send mail step. Here is another post I did with more low level detail on how this works.

Devlina

Thank you so much