Data quality : Monitoring on datasets processing

Romain_NIO

Hi,

I'm asking about how DSS monitors issues during datasets processing. I see two kinds of potential issues:

• Volume : Inconsistant number of records in a dataset (eg : I expected at least 1k records per day for my "webtraffic" dataset)
• Schema / values: One or more rows have fields that don't respect the defined schema or expected values (eg : in webtraffic dataset, IP adresses are not valid or values of a date field are not expected).

Is there a way to monitor / handle those errors in DSS and be notified by email or something ?

Thanks,

Romain.

jereze

Hi Romain,

These features are on our roadmap. You can get in touch with our Sales team if you'd like more details.

As of today, you could write a custom recipe in Python for instance and write your tests.

Romain_NIO

Good new!

Thanks for the quick reply

sbourgeois-k

Hello Jeremy,

Have those monitoring features been developed since?

Or is it still on your roadmap?

Sébastien

Ignacio_Toledo

Hi @sbourgeois-k
,

They have been developed! You should check the metrics and checks documentation:

• Metrics: https://doc.dataiku.com/dss/latest/scenarios/metrics.html
• Checks: https://doc.dataiku.com/dss/latest/scenarios/checks.html

Also, there is a hands-on activity at https://knowledge.dataiku.com/latest/courses/automation/metrics-checks-hands-on.html

Cheers!

sbourgeois-k

Hello @Ignacio_Toledo
!

Great!

Thank you so much for your message and the links.

Have a nice day!

adamnieto

Another tip that might be useful with the metrics and checks is that you can automate them using scenarios in your project. Here is a good resource from the academy that walks through this: https://academy.dataiku.com/automation-course-1?next=%2Fautomation-course-1%2F668968