Community Conundrum 10: The Titanic is now live Learn more

Data quality : Monitoring on datasets processing

Level 2
Data quality : Monitoring on datasets processing

Hi,



I'm asking about how DSS monitors issues during datasets processing. I see two kinds of potential issues: 




  • Volume : Inconsistant number of records in a dataset (eg : I expected at least 1k records per  day for my "webtraffic" dataset)

  • Schema / values:  One or more rows have fields that don't respect the defined schema or expected values (eg : in webtraffic dataset, IP adresses are not valid or values of a date field are not expected). 



 Is there a way to monitor / handle those errors in DSS and be notified by email or something ?



Thanks,



Romain.



 



 

0 Kudos
2 Replies
Dataiker
Dataiker
Hi Romain,

These features are on our roadmap. You can get in touch with our Sales team if you'd like more details.

As of today, you could write a custom recipe in Python for instance and write your tests.
Jeremy, Product Manager at Dataiku
Level 2
Author
Good new!

Thanks for the quick reply 🙂
0 Kudos
Labels (3)