Data Quality Rules
There are some data quality rules that allow you to set a soft fail to return a warning:
https://knowledge.dataiku.com/latest/ensure-quality/data/tutorial-data-quality.html
Others that the only option is to fail them such as the "No empty values exist in column". Is there a way to change the behavior of these to warn vs fail?
or do i need to create a custom python to do that for me?
Dataiku version used: 14.5.2
Answers
-
Hi @stainlessteelra ,
Thank you for reaching out and for the question!
>Is there a way to change the behavior of these to warn vs fail?
No, indeed that's a limitation. But we're happy to extend that. If you want to request it.
Could you tell me what are the rules you want to be able to set it to warn/fall? And do you know which company you're working for?
Thanks,
Han
-
Hi Han,
I'd like to request it.
I'd like it if all default rules had a box you could check to trigger a warning instead of a failure (especially for testing new data quality rules)
Also along these same lines, i'd love to be able to use the "row-level extraction" on data quality rules that fail with custom SQL or Python that triggers row level Pass, Fail with a new column that shows the rule(s) that the row failed to get placed in the failed rows dataset
I work for GE Aerospace -
Hi @stainlessteelra,
You mentioned wanting warnings especially when testing new data quality rules. Could you walk me through a recent example?
I'm trying to understand how you see error and warning and what are their concrete use case for you.
- What were you checking, and what made you hesitant to have the rule fail?
- What happens today when that rule fails, and how do you work around it?
- What would you need to learn before trusting the rule enough to make it a failure? Would some rules always stay as warnings?
- If it produced a warning, who would review it, where would they expect to see it, and what would they do next?
-
- What were you checking, and what made you hesitant to have the rule fail?
- We see issues where key fields aren't populated (not table keys) and we need to be able to trigger the data steward to review and correct these rows in the source. We don't want the dataset built to fail but we want to be alerted and have records show up in a data quality table.
- What happens today when that rule fails, and how do you work around it?
- We have yet to setup rules outside of hard constraints (due to this)
- What would you need to learn before trusting the rule enough to make it a failure? Would some rules always stay as warnings?
- Some would always stay as warning but i would think of the process of going from warning to failure as a similar process to Statistical process control. Once we prove that we have process stability then we could flip from warning to failure to prevent data trust issues. Or our Subject Matter Expert may never want us to convert.
- If it produced a warning, who would review it, where would they expect to see it, and what would they do next?
- The SME, Ideally we'd use the "extract failed rows" dataiku functionality and alert the data steward to review and triage.
- What were you checking, and what made you hesitant to have the rule fail?
