Data Quality Rules

stainlessteelra

There are some data quality rules that allow you to set a soft fail to return a warning:
https://knowledge.dataiku.com/latest/ensure-quality/data/tutorial-data-quality.html

Others that the only option is to fail them such as the "No empty values exist in column". Is there a way to change the behavior of these to warn vs fail?

or do i need to create a custom python to do that for me?

Dataiku version used: 14.5.2

Han_Han

Hi @stainlessteelra ,

Thank you for reaching out and for the question!

>Is there a way to change the behavior of these to warn vs fail?

No, indeed that's a limitation. But we're happy to extend that. If you want to request it.

Could you tell me what are the rules you want to be able to set it to warn/fall? And do you know which company you're working for?

Thanks,

Han

stainlessteelra

Hi Han,

I'd like to request it.

I'd like it if all default rules had a box you could check to trigger a warning instead of a failure (especially for testing new data quality rules)

Also along these same lines, i'd love to be able to use the "row-level extraction" on data quality rules that fail with custom SQL or Python that triggers row level Pass, Fail with a new column that shows the rule(s) that the row failed to get placed in the failed rows dataset

I work for GE Aerospace

Han_Han

Hi @stainlessteelra,

You mentioned wanting warnings especially when testing new data quality rules. Could you walk me through a recent example?

I'm trying to understand how you see error and warning and what are their concrete use case for you.

• What were you checking, and what made you hesitant to have the rule fail?
• What happens today when that rule fails, and how do you work around it?
• What would you need to learn before trusting the rule enough to make it a failure? Would some rules always stay as warnings?
• If it produced a warning, who would review it, where would they expect to see it, and what would they do next?