raise error when a check does not work

Options
Tanguy
Tanguy Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2023 Posts: 112 Neuron

Dataiku offers the ability to add custom metrics and checks which is very useful for testing the flow.

However, some checks can be improperly defined or break for whatever reason. In this case, dataiku ignores the check, and the flow can be built without the user being notified of a problem. This is a silent error: the check fails but no error is raised.

Here is a demo showing this problem:

  • the user has a custom python check containing an error (the check fails)

1.jpg

  • the check is supposed to run at each build on the table (e.g. the following option is checked) 2.jpg
  • and yet, when the table is built, no errors is raised: the user is not notified of the problem on the check potentially (and more importantly) on the table

3.jpg

It would be appreciated if dataiku could raise an error in this case (which, from what I know, is the default behaviour of common testing frameworks).

7
7 votes

Released · Last Updated

Comments

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,726 Neuron
    Options

    Wow! How on earth this is not considered a bug? ("It's not a bug it's a feature"). This is a bug 100% to me since there is no way that you would expect that a custom piece of code that fails will do so silently. This bug is extremely dangerous. We rely a lot on metrics and checks to assure our data meets our data quality requirements. In particular machine learning models are extremelly susceptible to values out of range, nulls, etc. Gets my vote 110% and I will ask others on my team to vote for this.

  • Katie
    Katie Dataiker, Registered, Product Ideas Manager Posts: 105 Dataiker
    Options

    Hi @tanguy
    ,

    Thank you very much for the feedback! We are exploring general improvements here and will let you know if we have any updates.

    Best,

    Katie

  • AshleyW
    AshleyW Dataiker, Alpha Tester, Dataiku DSS Core Designer, Registered, Product Ideas Manager Posts: 161 Dataiker
    Options

    Hi,

    Updating this thread to let you know this has been fixed in all 12.1+ versions of Dataiku.

    Cheers,

    Ashley

Setup Info
    Tags
      Help me…