data checks based on validation data

danf101
danf101 Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 4 Partner
in Product Ideas

A common scenario is to validate new incoming data quality based on statistics calculated as metrics over an older/original/validation data set. If the dropdown checks over data metrics could include an option to validate in relation to previously calculated metrics on a different data set - that would solve this.

see:

https://community.dataiku.com/t5/Using-Dataiku/Recommend-Way-for-Comparing-Metrics-between-Multiple-Flow/m-p/23166/emcs_t/S2h8ZW1haWx8bWVudGlvbl9zdWJzY3JpcHRpb258S1pXU0ZCVEdDTlpQUkN8MjMxNjZ8QVRfTUVOVElPTlN8aEs#M9080

for a code implementation of the idea

Tagged:
Share on FacebookShare on Twitter
1
Up
1 votes

Developing · Last Updated

Hello! Wanted to let you know we are currently working on something quite similar - we will have new data quality rules for anomaly detection compared to observed historical ranges coming soon. I will update you as soon as we have more details to share!

Comments

  • Elie
    Elie Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered, Product Ideas Manager Posts: 32 Dataiker

    Very interesting ! Thanks for your idea @danf101
    . Y    our idea meets the criteria for submission, we'll reach out should we require more information.

    Feel free to leave a comment in the discussion about how this capability would help you or your team.

    Take care,


    Elie

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…