Automating the model based on drift

devapavan
devapavan Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 2

Hi Team,

I have build a flow in designer with model, evaluation recipe and evaluation store. How to automate my scenario to retrain my model whenever there is drop in auc or whenever there is input data drift.

Input data drift as in not with the trigger but with the automation acenario steps

Alternatively if you could provide any video references it would be great.

Note: Videos like model lifecycle or drift analysis on the knowledge base are just saying what are these things but are not explaining end to end right from deployer to production. It would be great if could provide end to end video

Thanks,

Pavan

Answers

  • Emma
    Emma Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 52 Dataiker

    Hey @devapavan
    ,

    You want to take advantage of Dataiku's "Metrics and Checks." Documentation here.

    To use the AUC, go to your Evaluation Store > Status and add in AUC (see screenshot). Then, in the Evaluation Store's Settings > Status checks, create a Check that sets the threshold for acceptable performance before you want your model to be retrained. In my case, it is 0.8 (see screenshot).

    Then, from within your Scenario, you can add a Step called "Run checks." Afterwards, feel free to add as many other necessary steps to your Scenario. In my example, I rebuild the training dataset and then retrain the model (see screenshot).

    It works similarly for drift! Take your drift Metric and use it to create a Check and then include it in a scenario.

    Hope that helps,

    Emma

Setup Info
    Tags
      Help me…