Automating the model based on drift

devapavan
Level 1
Automating the model based on drift

Hi Team,

 

I have build a flow in designer with model, evaluation recipe and evaluation store. How to automate my scenario to retrain my model whenever there is drop in auc or whenever there is input data drift.

Input data drift as in not with the trigger but with the automation acenario steps

Alternatively if you could provide any video references it would be great. 

 

Note: Videos like model lifecycle or drift analysis on the knowledge base are just saying what are these things but are not explaining end to end right from deployer to production. It would be great if could provide end to end video

Thanks,

Pavan 

0 Kudos
1 Reply
Emma
Dataiker

Hey @devapavan , 

You want to take advantage of Dataiku's "Metrics and Checks." Documentation here

To use the AUC, go to your Evaluation Store > Status and add in AUC (see screenshot). Then, in the Evaluation Store's Settings > Status checks, create a Check that sets the threshold for acceptable performance before you want your model to be retrained. In my case, it is 0.8 (see screenshot). 

Then, from within your Scenario, you can add a Step called "Run checks." Afterwards, feel free to add as many other necessary steps to your Scenario. In my example, I rebuild the training dataset and then retrain the model (see screenshot). 

It works similarly for drift! Take your drift Metric and use it to create a Check and then include it in a scenario.

Hope that helps, 

Emma 

 

 

0 Kudos