Submit your use case or success story to the 2023 edition of the Dataiku Frontrunner Awards ENTER YOUR SUBMISSION

Model Performance Alerting

It would be nice if we could trigger automatic alerts based on model evaluation stores with a simple click while in the evaluation store. Today you would have to write a bunch of custom python scripting to create a check, schedule a scenario that reports it back out.

It should be as simple as this:

1. Create Evaluation Store

2. Select metric in evaluation store and pick a threshold (AUC <0.8) then alert, retrain etc

3. Pick method of delivering alert (slack, email etc)

Status changed to: Needs Info


The logic of automating actions after model monitoring is based on the metrics & checks computed on Model Evaluation Store. Those can then be used in Scenarios to build any logic (as explained in I am not sure in which cases you need to write Python?

And from the current logic, this is very close to what you describe in your bullet list, the only difference is that the actions on point 2 & the point 3 are done in a scenario, and not on Model Evaluation Store.

Level 3

Yeah pretty much. You have to go into the scenario and run all the steps.

In an ideal state, I hit something like monitor this model - set up some thresholds in the evaluation store and it alerts me every time that evaluation store is built if any of the metrics dropped below their threshold.