It would be nice if we could trigger automatic alerts based on model evaluation stores with a simple click while in the evaluation store. Today you would have to write a bunch of custom python scripting to create a check, schedule a scenario that reports it back out.
It should be as simple as this:
1. Create Evaluation Store
2. Select metric in evaluation store and pick a threshold (AUC <0.8) then alert, retrain etc
3. Pick method of delivering alert (slack, email etc)