Comparing the most recently created model to the second most recently created model

[ka]

Hello,

I am trying to have a scenario step where after a new version of a model is created, it gets compared to the previously created version of the model using the "Model Comparisons" functionality. I then want to display the comparison results in a dashboard. I am having a really difficult time achieving this, and would appreciate any and all help. Thanks!

---

Operating system used: Windows

[pmasiphelps · Accepted solution]

Hi,

One way to do this is to use the Evaluation Store (detailed doc: https://doc.dataiku.com/dss/latest/mlops/model-evaluations/dss-models.html).

Once you've deployed your model (as a green diamond in the flow), you can click on a dataset to evaluate its performance on (say a validation set), then create an evaluate recipe:

Make sure to create output datasets for the metrics, input dataset + predictions, and evaluation store:

Whenever your model is retrained + redeployed to the flow, run this evaluate recipe to re-compute performance metrics for the new model. Then you'll see comparison metrics in your eval store icon, all in a nice dashboard:

 

Hope this helps!

Best,

Pat

[ka]

That actually helps quite a bit, thank you so very much!