Comparing the most recently created model to the second most recently created model

ka
ka Registered Posts: 3 ✭✭✭

Hello,

I am trying to have a scenario step where after a new version of a model is created, it gets compared to the previously created version of the model using the "Model Comparisons" functionality. I then want to display the comparison results in a dashboard. I am having a really difficult time achieving this, and would appreciate any and all help. Thanks!


Operating system used: Windows

Best Answer

  • pmasiphelps
    pmasiphelps Dataiker, Dataiku DSS Core Designer, Registered Posts: 33 Dataiker
    Answer ✓

    Hi,

    One way to do this is to use the Evaluation Store (detailed doc: https://doc.dataiku.com/dss/latest/mlops/model-evaluations/dss-models.html).

    Once you've deployed your model (as a green diamond in the flow), you can click on a dataset to evaluate its performance on (say a validation set), then create an evaluate recipe:

    Screen_Shot_2022-05-03_at_9_48_57_AM.png

    Make sure to create output datasets for the metrics, input dataset + predictions, and evaluation store:

    Screen_Shot_2022-05-03_at_9_53_30_AM.png

    Whenever your model is retrained + redeployed to the flow, run this evaluate recipe to re-compute performance metrics for the new model. Then you'll see comparison metrics in your eval store icon, all in a nice dashboard:

    Screen_Shot_2022-05-03_at_9_48_57_AM.png

    Screen Shot 2022-05-03 at 9.54.50 AM.png

    Hope this helps!

    Best,

    Pat

Answers

Setup Info
    Tags
      Help me…