Any template or example of model monitoring?

[marcochang1028]

Hi,

my client want to have a summary page to watch a model api status with the following information:

1. Evaluation store info

2. api health info

3. other important info for the model monitoring or business

Do you have any template or example that can achieve this requirement by Dashboard, Webapps or Application Designer?

[JordanB]

Hi @marcochang1028,

I've provided a few examples with documentation on how users may set up model monitoring in DSS.

Set up a custom event server to monitor your model.

1. Setup your instance with an Event Server - https://doc.dataiku.com/dss/latest/operations/audit-trail/eventserver.html
2. Configure your API node to send health info and prediction logs to the event server - https://doc.dataiku.com/dss/latest/operations/audit-trail/apinode.html.  How-to for monitoring an API Endpoint: https://knowledge.dataiku.com/latest/mlops-o16n/real-time-apis/concept-monitor-api-endpoint-output.h...

Set up a model feedback loop:

1. Create a dataset that points to the api logs stored by the Event server
2. Cleanup this dataset to keep only the features + prediction columns
3. Add an ‘Evaluate recipe’ taking this dataset as an input and the model and output it into a Model Evaluation Store
4. run it (and optionally automate it that with a scenario)

Create a monitoring Dashboard: Pull the data you need from APIs and display model metrics with visuals on a design node.

Utilize the Govern Node's Model Registry:
In the model registry page, all model versions grouped by model from all projects on all design nodes are displayed via a table view. You are able to see the evolution of the metrics that are computed in model dedicated MES for each model.

I recommend taking a look at the Model Monitoring Tutorial: https://knowledge.dataiku.com/latest/mlops-o16n/model-monitoring/tutorial-model-monitoring-basics.ht...

Please let me know if you have questions.

Kind Regards,

Jordan

[marcochang1028]

Hi @JordanB 

Glad to receive such a detailed instruction. I will try it ASAP and give you my feedback.

Thanks.