Custom Metrics for Custom Model in Evaluation Store

chkmcnugget

Is it possible to code out custom metrics to be used for a custom model in the Evaluation Store?

Currently, I have a custom model, and I have created a dataset of true values and predicted values to be used in the Standalone Evaluation Recipe. The Evaluation Store computes a set of metrics based on the dataset as a result. But as mentioned above, I would actually like to provide an entire function which calculates the metrics for the model output, and show that in the Evaluation Store.

Does anyone know if it is possible, or if the only alternative is to log the custom metrics results under Experimental Tracking?

fsergot

Good morning,

You can define custom metrics for a Dataiku model in the Visual Analysis screen, under BASIC > Metrics.

Once defined for your model, this metric will be computed as part of the performance drift in the Model Evaluation Store.

Note that, from your comment, I am unsure on your setup as you are mention the standalone evaluate recipe, which is meant to monitor model-less drifts. For a Dataiku model, the standard recipe to use is more the Evaluate recipe.