How to add a custom calculated model metric into the evaluation store in Dataiku?

Suhail

Hello everyone,

I am currently working on a project in Dataiku and trying to log a custom model metric into the evaluation store. The model is not a visual ml model its a custom model that is logged in Dataiku as a saved model using the mlflow integration that Dataiku offers.
However, I am not sure how to add a custom calculated model metric into the evaluation store output.

Can anyone help me with this? Any guidance or insights would be much appreciated.

Thank you in advance!

Pierre-MaëlM

Hello @Suhail

Thank you for your message. At the moment it is not possible to use a custom metric in an evaluation with a MLflow model.

We have forwarded your inquiry to our product team.

Best regards,
Pierre-Maël

Anis

Is this still the case? i think this capability has been introduced in 12.3. Can you provide an example of how one might add custom metrics to the model evaluation store for mlflow models. thank you!