Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi all,
I am running the model evaluation recipe and writing the results out to the dataset, I want to add checks to this dataset to look for model drift.
The evaluation recipe appends to the dataset with every run, creating a handy archive of model performance, but how can I configure my checks to look at the most recent scores only, since I am only interested in triggering a retrain based on the most recent performance, not an average?
Ben
Hi,
You can create a custom metric that show you the most recent score, and then your check will be based on this metric.
To do that, you can go to the metrics tab -> edit -> python probe and do something like this
def process(dataset, partition_id):
df = dataset.get_dataframe()
most_recent_score = df[df['date'] == np.max(df['date'])][MY_SCORE].values[0]
metric_values = {'My most recent score':most_recent_score}
return metric_values
Cheers,
Once adding the custom python probe, it can be found in the Metrics list in the screenshot, here I have chosen 3 on the 11 available metrics.
Hi,
You can create a custom metric that show you the most recent score, and then your check will be based on this metric.
To do that, you can go to the metrics tab -> edit -> python probe and do something like this
def process(dataset, partition_id):
df = dataset.get_dataframe()
most_recent_score = df[df['date'] == np.max(df['date'])][MY_SCORE].values[0]
metric_values = {'My most recent score':most_recent_score}
return metric_values
Cheers,
Thanks @duphan - when once creates a Python metrics like this, where can it then be selected? I don't see it appear as an option in the available metrics?
I also found another solution to this, syncing to an additional table, which is set to append, and changing the first metrics table to be overwrite, where I then set my checks:
Ben