Creatring a custom scoring metric

mkamal
mkamal Registered Posts: 2 ✭✭✭✭

Hello @Alex_Combessie
,

I want to code a custom scoring function for scikit-learn's average-precision-score. It's a for a Keras image binary classification model. I'm not familiar with how the Dataiku API works, but I want to pass the array dimensions to y_valid and y_pred.

Here's what I have:

from sklearn.metrics import average_precision_score

def score(y_valid, y_pred):
"""
Custom scoring function.
Must return a float quantifying the estimator prediction quality.
- y_valid is a pandas Series
- y_pred is a numpy ndarray with shape:
- (nb_records,) for regression problems and classification problems
where 'needs probas' (see below) is false
(for classification, the values are the numeric class indexes)
- (nb_records, nb_classes) for classification problems where
'needs probas' is true
- [optional] X_valid is a dataframe with shape (nb_records, nb_input_features)
- [optional] sample_weight is a numpy ndarray with shape (nb_records,)
NB: this option requires a variable set as "Sample weights"
"""
average_precision = average_precision_score(y_valid, y_pred)

return average_precision

Best Answer

  • Alex_Combessie
    Alex_Combessie Alpha Tester, Dataiker Alumni Posts: 539 ✭✭✭✭✭✭✭✭✭
    Answer ✓

    Hi,

    That seems to be a good piece of code to start with. Are you encountering an error when training? If so, could you post the error message, please?

    Hope it helps,

    Alex

Answers

Setup Info
    Tags
      Help me…