Custom Metrics Value Error

Josh

Hi,

I am trying to build a custom metric like below:

Trying to enrich exception: com.dataiku.dip.io.SocketBlockLinkKernelException: Failed to train : <class 'ValueError'> : Custom evaluation function not defined from kernel 

However I am getting a value error:

from sklearn.metrics import cohen_kappa_score

def quadratic_weighted_kappa(y_valid, y_pred):
    """
    Custom scoring function.
    Must return a float quantifying the estimator prediction quality.
      - y_valid is a pandas Series
      - y_pred is a numpy ndarray with shape:
           - (nb_records,) for regression problems and classification problems
             where 'needs probas' (see below) is false
             (for classification, the values are the numeric class indexes)
           - (nb_records, nb_classes) for classification problems where
             'needs probas' is true
      - [optional] X_valid is a dataframe with shape (nb_records, nb_input_features)
      - [optional] sample_weight is a numpy ndarray with shape (nb_records,)
                   NB: this option requires a variable set as "Sample weights"
    """
    
    metric = cohen_kappa_score(y_valid, y_pred, weights='quadratic')
    
    return metric

Any Help with this would be greatly appreciated.

fchataigner2

Hi

your scoring function has to be named score(...), you can't use quadratic_weighted_kappa(...)