Saved model cannot be used in Python

Jennnnnny

Hello,

I've got this error message when using a custom python recipe

Code:

# -*- coding: utf-8 -*-
import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu
from sklearn.metrics import silhouette_samples



# Read recipe inputs
EBS = dataiku.Dataset("EBS")

df = EBS_.get_dataframe()

model = dataiku.Model("NNNN") # replace by your model id

predictor = model.get_predictor() # get model
preprocessing = predictor.get_preprocessing() # get preprocessor

df_transformed = predictor.preprocess(df)[0] # transform data

scores = silhouette_samples(df_transformed, df.cluster_labels) # compute silhouette scores

df['score'] = scores

# Compute recipe outputs from inputs
# TODO: Replace this part by your actual code that computes the output, as a Pandas dataframe
# NB: DSS also supports other kinds of APIs for reading and writing data. Please see doc.


# Write recipe outputs

EBSn = dataiku.Dataset("score") # replace by your output dataset
EBSn.write_with_schema(df)

Error:

Job failed: Error in Python process: At line 20: <class 'Exception'>: Saved model NNNN cannot be used : declare it as input or output of your recipe More info about this error

FlorentD

Hi,

if you did not declare your save model as an input, you won't be able to use it.

In any case, you should see your saved model in the input/output tabs (of the code editor of your recipe). If it is not the case, please use the "+Add" button to add your saved model to the recipe.

If your custom python recipe is a plugin recipe, you will also have to add the good input parameter in yous JSON file:

"inputRoles": [
  ...
  {
            "name": ...,
            "label": ...,
            "description": ...,
            "required": ...,
            "arity": ...,
            "acceptsDataset": false,
            "acceptsSavedModel": true
  },
...]

And the in the code retrieve it :

from dataiku.customrecipe import get_input_names_for_role

saved_model_id = get_input_names_for_role(YOUR_INPUT_NAME)[0]

Then you should be able to use it.

Hope this helps.

Best

Jennnnnny

Thank you this problem is solved!

But i got another error:

Job failed: Error in Python process: At line 26: <class 'TypeError'>: 'NoneType' object is not subscriptable More info about this error
View job details

Can you help me with this please?

Jennnnnny

Line 26 :

predictor = model.get_predictor() # get model

FlorentD

Very hard to debug without the code...

But I would suspect, if your code is the "same" that the one you shared before, this line

df_transformed = predictor.preprocess(df)[0]

could be the root of your problem.

You should have more details in the job log, and also do not hesitate to log if you really have something when you run

predictor.preprocess(df)

Jennnnnny

Yes the code is the same as before

FlorentD

So it seems your model is empty. You should check the model Id, if it the good one. You should also pay attention that your model could have been trained with a different version of (let's suppose) scikit-learn than the one you use in your custom recipe, which can lead to incompatibilities.

But I don't get the point where you want to go... If you are trying to re-code an evaluation recipe, you should do rather go to the associated recipe, as replicating the preprocessing manually is a bit of a doomed endeavour (some preprocess are "simple", but many are not)

Jennnnnny

Hello,

The Model ID it's ok so i really don't understand why...

Why i'm doing this, it's because in my previous receipe we don't know how to calculate the silhouette score, so i'm trying to do it with a python code.

Br