Custom preprocessing steps in the Features Handling section in model design
cronos003
Registered Posts: 7 ✭✭✭✭
I'm trying to create a custom transformation but haven't been successful. The sample code provided works fine but when I define my own function with the same transformation it fails. See below for the exact snippets.
Works:
from sklearn import preprocessing
import numpy as np
# Applies log transformation to the feature
processor = preprocessing.FunctionTransformer(np.log1p)
Does not work:
from sklearn import preprocessing
import numpy as np
def CustomT(X):
return np.log1p(X)
# Applies log transformation to the feature
processor = preprocessing.FunctionTransformer(CustomT)
Error (more detailed logs available if required):
Failed to train : <type 'exceptions.TypeError'> : expected string or Unicode object, NoneType found
I used a pared down dataset to generate this log: https://we.tl/t-YWJABcMO5s
Tagged:
Best Answer
-
Hello, (Repost from our official customer support channel) In a custom preprocessing, you cannot declare a function nor class as those won't be pickled. You need to define it in a library, at either the project level or instance level, see https://doc.dataiku.com/dss/latest/python/reusing-code.html, and then import and use it in the custom preprocessing code. Hope it helps, Alex
Answers
-
Hello @Alex_Combessie
,Is it possible to do a Robustscaler as custom preprocessing ? If yes how possible is it possible to fit_transform the same model to all the features ?
Thanks in advance
-
Sure, there you go:
Note: this custom processor should be applied to each feature.
