Community Conundrums are live! Learn more

Custom preprocessing steps in the Features Handling section in model design

Level 2
Custom preprocessing steps in the Features Handling section in model design

I'm trying to create a custom transformation but haven't been successful. The sample code provided works fine but when I define my own function with the same transformation it fails. See below for the exact snippets.



Works:




from sklearn import preprocessing
import numpy as np

# Applies log transformation to the feature
processor = preprocessing.FunctionTransformer(np.log1p)






Does not work:




from sklearn import preprocessing
import numpy as np

def CustomT(X):
return np.log1p(X)

# Applies log transformation to the feature
processor = preprocessing.FunctionTransformer(CustomT)


 



Error (more detailed logs available if required):




Failed to train : <type 'exceptions.TypeError'> : expected string or Unicode object, NoneType found


 



I used a pared down dataset to generate this log: https://we.tl/t-YWJABcMO5s

0 Kudos
1 Reply
Dataiker
Dataiker
Hello, (Repost from our official customer support channel) In a custom preprocessing, you cannot declare a function nor class as those won't be pickled. You need to define it in a library, at either the project level or instance level, see https://doc.dataiku.com/dss/latest/python/reusing-code.html, and then import and use it in the custom preprocessing code. Hope it helps, Alex
Labels (2)