Custom preprocessing steps in the Features Handling section in model design

Solved!
cronos003
Level 2
Custom preprocessing steps in the Features Handling section in model design

I'm trying to create a custom transformation but haven't been successful. The sample code provided works fine but when I define my own function with the same transformation it fails. See below for the exact snippets.



Works:




from sklearn import preprocessing
import numpy as np

# Applies log transformation to the feature
processor = preprocessing.FunctionTransformer(np.log1p)






Does not work:




from sklearn import preprocessing
import numpy as np

def CustomT(X):
return np.log1p(X)

# Applies log transformation to the feature
processor = preprocessing.FunctionTransformer(CustomT)


 



Error (more detailed logs available if required):




Failed to train : <type 'exceptions.TypeError'> : expected string or Unicode object, NoneType found


 



I used a pared down dataset to generate this log: https://we.tl/t-YWJABcMO5s

0 Kudos
1 Solution
Alex_Combessie
Dataiker Alumni
Hello, (Repost from our official customer support channel) In a custom preprocessing, you cannot declare a function nor class as those won't be pickled. You need to define it in a library, at either the project level or instance level, see https://doc.dataiku.com/dss/latest/python/reusing-code.html, and then import and use it in the custom preprocessing code. Hope it helps, Alex

View solution in original post

3 Replies
Alex_Combessie
Dataiker Alumni
Hello, (Repost from our official customer support channel) In a custom preprocessing, you cannot declare a function nor class as those won't be pickled. You need to define it in a library, at either the project level or instance level, see https://doc.dataiku.com/dss/latest/python/reusing-code.html, and then import and use it in the custom preprocessing code. Hope it helps, Alex
Boris
Level 2

Hello @Alex_Combessie , 

Is it possible to do a Robustscaler as custom preprocessing ? If yes how possible is it possible to fit_transform the same model to all the features ?  

Thanks in advance

0 Kudos
Alex_Combessie
Dataiker Alumni

Sure, there you go:

Screenshot 2021-03-08 at 16.52.03.png

Note: this custom processor should be applied to each feature.

0 Kudos

Labels

?
Labels (2)
A banner prompting to get Dataiku