Discover this year's submissions to the Dataiku Frontrunner Awards and give kudos to your favorite use cases and success stories!READ MORE

Custom preprocessing steps in the Features Handling section in model design

Solved!
cronos003
Level 2
Custom preprocessing steps in the Features Handling section in model design

I'm trying to create a custom transformation but haven't been successful. The sample code provided works fine but when I define my own function with the same transformation it fails. See below for the exact snippets.



Works:




from sklearn import preprocessing
import numpy as np

# Applies log transformation to the feature
processor = preprocessing.FunctionTransformer(np.log1p)






Does not work:




from sklearn import preprocessing
import numpy as np

def CustomT(X):
return np.log1p(X)

# Applies log transformation to the feature
processor = preprocessing.FunctionTransformer(CustomT)


 



Error (more detailed logs available if required):




Failed to train : <type 'exceptions.TypeError'> : expected string or Unicode object, NoneType found


 



I used a pared down dataset to generate this log: https://we.tl/t-YWJABcMO5s

0 Kudos
1 Solution
Alex_Combessie
Dataiker Alumni
Hello, (Repost from our official customer support channel) In a custom preprocessing, you cannot declare a function nor class as those won't be pickled. You need to define it in a library, at either the project level or instance level, see https://doc.dataiku.com/dss/latest/python/reusing-code.html, and then import and use it in the custom preprocessing code. Hope it helps, Alex

View solution in original post

3 Replies
Alex_Combessie
Dataiker Alumni
Hello, (Repost from our official customer support channel) In a custom preprocessing, you cannot declare a function nor class as those won't be pickled. You need to define it in a library, at either the project level or instance level, see https://doc.dataiku.com/dss/latest/python/reusing-code.html, and then import and use it in the custom preprocessing code. Hope it helps, Alex
Boris
Level 2

Hello @Alex_Combessie , 

Is it possible to do a Robustscaler as custom preprocessing ? If yes how possible is it possible to fit_transform the same model to all the features ?  

Thanks in advance

0 Kudos
Alex_Combessie
Dataiker Alumni

Sure, there you go:

Screenshot 2021-03-08 at 16.52.03.png

Note: this custom processor should be applied to each feature.

0 Kudos

Labels

?
Labels (2)
A banner prompting to get Dataiku