Submit your inspiring success story or innovative use case to the 2022 Dataiku Frontrunner Awards! ENTER YOUR SUBMISSION

How to work python recipe with function get_dataframe

Grixis6
Level 2
How to work python recipe with function get_dataframe

I use a python recipe to do a drop_duplicates on columns and I see that the output variables are altered.

It seems to be specific to pandas which transforms the integer with NA into a float.

OK

So it comes that the get_dataframe function would have an argument to add in the function to overcome this and lock my variable:

infer_with_pandas=False

 

And in the case of a missing value it crashes because NA does not respect the format.

log :

Job failed: Error in Python process: At line 11: <class 'ValueError'>: Integer column has NA values in column 6

0 Kudos
1 Reply
Grixis6
Level 2
Author

I think I have found an alternative

Embed my python code in a PySpark recipe so as not to have the transformation due by pandas

But I dont know how its work : maybe it's pandas is not using RAM for operation and dont need to convert anymore ?

0 Kudos

Labels

?
Labels (1)
A banner prompting to get Dataiku