Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Writing Pyspark dataframe to partitioned dataset
Added on April 25, 2024 5:01PM
Likes: 0
Replies: 1
Gipple
Registered Posts: 11 ✭✭✭
I am trying to write the output of a pyspark recipe to a partitioned dataset, but I am receiving an error.
Py4JJavaError: An error occurred while calling o261.savePyDataFrame. : com.dataiku.common.server.APIError$APIErrorException: Illegal time partitioning value : 'null' with a MONTH partitioning
This is how I am trying to write it
# Write recipe outputs CCW_Output_Windows = dataiku.Dataset("CCW_Output_Windows") dkuspark.write_with_schema(CCW_Output_Windows, CCW_Output_df)
I haven't been able to find any documentation on this. Thanks
Answers
-
JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 297 DataikerHi @Gipple
,It appears that the recipe is finding a null where it expects a month date value. Please check your data as well as how you are selecting the partitions to build from:
Kind Regards,
Jordan