Right now I'm developing a Dataiku recipe to save parquet format file into Dataiku folder.
First I need to call other service to get a dataframe and transform the dataframe into parquet format. But after running the recipe, the parquet file size is always 0KB with nothing inside. I used folder.upload_stream() provided by Dataiku. I have verified that there is no problem with dataframe
import io f = io.BytesIO() df.to_parquet(f) folder.upload_stream("name of file.parquet", f)
I don't know how to fix this problem, does someone have the same issue?
Thank you for your help
For this alternative way, we need to specify the path and generate that parquet file in that path. And the method itself returns a None type
I don't know if will work with dataiku. Even it works it will create extra file somewhere.
Do you know other ways to solve the problem?
Thanks a lot