Community Conundrum 25: Feature Visualization is now live! Read More

Manage folder partition in python recipes

Level 1
Manage folder partition in python recipes

I have a partitionned folder and I need to python recipe to create the associated partitionned dataset.
In the notebook, I can use the following code :


files = dataiku.Folder("xxxx")
files_info = files.get_info()

#chemin des fichiers

for itemName in paths:

with files.get_download_stream(itemName) as j:

df_measure=df_measure.append(sub_df, ignore_index=True)

# Write recipe outputs
measurement = dataiku.Dataset("Measurement")


But, when back in the recipe, the partitions are managed by DSS. 
I also need to remove files.list_paths_in_partition() and for itemName in paths.


How can I load the right file in files.get_download_stream(itemName) ???


Thanks a lot

Best regards

0 Kudos
1 Reply

In the actions of your partitioned folder, you can pick the "create dataset" one. This will create a dataset which is merely a view of the files in the folder. You can then activate partitioning on this dataset.

0 Kudos