How to create subfolders inside a Managed Folder using Python?

tariq

Hi everyone

I’m currently working on a project where I need to organize files inside a Managed Folder in a hierarchical way (for example: mainfolder/subfolder/data.xlsx, mainfolder/REPORTS/report1.csv, etc.).

Is there any supported or alternative way (via the API, SDK, or external libraries) to create real subfolders inside a Managed Folder in Dataiku (All in dataiku enviroment)?

Turribeach

First of all be aware that Dataiku won’t create managed folders in your storage technology until after you put a file on them. Secondly you don’t need to do anything else for subfolders other than specify the path in the file you are saving. Dataiku will create the folders as needed.

folder = dataiku.Folder("Export_Folder")
folder.upload_stream("subfolder/anothersubfolder/dataset1_" + current_date + ".csv", input_dataset1_df.to_csv(index=False))

ThierryC

https://community.dataiku.com/discussion/45311/how-to-create-subfolders-inside-a-managed-folder-using-python

If the technology used as a storage is S3 compatible, it means you will store objects in a flat system and not really files in a filesystem.
S3 allows you to use the name prefix of the blobs as a hierarchy. S3 console or the majority of tools supports the folder concept as a means of grouping objects.
So what you will consider as the path of your file will be used to represent the file hierarchy.