Enhance Managed Folders APIs to be able to handle local-vs-non-local folders automatically

0 Kudos

Dataiku supports creating Managed Folders over different storage layers including local storage, network storage, cloud storage (ie buckets) or even Sharepoint. However the way you deal with these folders depends on where the API client is located (inside or outside DSS) and where the storage is:

https://doc.dataiku.com/dss/latest/connecting/managed_folders.html#local-vs-non-local

This additional complexity leads to a lot of confusion in the Dataiku user community because you have to use different API methods to deal with a Dataiku Managed Folder depending on where the folder is located.

This Product Idea is to enhance the Managed Folders APIs so that the same set of methods can be used across all possible storage layers and API client location (inside or outside DSS), having the API itself decide what's the best way to read and write to the folder in the most appropriate and performant way.

1 Comment
apichery
Dataiker

Hi !

Using list_paths_in_partition, get_download_stream and get_writer should give you everything to work efficiently with both local and non-local folders.

Hi !

Using list_paths_in_partition, get_download_stream and get_writer should give you everything to work efficiently with both local and non-local folders.