Enhance Managed Folders APIs to be able to handle local-vs-non-local folders automatically
Dataiku supports creating Managed Folders over different storage layers including local storage, network storage, cloud storage (ie buckets) or even Sharepoint. However the way you deal with these folders depends on where the API client is located (inside or outside DSS) and where the storage is:
https://doc.dataiku.com/dss/latest/connecting/managed_folders.html#local-vs-non-local
This additional complexity leads to a lot of confusion in the Dataiku user community because you have to use different API methods to deal with a Dataiku Managed Folder depending on where the folder is located.
This Product Idea is to enhance the Managed Folders APIs so that the same set of methods can be used across all possible storage layers and API client location (inside or outside DSS), having the API itself decide what's the best way to read and write to the folder in the most appropriate and performant way.
Comments
-
Hi !
Using list_paths_in_partition, get_download_stream and get_writer should give you everything to work efficiently with both local and non-local folders.
