Enhance Managed Folders APIs to be able to handle local-vs-non-local folders automatically

Options
Turribeach
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,727 Neuron

Dataiku supports creating Managed Folders over different storage layers including local storage, network storage, cloud storage (ie buckets) or even Sharepoint. However the way you deal with these folders depends on where the API client is located (inside or outside DSS) and where the storage is:

https://doc.dataiku.com/dss/latest/connecting/managed_folders.html#local-vs-non-local

This additional complexity leads to a lot of confusion in the Dataiku user community because you have to use different API methods to deal with a Dataiku Managed Folder depending on where the folder is located.

This Product Idea is to enhance the Managed Folders APIs so that the same set of methods can be used across all possible storage layers and API client location (inside or outside DSS), having the API itself decide what's the best way to read and write to the folder in the most appropriate and performant way.

0
0 votes

New · Last Updated

Comments

  • apichery
    apichery Dataiker, Alpha Tester, Product Ideas Manager Posts: 62 Dataiker
    Options

    Hi !

    Using list_paths_in_partition, get_download_stream and get_writer should give you everything to work efficiently with both local and non-local folders.

Setup Info
    Tags
      Help me…