Copying data from local managed folder to S3 managed folder

Hi @harsha_dataiku
,

Yes, you can do so with the managed folder read/write APIs: https://doc.dataiku.com/dss/latest/connecting/managed_folders.html#usage-in-python

The following code should work for transferring files from a local folder to a remote folder:

input_folder = dataiku.Folder("lE3JuuYn") # replace with folder name or folder ID (retrieved from URL)
input_folder_files = input_folder.list_paths_in_partition()

output_folder = dataiku.Folder("Pxyks4jt")
x = 0
for input_folder_files[x] in input_folder_files:
    with input_folder.get_download_stream(input_folder_files[x]) as f:
        data = f.read()
    output_path = input_folder_files[x].split('/')[-1]
    with output_folder.get_writer(output_path) as w:
        w.write(data)
        print("Successfully transferred {}".format(output_path))
    x += 1

Best,

Jordan