Copying data from local managed folder to S3 managed folder

harsha_dataiku
harsha_dataiku Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 4

Hi,

I have some model files in a managed folder stored on DSS, I want to copy them to a new folder in S3, is there a way to do it using the python recipe?

Best Answer

  • JordanB
    JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 297 Dataiker
    edited July 17 Answer ✓

    Hi @harsha_dataiku
    ,

    Yes, you can do so with the managed folder read/write APIs: https://doc.dataiku.com/dss/latest/connecting/managed_folders.html#usage-in-python

    The following code should work for transferring files from a local folder to a remote folder:

    input_folder = dataiku.Folder("lE3JuuYn") # replace with folder name or folder ID (retrieved from URL)
    input_folder_files = input_folder.list_paths_in_partition()
    
    output_folder = dataiku.Folder("Pxyks4jt")
    x = 0
    for input_folder_files[x] in input_folder_files:
        with input_folder.get_download_stream(input_folder_files[x]) as f:
            data = f.read()
        output_path = input_folder_files[x].split('/')[-1]
        with output_folder.get_writer(output_path) as w:
            w.write(data)
            print("Successfully transferred {}".format(output_path))
        x += 1

    Best,

    Jordan

Answers

Setup Info
    Tags
      Help me…