copy files from bucket to bucket

[mass84]

Hi !!

I have two folders both connected to the same S3 bucket but not the same directory.

I want to copy a subset of first folder in the second folder with a recipe (python if possible). 

I've already tried to do this with :

f = Folder_A.get_download_stream(filename) and Folder_B.upload_stream(file_copy, f)

But it's veeeery slow (5 min to copy a 20mo file)

Is there a better  method to copy a file from bucket to bucket ?

Thank you !!

[Alex_Combessie · Accepted solution]

Hi,

The get_download_stream and upload_stream method imply data streaming in-and-out of S3 onto the DSS server.

You can achieve higher efficiency by using a cloud-specific API: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client

Cheers,

Alex