I have two folders both connected to the same S3 bucket but not the same directory.
I want to copy a subset of first folder in the second folder with a recipe (python if possible).
I've already tried to do this with :
f = Folder_A.get_download_stream(filename) and Folder_B.upload_stream(file_copy, f)
But it's veeeery slow (5 min to copy a 20mo file)
Is there a better method to copy a file from bucket to bucket ?
Thank you !!
The get_download_stream and upload_stream method imply data streaming in-and-out of S3 onto the DSS server.
You can achieve higher efficiency by using a cloud-specific API: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client