Move File between folders in SFTP

Hi @Kyest00
,

You can indeed do this using Managed folders and the Python API for managed folders. If you want, you could create separate managed folders to point to the file paths for "DONE", "ERROR" and "Processing". Or, you could just have a single Managed folder and use the Python API to move the files to the correct paths within the managed folder. Here's an example of using a single Managed Folder that points to your SFTP server and how you could delete, "move" and rename your files with the current date:

import dataiku
from datetime import datetime

ftp = dataiku.Folder("YOUR_FOLDER")
ftp_info = ftp.get_info()

# add your timestamp formatting here
timestamp = datetime.strftime(datetime.now(), '%Y-%m-%d_%H%M%S')
paths = ftp.list_paths_in_partition()

# can iterate through all files, or add specific conditions to proceed
for path in paths:
    with ftp.get_download_stream(path) as f:
        data = f.read()
        # <process your data, make sure that you write to postgres successfully>
        # if some success condition, write to your "Done" folder
        with ftp.get_writer("/successfully_processed/" + path + "_" + timestamp) as w:
​            # write your data to the new "successfully_processed" path, and append the timestamp
            w.write(data)
        # else if there are errors, add a condition and write to the error path: 
        with ftp.get_writer("/errors/" + path + "_" + timestamp) as w:
            w.write(data)
        # finally, delete the original file from the original directory now that the file has been processed
        ftp.delete_path(path)

I hope that process makes sense. Please let me know if you have any questions about this!

Thank you,
Sarina