Move File between folders in SFTP

Kyest00

Hi,

I am trying to move a file from one folder to another in a SFTP connection.

I would like to move the file in a DONE folder if the import of the excel was successfully imported in SQL database, and to move to a ERROR folder if the import has errors. But this can be chosen in Scenarios after.

I would like for the file to be deleted from the source folder after it has been moved/copied in the DONE/ERROR folder.

I would also prefer to rename the file with today's date.

Also, can a folder with today's date be created while running the scenarios?

Let me know your thoughts on the best practices.

Thanks!!

---

Operating system used: Virtual Machine

Sarina

Hi @Kyest00
,

You can indeed do this using Managed folders and the Python API for managed folders. If you want, you could create separate managed folders to point to the file paths for "DONE", "ERROR" and "Processing". Or, you could just have a single Managed folder and use the Python API to move the files to the correct paths within the managed folder. Here's an example of using a single Managed Folder that points to your SFTP server and how you could delete, "move" and rename your files with the current date:

import dataiku
from datetime import datetime

ftp = dataiku.Folder("YOUR_FOLDER")
ftp_info = ftp.get_info()

# add your timestamp formatting here
timestamp = datetime.strftime(datetime.now(), '%Y-%m-%d_%H%M%S')
paths = ftp.list_paths_in_partition()

# can iterate through all files, or add specific conditions to proceed
for path in paths:
    with ftp.get_download_stream(path) as f:
        data = f.read()
        # <process your data, make sure that you write to postgres successfully>
        # if some success condition, write to your "Done" folder
        with ftp.get_writer("/successfully_processed/" + path + "_" + timestamp) as w:
​            # write your data to the new "successfully_processed" path, and append the timestamp
            w.write(data)
        # else if there are errors, add a condition and write to the error path: 
        with ftp.get_writer("/errors/" + path + "_" + timestamp) as w:
            w.write(data)
        # finally, delete the original file from the original directory now that the file has been processed
        ftp.delete_path(path)

I hope that process makes sense. Please let me know if you have any questions about this!

Thank you,
Sarina