Move File between folders in SFTP

Options
Kyest00
Kyest00 Registered Posts: 8 ✭✭✭

Hi,

I am trying to move a file from one folder to another in a SFTP connection.

I would like to move the file in a DONE folder if the import of the excel was successfully imported in SQL database, and to move to a ERROR folder if the import has errors. But this can be chosen in Scenarios after.

I would like for the file to be deleted from the source folder after it has been moved/copied in the DONE/ERROR folder.

I would also prefer to rename the file with today's date.

Also, can a folder with today's date be created while running the scenarios?

Let me know your thoughts on the best practices.

Thanks!!


Operating system used: Virtual Machine

Answers

  • Sarina
    Sarina Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer Posts: 315 Dataiker
    Options

    Hi @Kyest00
    ,

    You can indeed do this using Managed folders and the Python API for managed folders. If you want, you could create separate managed folders to point to the file paths for "DONE", "ERROR" and "Processing". Or, you could just have a single Managed folder and use the Python API to move the files to the correct paths within the managed folder. Here's an example of using a single Managed Folder that points to your SFTP server and how you could delete, "move" and rename your files with the current date:

    import dataikufrom datetime import datetimeftp = dataiku.Folder("YOUR_FOLDER")ftp_info = ftp.get_info()# add your timestamp formatting heretimestamp = datetime.strftime(datetime.now(), '%Y-%m-%d_%H%M%S')paths = ftp.list_paths_in_partition()# can iterate through all files, or add specific conditions to proceedfor path in paths:with ftp.get_download_stream(path) as f:data = f.read()# <process your data, make sure that you write to postgres successfully># if some success condition, write to your "Done" folderwith ftp.get_writer("/successfully_processed/" + path + "_" + timestamp) as w:​ # write your data to the new "successfully_processed" path, and append the timestampw.write(data)# else if there are errors, add a condition and write to the error path:with ftp.get_writer("/errors/" + path + "_" + timestamp) as w:w.write(data)# finally, delete the original file from the original directory now that the file has been processedftp.delete_path(path)


    I hope that process makes sense. Please let me know if you have any questions about this!

    Thank you,
    Sarina

Setup Info
    Tags
      Help me…