Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
I am trying to move a file from one folder to another in a SFTP connection.
I would like to move the file in a DONE folder if the import of the excel was successfully imported in SQL database, and to move to a ERROR folder if the import has errors. But this can be chosen in Scenarios after.
I would like for the file to be deleted from the source folder after it has been moved/copied in the DONE/ERROR folder.
I would also prefer to rename the file with today's date.
Also, can a folder with today's date be created while running the scenarios?
Let me know your thoughts on the best practices.
Operating system used: Virtual Machine
You can indeed do this using Managed folders and the Python API for managed folders. If you want, you could create separate managed folders to point to the file paths for "DONE", "ERROR" and "Processing". Or, you could just have a single Managed folder and use the Python API to move the files to the correct paths within the managed folder. Here's an example of using a single Managed Folder that points to your SFTP server and how you could delete, "move" and rename your files with the current date:
import dataiku from datetime import datetime ftp = dataiku.Folder("YOUR_FOLDER") ftp_info = ftp.get_info() # add your timestamp formatting here timestamp = datetime.strftime(datetime.now(), '%Y-%m-%d_%H%M%S') paths = ftp.list_paths_in_partition() # can iterate through all files, or add specific conditions to proceed for path in paths: with ftp.get_download_stream(path) as f: data = f.read() # <process your data, make sure that you write to postgres successfully> # if some success condition, write to your "Done" folder with ftp.get_writer("/successfully_processed/" + path + "_" + timestamp) as w: # write your data to the new "successfully_processed" path, and append the timestamp w.write(data) # else if there are errors, add a condition and write to the error path: with ftp.get_writer("/errors/" + path + "_" + timestamp) as w: w.write(data) # finally, delete the original file from the original directory now that the file has been processed ftp.delete_path(path)
I hope that process makes sense. Please let me know if you have any questions about this!