Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
With a Python recipe, I am trying to upload a CSV to a managed folder, partitioned by year-month-day :
I have tried everything that I can think of, and each time I get this error :
Invalid partitioning: in running compute_AAAAAAA_2022-05-30: Partitioning scheme is not representable as folders
At the moment, my script looks like this :
# Write recipe outputs output_folder = dataiku.Folder("AAAAAAAA") filepath = (datetime.datetime.now() - datetime.timedelta(days = 1)).strftime('%Y-%m-%d') + "/" filename = 'DAILY_' + (datetime.datetime.now() - datetime.timedelta(days = 1)).strftime('%Y-%m-%d') + '.csv' df.to_csv(filename, encoding='utf-8') output_folder.put_file(filepath, filename)
Any help would be greatly appreciated, as I've been banging my head against the wall for quite some time now on his 😅
Operating system used: MacOS
Some questions before trying to help you:
With that clear, I think I can give you some ideas.
It is indeed a python recipe with no input (I am using Paramiko to pull data from an SFTP server).
Regarding the pattern, both choices are OK for me. The use of the partitioned folder would be to pull the last file from it, using max(folder.list_partitions())
Thank you for your time and help!
Given your use case, I think you don't need a partitioned folder in principle: if you write all your files with the format 'DAILY_%Y-%m-%d.csv' in the root of the output_folder, you can easily use another python function to get the latest file written:
output_folder = dataiku.Folder("AAAAAAAA") import glob import os list_of_files = glob.glob(output_folder.get_path() + '/*.csv') latest_file = max(list_of_files, key=os.path.getctime) print(latest_file)
I think this is the simplest option. However, learning to use partitioned folders and datasets is useful, so if you are interested in the alternative solution, let me know and I can share another example
Thanks for this solution, works perfectly.
Indeed, I would be very happy to learn how to do this with partitioned folders! I will need those in the future, and knowing how they work would be great.
Again, thanks for your time and help.