Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
API to create a directory under a managed folder not local
Added on July 2, 2020 10:33AM
Likes: 0
Replies: 3
gnaldi62
Partner, L2 Designer, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Frontrunner 2022 Participant, Neuron 2023 Posts: 79 Neuron
Hi again,
I've a managed folder under HDFS and want to create a subfolder under it but cannot find under the
API documentation the method or the function to do it. How can this be done ? It should be possible
because from the UI that is possible...
Thanks. Regards.
Giuseppe
Best Answer
-
since the notion of folder does not exist for all connection types that you can put managed folders on (as in: it doesn't exist for cloud storages), the is no API to directly create directories inside managed folders. The directories are automatically created as needed when you write a file. For example calling https://doc.dataiku.com/dss/latest/python-api/managed_folders.html#dataiku.Folder.upload_data with `upload_data("/subdir1/subdir2/stuff", [])` will create the sub-directories subdir1 and subdir2 if they don't exist. You could write empty files to force directories to be created.
Answers
-
Hi fchataigner2,
the upload_data() only exists in dataiku package. How can I do it outside of DSS? In the dataikuapi package there is no this method but put_file(). However put_file() doesn't create the subdirectory automatically like upload_data().
-
Here is a function that addresses your need
def make_dir(managed_folder_id, subfolder_rel_path): """ create a subfolder in a folder subfolder_path is relative path such as "subfolder/" or "my/subfolder/path/" the function will create all necessary subfolders if several are nested and not created yet if a folder already exists, the function will not create it, and not delete the content of it """ # Get the folder client = dataiku.api_client() project = client.get_default_project() folder = project.get_managed_folder(managed_folder_id) # Create an empty file called "empty_file.txt" in the current working directory file_path = "empty_file.txt" open(file_path, "w").close() subfolder_rel_path_with_file = os.path.relpath( os.path.join(subfolder_rel_path, "empty_file.txt") ) # Put the file into the DSS managed folder contents = folder.put_file(subfolder_rel_path_with_file, os.path.abspath(file_path)) # delete the file that has been copied to the subfolder folder.delete_file(subfolder_rel_path_with_file)
