Survey banner
The Dataiku Community is moving to a new home! Some short term disruption starting next week: LEARN MORE

API to create a directory under a managed folder not local

Solved!
gnaldi62
API to create a directory under a managed folder not local

Hi again, 

  I've a managed folder under HDFS and want to create a subfolder under it but cannot find under the

  API documentation the method or the function to do it. How can this be done ? It should be possible

  because from the UI that is possible...

Thanks. Regards.

  Giuseppe 

0 Kudos
1 Solution
fchataigner2
Dataiker

since the notion of folder does not exist for all connection types that you can put managed folders on (as in: it doesn't exist for cloud storages), the is no API to directly create directories inside managed folders. The directories are automatically created as needed when you write a file. For example calling https://doc.dataiku.com/dss/latest/python-api/managed_folders.html#dataiku.Folder.upload_data with `upload_data("/subdir1/subdir2/stuff", [])` will create the sub-directories subdir1 and subdir2 if they don't exist. You could write empty files to force directories to be created.

View solution in original post

3 Replies
fchataigner2
Dataiker

since the notion of folder does not exist for all connection types that you can put managed folders on (as in: it doesn't exist for cloud storages), the is no API to directly create directories inside managed folders. The directories are automatically created as needed when you write a file. For example calling https://doc.dataiku.com/dss/latest/python-api/managed_folders.html#dataiku.Folder.upload_data with `upload_data("/subdir1/subdir2/stuff", [])` will create the sub-directories subdir1 and subdir2 if they don't exist. You could write empty files to force directories to be created.

daida
Level 2

Hi fchataigner2,

the upload_data() only exists in dataiku package. How can I do it outside of DSS? In the dataikuapi package there is no this method but put_file(). However put_file() doesn't create the subdirectory automatically like upload_data().

Damien13
Level 1

Here is a function that addresses your need

def make_dir(managed_folder_id, subfolder_rel_path):
    """
    create a subfolder in a folder
    subfolder_path is relative path such as "subfolder/"
        or "my/subfolder/path/"
    the function will create all necessary subfolders if several are nested and not created yet
    if a folder already exists, the function will not create it, and not delete the content of it
    """
    # Get the folder
    client = dataiku.api_client()
    project = client.get_default_project()
    folder = project.get_managed_folder(managed_folder_id)

    # Create an empty file called "empty_file.txt" in the current working directory
    file_path = "empty_file.txt"
    open(file_path, "w").close()

    subfolder_rel_path_with_file = os.path.relpath(
        os.path.join(subfolder_rel_path, "empty_file.txt")
    )

    # Put the file into the DSS managed folder
    contents = folder.put_file(subfolder_rel_path_with_file, os.path.abspath(file_path))

    # delete the file that has been copied to the subfolder
    folder.delete_file(subfolder_rel_path_with_file)
0 Kudos