Reading a file from a Managed Folder

pao800a
pao800a Registered Posts: 1
edited July 16 in Using Dataiku

Hi everyone

I am trying to create dataset from a csv file contained in a managed folder. This is something that I did several times without having any particular issue.

This time however, once I have imported the list of paths, when I try to read the csv file, I get error:

FileNotFoundError: [Errno 2] No such file or directory: '/app/dataiku/data_dir/managed_folders/ANNUAL_SHAREHOLDING_STUDY/Wz3QQFf4/FS0000AQGD.csv'

The code I have written is the following.

folder_path = dataiku.Folder("Wz3QQFf4").get_path()
folder_files = dataiku.Folder("Wz3QQFf4").list_paths_in_partition()

path_of_csv = os.path.join(folder_path, folder_files[0][1:]) 
print(path_of_csv)
df = pd.read_csv(path_of_csv)

Thanks a lot

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,226 Dataiker

    @pao800a
    ,
    Is this a locally managed folder or remotely managed folder( S3, GCS, Azure Blob)
    Are you using local execution or containerized?

    In both of the above cases using os.path.join will not work and you should use the read/write API get_download_stream to read the file instead.

    or you can use +Dataset - Internal Files in Folder dataset type.

    Thanks

Setup Info
    Tags
      Help me…