Reading a file from a Managed Folder

pao800a
Level 1
Reading a file from a Managed Folder

Hi everyone ๐Ÿ™‚

I am trying to create dataset from a csv file contained in a managed folder. This is something that I did several times without having any particular issue.

This time however, once I have imported the list of paths, when I try to read the csv file, I get error:

FileNotFoundError: [Errno 2] No such file or directory: '/app/dataiku/data_dir/managed_folders/ANNUAL_SHAREHOLDING_STUDY/Wz3QQFf4/FS0000AQGD.csv'

 The code I have written is the following.

 

folder_path = dataiku.Folder("Wz3QQFf4").get_path()
folder_files = dataiku.Folder("Wz3QQFf4").list_paths_in_partition()

path_of_csv = os.path.join(folder_path, folder_files[0][1:]) 
print(path_of_csv)
df = pd.read_csv(path_of_csv)

 

Thanks a lot ๐Ÿ™‚ 

0 Kudos
1 Reply
AlexT
Dataiker

@pao800a ,
Is this a locally managed folder or remotely managed folder( S3, GCS, Azure Blob)
Are you using local execution or containerized?

In both of the above cases using os.path.join will not work  and you should use the read/write API get_download_stream to read the file instead.

or you can use +Dataset - Internal Files in Folder dataset type.

Thanks

0 Kudos