Reading all Files from a directory

Appo

Hello !

I'm totally new to dataiku coming from "knime". I'm trying to learn and replicate what i have done on the other software.

I would like to read all the Excel file contained in a folder located in a shared drive, for example in a U:// or G://.

Once everything is read, i simply monitor the file that have added/changed and run through the process.

How can i do it ? I'm trying to use some sort of managed folder but i can't figure it out.

Cheers

Turribeach

Hi, welcome to Dataiku. My advice will be that you don't try replicate exactly what you did on other tools as they might be better ways to do things in Dataiku. For instance in Dataiku you can have a managed folder with files in it and then you can have a dataset change trigger which will start a scenario when the files change. You can do this without any code at all. In terms of creating a managed folder in a network share you need to consider that drive letter mounts are session specific so Dataiku will not see any of your drive mappings. It might possible to use a UNC path and have a managed folder created in a shared folder. However note that Dataiku is not supported under Windows and while Dataiku does allow it to be installed it's for experimentation only. Some features might not work. The correct way to do this will be to have Dataiku running in Linux where you would then "mount" your network drive using the mount command and then you will be able to create managed folders in the mount using Dataiku.

Appo

Hi !

Thanks for the warm welcome and rapid response !

I'm struggling to find where to insert the UNC path in the folder. I can only drag one file at the time.

Turribeach

Dataiku Managed folders get created over Dataiku Connections so the actual physical path is in the Dataiku Connection. I doubt this will work since the Root path of a File System Dataiku Connection is meant to work with Linux paths not UNCs, but you are welcome to try. I don't run Dataiku in Windows so I can't test.

Appo

I might be mistaken but it might be because i'm using the 14 days online free trial and not the Windows version.

Turribeach

The online edition of Dataiku has no way to access any of your network shares. You can however use Cloud Storage buckets although I am not familiar with the limitations of the trial offer the online version does support Cloud Storage:

https://www.dataiku.com/product/plans-and-features/