How to loop on multiple files contained in a folder ?

Lo96 · March 6

i have multiple xlsx files contained in a folder . How can i loop on them one by one to concatenate them in a single dataframe ?

Dataiku version used: 13.1

Lo96 · March 6

also i want to automate the whole process. each time a new xslx file is updated , run the flow that prepares the data push it to sql table. Then, for visualization the idea is to build a Dash app that fetchs data from SQL table and plot it. Any ideas to put this together in Dataiku ? maybe, there is a more simple way to do ?

many thanks

Turribeach · March 6

Are all the XLS files of the same format? If so you can use a Dataiku Managed folder and then use a Files in Folder dataset. To trigger the refresh you can use a scenario with a dataset change trigger which can point to the Managed Folder.