Survey banner
The Dataiku Community is moving to a new home! Some short term disruption starting next week: LEARN MORE

Create a new dataset by stacking files in a folder

Solved!
e_pap
Level 2
Create a new dataset by stacking files in a folder

Hi team,

I am connecting with sharepoint folder to access a couple of files and I was wondering if there is a way to create a vertically-stacked dataset from all the files in the folder.

Thank you

0 Kudos
1 Solution
AlexT
Dataiker

Assuming the files in the folder have different schemas, you must use a code recipe.

If they all have the same schema, you can just use Files in Folder dataset. If you only have a few files you can use multiple and use a Visual Stack recipe


If you have a lot of files to read all files in a Folder and stack them into a data frame you can use something like below, but you need to decide on how you stack / concat these and how you consolidate the schema.


Screenshot 2023-05-18 at 1.02.19 PM.png

View solution in original post

2 Replies
AlexT
Dataiker

Assuming the files in the folder have different schemas, you must use a code recipe.

If they all have the same schema, you can just use Files in Folder dataset. If you only have a few files you can use multiple and use a Visual Stack recipe


If you have a lot of files to read all files in a Folder and stack them into a data frame you can use something like below, but you need to decide on how you stack / concat these and how you consolidate the schema.


Screenshot 2023-05-18 at 1.02.19 PM.png

e_pap
Level 2
Author

Thank you @AlexT !

0 Kudos