Create a new dataset by stacking files in a folder

e_pap
e_pap Registered Posts: 11

Hi team,

I am connecting with sharepoint folder to access a couple of files and I was wondering if there is a way to create a vertically-stacked dataset from all the files in the folder.

Thank you

Best Answer

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,226 Dataiker
    Answer ✓

    Assuming the files in the folder have different schemas, you must use a code recipe.

    If they all have the same schema, you can just use Files in Folder dataset. If you only have a few files you can use multiple and use a Visual Stack recipe


    If you have a lot of files to read all files in a Folder and stack them into a data frame you can use something like below, but you need to decide on how you stack / concat these and how you consolidate the schema.


    Screenshot 2023-05-18 at 1.02.19 PM.png

Answers

Setup Info
    Tags
      Help me…