Load multiple dataset at the same time

ele_f
ele_f Registered Posts: 17 ✭✭✭✭
Hi,

I need to load multiple dataset from my local pc to a dss project.

Is there a way to load al the file together and assign to each of them a different dataset name?

At the moment if multiple files are loaded in dss (and have the same schema) they are union together into a unique file.

This functionality would really streamline the data import process, if available in dss.

Thanks
Tagged:

Answers

  • matthias_funke
    matthias_funke Dataiker Alumni Posts: 12 ✭✭✭✭✭
    edited July 17

    I agree this would be useful feature! Most people solve it using python. You only need a few lines of code, essentially a for loop which reads in the files one by one into a pd.DataFrame(), appends a column with the file name, and finally appends that DataFrame to the output dataframe, which of course you save back to dataiku.


    out = pd.DataFrame(columns=columns)
    for i, data in enumerate(useful_station):
    print i,
    try:
    with open(path+data) as f:
    lines=f.readlines()
    a = [l.replace("\n","").split() for l in lines]
    d = pd.DataFrame(a, columns=columns)
    d[["Month","Day","Hour"]] = d[["Month","Day","Hour"]].astype(int).astype(str)
    d["key"] = data+"-"+d["Month"]+"-"+d["Day"]+"-"+d["Hour"] # new column with file name!
    if i:
    out = pd.concat((out,d))
    else:
    out = d
    except IOError:
    print
    print "********"

    The code sample above was built for something similar, you can use it as inspiration, but not just run it.

  • bcb
    bcb Registered Posts: 2 ✭✭✭✭
    @matthias.funke , after creating the DataFrame with filenames in a new column, how would one create multiple datasets from it?
Setup Info
    Tags
      Help me…