Community Conundrum 25: Feature Visualization is now live! Read More

Load multiple dataset at the same time

Level 2
Load multiple dataset at the same time
Hi,

I need to load multiple dataset from my local pc to a dss project.

Is there a way to load al the file together and assign to each of them a different dataset name?

At the moment if multiple files are loaded in dss (and have the same schema) they are union together into a unique file.

This functionality would really streamline the data import process, if available in dss.

Thanks
0 Kudos
2 Replies
Dataiker
Dataiker

I agree this would be useful feature! Most people solve it using python. You only need a few lines of code, essentially a for loop which reads in the files one by one into a pd.DataFrame(), appends a column with the file name, and finally appends that DataFrame to the output dataframe, which of course you save back to dataiku.




out = pd.DataFrame(columns=columns)
for i, data in enumerate(useful_station):
print i,
try:
with open(path+data) as f:
lines=f.readlines()
a = [l.replace("\n","").split() for l in lines]
d = pd.DataFrame(a, columns=columns)
d[["Month","Day","Hour"]] = d[["Month","Day","Hour"]].astype(int).astype(str)
d["key"] = data+"-"+d["Month"]+"-"+d["Day"]+"-"+d["Hour"] # new column with file name!
if i:
out = pd.concat((out,d))
else:
out = d
except IOError:
print
print "********"


The code sample above was built for something similar, you can use it as inspiration, but not just run it.

0 Kudos
Level 1
@matthias.funke , after creating the DataFrame with filenames in a new column, how would one create multiple datasets from it?
0 Kudos
Labels (1)