This website uses cookies. By clicking OK, you consent to the use of cookies. Click Here to learn more about how we use cookies.
OK
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Community Conundrum 27: Stacks of Questions is live! Read More

How to load a large HTTP dataset in DSS ?

UserBird
Dataiker
Dataiker
‎07-06-2017 02:43 AM
How to load a large HTTP dataset in DSS ?
Hi,

I actually work on Dataiku build on a cluster.

I want to load an HDFS dataset from an HTTP file (Zip contains csv). I use without problem the Network dataset interface on DSS. Problem is when I want to load a large csv file on HTTP (8Go). Dataiku can't detect it and returns me the Preview of the json.

It is possible to load directly a large files in DSS or I need to make a Recipes for this big file ?

Another solution ?
0 Kudos
Reply
2 Replies
cperdigou
Dataiker
Dataiker
‎07-10-2017 06:13 PM
I'm sorry I'm not sure I understand exactly your problem.

What do you call a HTTP file?

Is it a file that is currently on a remote server? In that case you can connect to it to create a dataset, then sync it to HDFS.

Is it a file that currently is on your local computer? In that case, I guess because of its size the upload does not work and you should try scp ing the file directly on the server, then creating a filesystem dataset pointing to that file.
0 Kudos
Reply
Helka
Level 1
In response to cperdigou
‎07-10-2017 07:08 PM
Hi cperdigou,

My csv file is stocked online in a remote server (into a zip file : http://files.data.gouv.fr/sirene/). When I use the Network dataset interface on DSS I have no problem connecting into a smaller csv (100Mo). But with a 8Go file I can't connect to it to create a dataset cause it's too big.

Dataiku is installed on a VM with a cluster Hadoop. Maybe can I use the Hadoop powerfull to load this big datafile into Dataiku ?
0 Kudos
Reply
Labels (1)
Labels