The February release for the Community is live! Read More

Read all the pages of a JSON file

sebastienH
Level 1
Read all the pages of a JSON file

Hello,

I would like to read a JSON file. But with the URL I only access to the first page (the file is paginated). I would like to read ALL the pages.

With Talend I do a loop, but whith JSON I don't know. Can I get help ?

Thank you very much

 

0 Kudos
5 Replies
ATsao
Dataiker
Dataiker

Hi Sebastien,

Could you provide more details about this JSON file and where it's located? Are you using a download recipe or how are you trying to ingest this data into your Flow in DSS? 

Thanks,

Andrew

0 Kudos
sebastienH
Level 1
Author

Hello,

Thank you for your answer.

I get the JSON by URL https://porta....

I can only give the end of the url :

e5e3&query_id=735&offset=0    for the first page

e5e3&query_id=735&offset=1    for the second page

 

etc. There are thousands pages.

I tried a download recipe, but I can’t use « add another source » for each page. Too many pages.

 

Is it possible to do a loop ?

 

Thank you

 

0 Kudos
ATsao
Dataiker
Dataiker

Hi Sebastien,

Your best bet might be to create your own code recipe, such as python, where you could handle iterating through the different pages to read in the necessary data, create your own dataframe, and then write this dataframe as an output into a dataset in DSS. 

Best,

Andrew

0 Kudos
sebastienH
Level 1
Author

Hi Andrew,

Thank you for your answer. I will use a python recipe without input. I thought that an input was necessary, but it's not.

Thank you,

Best,

Sebastien

0 Kudos
ATsao
Dataiker
Dataiker

Hi Sebastien,

That's correct. For code recipes, only an output is necessary (even if it's a dummy output). 

Thanks,

Andrew

0 Kudos
A banner prompting to get Dataiku DSS