Number of Rows

[sefinew]

Hello,

I am using Dataiku Dss for managing for the organization I am working now. The idea is I am collecting above 230000 records (rows) by using Kobo Form. I use API to import the data from Kobo to Dataiku. Currently, I have above 50,000 records on Kobo but am able to get only 30K rows on Dataiku.

Is there any way to get all rows in real time?

[AlexT]

Hi @sefinew ,

The 30k row limit is Kobo API limitation. You will need to use pagination to retrieve your data in batches of up to 30k rows.
https://mixedanalytics.com/knowledge-base/import-kobotoolbox-data-to-google-sheets/#pagination

[GabrielMutua]

Thank you, this has become so handy for me today.

[GabrielMutua]

This solution has saved me big time.

[tgb417 · Accepted solution]

@sefinew 

Welcome to the Dataiku community.

I was interested in your question here because I do a bunch of work with REST APIs as data sources.

I just wanted to share that I have had good results with using the API connect plug-in from Dataiku. It can be setup to do pagination, which sounds like it may be helpful in your use case.  With an enterprise license to Dataiku you can also set up Scenarios that can wake up intermittently and pull data from your source.  Here is the documentation:

https://doc.dataiku.com/dss/latest/scenarios/index.html

Here is some training material on Scenarios,

https://knowledge.dataiku.com/latest/mlops-o16n/automation/concept-scenarios.html 

[sefinew]

Thank you so much this helped a lot