R Recipe Streaming

rmoore
Level 3
Level 3
R Recipe Streaming

We're working on a project utilizing an R notebook with some very large datasets and are wondering what the recommended approach is for working with a dataset that does not fit into memory. 

We are big fans of the streaming API for Python - is there any equivalent for R?

Thanks!

0 Kudos
2 Replies
Triveni
Dataiker
Dataiker

Hello,

The dataiku R library allows you to read your data in by chunks. More information can be found in the docs here : https://doc.dataiku.com/dss/api/8.0/R/dataiku/reference/dkuReadDataset.html

0 Kudos
rmoore
Level 3
Level 3
Author

Thanks @Triveni. The ability to read a random sample from the dataset is terrific, but can you tell me if it's possible to read records from a specific offset? For example, I want to iterate through the entire set but only take 10,000 records at a time.

0 Kudos
Labels (3)
A banner prompting to get Dataiku DSS
Public