Using swap memory on Design Node

tgb417

I'm working with a small DSS Design Node with 16GB RAM memory on AWS.

The models I'm trying to build are running out of Available Memory and crashing. I can cut the size of the sample I'm working with. However, to get this to work, I'm using only 1/3 of 1% (0.3333%) of the data in my model-building process. (data set size ranges from 300,000 to 1,000,000 records) With this model, I can build incremental models adding a different sample of records each time I run the model. However, it is clear to me that this is not typical best practice.

So I need some more working memory. I'm working with a non-profit so budget is a consideration.

Has anyone used swap memory for data science using a Python Model? Does anyone have any good news stories or horror stories where swap did and did not work?

Thanks for any insights you can share.

Sadig_Mammadov

Hey, You can use "chunksize" option for python recipes and so split into pieces your dataframe during the processing and optimize memory usage. for example:

for df in source_data.iter_dataframes(chunksize=3000000):

https://doc.dataiku.com/dss/latest/python-api/datasets-data.html