Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Out of Memory Issue in Python Recipe.
Added on April 21, 2022 11:35AM
Likes: 0
Replies: 1
Ankur30
Partner, Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Dataiku DSS Adv Designer Posts: 40 Partner
Hi @AlexT
,
I am getting out of memory issue while joining two dataframes using python recipe, one is having 5M records and other is having 44k records.
Error message:-
The Python process died (killed - maybe out of memory
Kindly Help.
Regards,
Ankur.
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,270 DataikerHi @Ankur30
,Doing large joins pandas is not usually recommended. The best approach here would be to use an SQL database and a Visual Join recipe in DSS and leverage SQL engine to perform joins.
Sync the datasets to SQL datasets and then perform the join. If you must perform this in Pandas you will need to ensure you have enough memory to perform this either:
1) You may need to adjust cgroups and/or add more RAM to your instance
2) You may need to use a larger container configuration ( more memory)