Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi @AlexT ,
I am getting out of memory issue while joining two dataframes using python recipe, one is having 5M records and other is having 44k records.
The Python process died (killed - maybe out of memory
Doing large joins pandas is not usually recommended. The best approach here would be to use an SQL database and a Visual Join recipe in DSS and leverage SQL engine to perform joins.
Sync the datasets to SQL datasets and then perform the join. If you must perform this in Pandas you will need to ensure you have enough memory to perform this either:
1) You may need to adjust cgroups and/or add more RAM to your instance
2) You may need to use a larger container configuration ( more memory)