DataIku Performance Pandas
RishuSharma
Registered Posts: 1 ✭
Hi,
We want to run multiple DQ SQL's against hive database and iterate across the results in python pandas.
Do we have any specific limit of rows that we pandas can handle?
Answers
-
Emma Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 52 DataikerHey @RishuSharma
,Pandas has no specific row limit. However you risk running out of memory if your datasets are too large.
You can read more from the pandas documentation (Scaling to large datasets) if you're interested.
I hope that helps,
Emma