DataIku Performance Pandas

RishuSharma
Level 1
DataIku Performance Pandas

Hi,

We want to run multiple DQ SQL's against hive database and iterate across the results in python pandas. 

Do we have any specific limit of rows that we pandas can handle?

 

 

0 Kudos
1 Reply
Emma
Dataiker

Hey @RishuSharma , 

Pandas has no specific row limit. However you risk running out of memory if your datasets are too large. 

You can read more from the pandas documentation (Scaling to large datasets) if you're interested. 

I hope that helps, 

Emma 

0 Kudos