Survey banner
Switching to Dataiku - a new area to help users who are transitioning from other tools and diving into Dataiku! CHECK IT OUT

Running partitions in parallel

Level 1
Running partitions in parallel

Only 10 paritions run in parallel on large datasets. is there a way tos et for more than 10 paritions to run in parallel.

0 Kudos
1 Reply

Hi @Iqbal ,

Concurrent activities are controlled by per job limit and global limit. So likely what you are seeing is a global limit/per job limit of 10. 

These can be changed but with caution to ensure enough resources(memory, CPU) are available to sustain this change. 

At the recipe level, they are defined -Parallelism - Concurrent activities 

When set to 0 only global limits or Per job limit. 
Screenshot 2021-10-28 at 14.46.02.png


0 Kudos