Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Only 10 paritions run in parallel on large datasets. is there a way tos et for more than 10 paritions to run in parallel.
Hi @Iqbal ,
Concurrent activities are controlled by per job limit and global limit. So likely what you are seeing is a global limit/per job limit of 10.
These can be changed but with caution to ensure enough resources(memory, CPU) are available to sustain this change.
https://doc.dataiku.com/dss/latest/flow/limits.html#global-and-per-job-limits
At the recipe level, they are defined -Parallelism - Concurrent activities
When set to 0 only global limits or Per job limit.