Running partitions in parallel
Iqbal
Registered Posts: 1 ✭✭✭
Only 10 paritions run in parallel on large datasets. is there a way tos et for more than 10 paritions to run in parallel.
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,225 DataikerHi @Iqbal
,Concurrent activities are controlled by per job limit and global limit. So likely what you are seeing is a global limit/per job limit of 10.
These can be changed but with caution to ensure enough resources(memory, CPU) are available to sustain this change.
https://doc.dataiku.com/dss/latest/flow/limits.html#global-and-per-job-limits
At the recipe level, they are defined -Parallelism - Concurrent activities
When set to 0 only global limits or Per job limit.
