Limiting per project jobs for all projects
Hi,
Is it possible to limit the number of jobs to say 5 for each project on a DSS instance? The documentation indicates I can do it for specific projects but doesn't make it clear if it's possible for all projects.
I'd like to set the global limit to 20 and limit every project so no project exceeds 5 jobs. It's too cumbersome to list out every project and would need to be updated as users create new projects.
Best Answer
-
Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 NeuronOptionsHi @sylvyr3
I think what you are looking for is in Administration > Settings > Flow build:
What can be limited is not directly the jobs, but the number of activities that can run simultaneously: if each job you run has one activity, then with a limit of 5 only 5 jobs will run concurrently; if some jobs have 2 or more activities, the number of jobs running will be less.
From here you can set more limits using keys and values: documentation is at https://doc.dataiku.com/dss/latest/flow/limits.html
Hope this helps!
Answers
-
@Ignacio_Toledo
, but can you limit all project to 5 concurrent activities like @sylvyr3
asks?
I mean for a specific single project you can put in "Additional Limits"- Limit key: project/projectkey1
- Limit Value: 5
But I think @sylvyr3
(and me too) wants a way to say "every project get a limit of 5". To me it seems that wildcards like "project/*" or "user/*" are not supported.
I submitted a project idea Limiting concurrent executions (activities) / default limit for all users / projects https://community.dataiku.com/t5/Product-Ideas/Limiting-concurrent-executions-activities-default-limit-for-all/idc-p/41269#M1643 -
Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 NeuronOptions
I guess you are right @ecerulm
, it looks like I misunderstood the original question, but since it was marked as "solved" I never went back to it.Creating a project idea for this was the right path!
