Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
The DSS allows to define a recipe type or tag based limits globally https://doc.dataiku.com/dss/latest/flow/limits.html.
However I dont know how to set it per engine, so for example I would like to set a limit to run at most 10 impala queries and 10 spark jobs at the time, in total at most 20 jobs. But if the scenario is building spark only flow, I want to keep the 10 limit, and vice versa if the flow is pure Impala, I want to keep 10 limit for Impala.
Impala - total 10
Spark - total 10
Global - 20 (or unlimited?)
Is it possible to define it in "Additional limits" section?