How can I override k8s resource requests/limits on a recipe-by-recipe basis?
We have several containerized execution configurations for a variety of workload types. There have been instances where we would like to override the resource limits/requests at a recipe-level. I have so far attempted to set the memory limit via a property (recipe settings → advanced → variables & properties → properties) to kubernetes.memory.limit 1Gi, but that has not been successful. Any suggestions?
Best Answer
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,161 NeuronThis post will show how to change the container configuration on the fly. And while there isn't an official API and it's unsupported we have been creating containerized execution configurations by directly manipulating the general-settings.json config file. This works well for us. In v13 Dataiku introduced a config cache so after making changes to the file you have to invalidate the config cache. So in other words you have all of this working on the fly behind the scenes without having to keep all the containerized execution configurations around.
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,161 Neuron
Why can not you use a different containerized execution configuration for your recipe?
-
We certainly could, but there are cases where we have a recipe that's a one-off and needs a different profile of resources. Our list of configs could grow to be a little unruly if we created a config for each case.