Survey banner
The Dataiku Community is moving to a new home! Some short term disruption starting next week: LEARN MORE

Automate Cgroup Limits - Resource Control

Solved!
mohdwaqar
Level 1
Automate Cgroup Limits - Resource Control

Hi Team, 

I have setup cgroup limits (Path Template and limits ) in resource control on one of our instance and I want to replicate this to all our existing instance and the new installations.

Is there a way to automate this ?

Thanks in advance!!


Operating system used: Centos 7.9, RHEL 8, ROcky Linux 8.8

0 Kudos
1 Solution
VincentRD
Dataiker

Hello,

For such automation tasks, I suggest using the Python API. You would use it to manipulate the general settings and enter your cgroup-specific configuration.

You can find more information on doing this in the developer guide . It's the get_general_settings  entry point you are interested in.

It might not be obvious from the doc but the general scheme to update an instance setting is the following (here for flow limits):

# Update the Flow limits
client = dataikuapi.DSSClient(dku_url, api_key)
settings_obj = client.get_general_settings()
settings = settings_obj.get_raw()
settings['maxRunningActivities'] = new_limit
settings_obj.save()

 For cgroups, it's a nested dictionary under the general settings 'cgroupSettings' key. You'll have to experiment a bit to set it to what you want. I suggest setting the right value in the UI and then using the Python code to dump the configuration before trying to use Python to set it programmatically.

HTH

View solution in original post

3 Replies
Turribeach

To automate Dataiku deployment and configuration you can use Cloud Stacks:

https://knowledge.dataiku.com/latest/admin-deploying/cloud-stacks/index.html

If you are not on Cloud Stacks you have to do it yourself. 

mohdwaqar
Level 1
Author

Thanks for the response!

I am not using cloud, Is configuring cgroups limits manually the only

0 Kudos
VincentRD
Dataiker

Hello,

For such automation tasks, I suggest using the Python API. You would use it to manipulate the general settings and enter your cgroup-specific configuration.

You can find more information on doing this in the developer guide . It's the get_general_settings  entry point you are interested in.

It might not be obvious from the doc but the general scheme to update an instance setting is the following (here for flow limits):

# Update the Flow limits
client = dataikuapi.DSSClient(dku_url, api_key)
settings_obj = client.get_general_settings()
settings = settings_obj.get_raw()
settings['maxRunningActivities'] = new_limit
settings_obj.save()

 For cgroups, it's a nested dictionary under the general settings 'cgroupSettings' key. You'll have to experiment a bit to set it to what you want. I suggest setting the right value in the UI and then using the Python code to dump the configuration before trying to use Python to set it programmatically.

HTH