Dss running Out of memory too fast

[Bap]

Hello !

I'm currently working on a DSS instance, having fun building data piplines, but it's running out of memory after 2hrs.

DSS is installed manually on a docker image. We tryed several setup with 'xmx' configuration but we ends-up with many idle processes (~20 jek) that filled our allocated memory. The number of jek is always increasing with no idea how to configure cache management.

Config :

- DSS v12.2.2
- install on a docker image : docker python:3.9-slim-buster
- backend.xmx=12g & jek.xwx = 4g

Usage:

- Single user on a single project (~50 datasets and ~30 recipes)

---

Operating system used: debian

---

Operating system used: debian

[AlexT]

Hi @Bap ,
It is hard to advise based on this information alone.

How much memory have you allocated to the DSS container itself?
What behavior are you seeing when, say it run "out of memory"? 

The jek 4g is more the default value. You should look to reduce this back to 2g.
The Jek processes remain open as DSS reuses them, but this should not cause issues or consume a high level of resources.

You need to check the number of concurrent executions allowed and the total memory.
The default is you should leave this at the default value unless you have adequate sized instance.
.
Note we recommend using groups to limit the usage of java processes, but this will not work in the container. Hence why installing DSS in the container is not recommended:https://doc.dataiku.com/dss/latest/installation/other/container.html

https://doc.dataiku.com/dss/latest/operations/cgroups.html

Please try to capture the instance diagnostics while the issue is occurring if possible.

https://doc.dataiku.com/dss/latest/troubleshooting/diagnosing.html



I would suggest you capture an instance diagnostics and submit a support to understand what is causing the memory usage.