Dss running Out of memory too fast

Bap
Bap Dataiku DSS Core Designer, Registered Posts: 1

Hello !

I'm currently working on a DSS instance, having fun building data piplines, but it's running out of memory after 2hrs.

DSS is installed manually on a docker image. We tryed several setup with 'xmx' configuration but we ends-up with many idle processes (~20 jek) that filled our allocated memory. The number of jek is always increasing with no idea how to configure cache management.

Config :

- DSS v12.2.2
- install on a docker image : docker python:3.9-slim-buster
- backend.xmx=12g & jek.xwx = 4g

Usage:

- Single user on a single project (~50 datasets and ~30 recipes)


Operating system used: debian


Operating system used: debian

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,212 Dataiker

    Hi @Bap
    ,
    It is hard to advise based on this information alone.

    How much memory have you allocated to the DSS container itself?
    What behavior are you seeing when, say it run "out of memory"?

    The jek 4g is more the default value. You should look to reduce this back to 2g.
    The Jek processes remain open as DSS reuses them, but this should not cause issues or consume a high level of resources.

    You need to check the number of concurrent executions allowed and the total memory.
    The default is you should leave this at the default value unless you have adequate sized instance.
    .
    Note we recommend using groups to limit the usage of java processes, but this will not work in the container. Hence why installing DSS in the container is not recommended:https://doc.dataiku.com/dss/latest/installation/other/container.html

    https://doc.dataiku.com/dss/latest/operations/cgroups.html

    Please try to capture the instance diagnostics while the issue is occurring if possible.

    https://doc.dataiku.com/dss/latest/troubleshooting/diagnosing.html



    I would suggest you capture an instance diagnostics and submit a support to understand what is causing the memory usage.

Setup Info
    Tags
      Help me…