Memory usage per scenario

mklaber

As a follow up to https://community.dataiku.com/t5/Setup-Configuration/Feature-request-Monitor-memory-usage-per-scenario/m-p/1823#M322, it sure seems like this has been implemented via https://doc.dataiku.com/dss/latest/operations/monitoring.html#configure-dss-to-push-metrics, which I've done. I'm now seeing this in job logs:

{
    "context": {
        "type": "JOB_ACTIVITY",
        "authIdentifier": "ADataScientistUser",
        "projectKey": "MY_PROJECT",
        "jobId": "Build_shap_values_unpivoted_2021-01-14T09-22-01.850",
        "activityId": "compute_shap_values_unpivoted_NP",
        "activityType": "recipe",
        "recipeType": "python",
        "recipeName": "compute_shap_values_unpivoted"
    },
    "type": "LOCAL_PROCESS",
    "id": "JwY8E2ga2tS4V42W",
    "startTime": 1610616122774,
    "localProcess": {
        "pid": 67709,
        "commandName": "/mnt/secondary/dataiku_data/bin/python",
        "cpuUserTimeMS": 654790,
        "cpuSystemTimeMS": 64300,
        "cpuChildrenUserTimeMS": 0,
        "cpuChildrenSystemTimeMS": 0,
        "cpuTotalMS": 719090,
        "cpuCurrent": 0.9978004399120176,
        "vmSizeMB": 82066,
        "vmRSSMB": 80657,
        "vmHWMMB": 80657,
        "vmRSSAnonMB": 80623,
        "vmDataMB": 80694,
        "vmSizePeakMB": 82066,
        "vmRSSPeakMB": 80657,
        "vmRSSTotalMBS": 42283649,
        "majorFaults": 18,
        "childrenMajorFaults": 0
    }
}

Question is: where are these details going? I can't seem to find any references to these processes or recipes or activities or whatnot in my Carbon (via dkumonitor) server. If it is being pushed to Carbon/statsd, what's the prefix and/or search query? If it's pushed elsewhere, where might that be?

MehdiH

Hi @mklaber

Those are two separate things:

• The metrics collected by dkumonitor are immediate values that can be graphed over time.
• The log that you attached is the Compute resource usage reporting for a given job. It is reported to the DSS audit log with the compute-resource-usage topic. For more details looks at this documentation page