RAM and CPU

Isaac Emmanuel
Isaac Emmanuel Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 16 ✭✭✭

When Dataiku execute processes in memory, what resources are used? :

1.-RAM on the server

2.- or CPU resources

Is there a way to know these metrics?

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,226 Dataiker

    Hi @IsaacChavez
    ,

    You can check the CRU logs in job/backend/audit logs :

    Reporting completion of CRU:{"context":{"type":"JOB_MAIN_PROCESS","authIdentifier":"dhidalgo","projectKey":"MYPROJECT","jobId":"Scenario_build_preprocessed"},"type":"LOCAL_PROCESS","id":"cfmcpwHMsxNqoGje","startTime":1707136651006,"localProcess":{"pid":2948620,"commandName":"java","cpuUserTimeMS":10850,"cpuSystemTimeMS":700,"cpuChildrenUserTimeMS":4210,"cpuChildrenSystemTimeMS":3190,"cpuTotalMS":18950,"cpuCurrent":0.11118832522585129,"cpuAverageOverPast60Seconds":0.0,"vmSizeMB":8312,"vmRSSMB":790,"vmHWMMB":790,"vmRSSAnonMB":742,"vmDataMB":2475,"vmSizePeakMB":8376,"vmRSSPeakMB":790,"vmRSSTotalMBS":32015,"majorFaults":0,"childrenMajorFaults":0}}


    Note these are available in Audit logs and there is a business solution on how to leverage them:
    https://knowledge.dataiku.com/latest/solutions/governance/solution-cru.html

    What each of things actually mean:

    - cpuCurrent is the index of the CPU used ;
    - pid: id of the process ;

    Memory:
    - vmSizeMB is related to vmsize (but in MB), the amount of virtual memory used by the process ;
    - vmSizePeakMB is vmPeak (peak virtual memory size), in MB - the peak of of virtual memory ;
    - vmDataMB is related to vmData (but in MB). It is, roughly, the size of the memory heap of the process ;
    - vmRSS is the use of physical RAM. So, vmRSSPeakMB is the peak use of such memory and vmRSSMB is the current physical RAM used (in MB) ;
    - vmHWMMB relates to VmHWM (with a factor 1024, as it is in MB while the original metric is in KB) - the peak resident set size (HWM stands for High Water Mark) ;
    - vmRSSAnonMB is related to rssanon, in MB. It is the used physical RAM that is no backed by a mapped file ;

    CPU:
    - cpuUserTimeMS is related to utome (with a 1000 / 100 Hz factor) and is the amount of time the process was scheduled in user mode ;
    - cpuSystemTimeMS is related to stime (with a 1000 / 100 Hz factor) and is the amount of time the process was scheduled in kernel mode ;
    - cpuTotalMS is the sum of utime+stime+cutime+stime (with a 1000 / 100 Hz factor) and is the amount of time the process was scheduled in user or kernel mode, or waited for a child in user or kernel mod ;
    - cpuChildrenUserTimeMS is related to cutime (with a 1000 / 100 Hz factor) and is the amount of time the process waited for a child process scheduled in user mode ;
    - cpuChildrenSystemTimeMS is related to cstime (with a 1000 / 100 Hz factor) and is the amount of time the process waited for a child process scheduled in kernel mode ;

    Kind Regards,

Setup Info
    Tags
      Help me…