RAM and CPU

Isaac Emmanuel

When Dataiku execute processes in memory, what resources are used? :

1.-RAM on the server

2.- or CPU resources

Is there a way to know these metrics?

Alexandru

Hi @IsaacChavez
,

You can check the CRU logs in job/backend/audit logs :

Reporting completion of CRU:{"context":{"type":"JOB_MAIN_PROCESS","authIdentifier":"dhidalgo","projectKey":"MYPROJECT","jobId":"Scenario_build_preprocessed"},"type":"LOCAL_PROCESS","id":"cfmcpwHMsxNqoGje","startTime":1707136651006,"localProcess":{"pid":2948620,"commandName":"java","cpuUserTimeMS":10850,"cpuSystemTimeMS":700,"cpuChildrenUserTimeMS":4210,"cpuChildrenSystemTimeMS":3190,"cpuTotalMS":18950,"cpuCurrent":0.11118832522585129,"cpuAverageOverPast60Seconds":0.0,"vmSizeMB":8312,"vmRSSMB":790,"vmHWMMB":790,"vmRSSAnonMB":742,"vmDataMB":2475,"vmSizePeakMB":8376,"vmRSSPeakMB":790,"vmRSSTotalMBS":32015,"majorFaults":0,"childrenMajorFaults":0}}


Note these are available in Audit logs and there is a business solution on how to leverage them:
https://knowledge.dataiku.com/latest/solutions/governance/solution-cru.html

What each of things actually mean:

- cpuCurrent is the index of the CPU used ;
- pid: id of the process ;

Memory:
- vmSizeMB is related to vmsize (but in MB), the amount of virtual memory used by the process ;
- vmSizePeakMB is vmPeak (peak virtual memory size), in MB - the peak of of virtual memory ;
- vmDataMB is related to vmData (but in MB). It is, roughly, the size of the memory heap of the process ;
- vmRSS is the use of physical RAM. So, vmRSSPeakMB is the peak use of such memory and vmRSSMB is the current physical RAM used (in MB) ;
- vmHWMMB relates to VmHWM (with a factor 1024, as it is in MB while the original metric is in KB) - the peak resident set size (HWM stands for High Water Mark) ;
- vmRSSAnonMB is related to rssanon, in MB. It is the used physical RAM that is no backed by a mapped file ;

CPU:
- cpuUserTimeMS is related to utome (with a 1000 / 100 Hz factor) and is the amount of time the process was scheduled in user mode ;
- cpuSystemTimeMS is related to stime (with a 1000 / 100 Hz factor) and is the amount of time the process was scheduled in kernel mode ;
- cpuTotalMS is the sum of utime+stime+cutime+stime (with a 1000 / 100 Hz factor) and is the amount of time the process was scheduled in user or kernel mode, or waited for a child in user or kernel mod ;
- cpuChildrenUserTimeMS is related to cutime (with a 1000 / 100 Hz factor) and is the amount of time the process waited for a child process scheduled in user mode ;
- cpuChildrenSystemTimeMS is related to cstime (with a 1000 / 100 Hz factor) and is the amount of time the process waited for a child process scheduled in kernel mode ;

Kind Regards,