Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
When Dataiku execute processes in memory, what resources are used? :
1.-RAM on the server
2.- or CPU resources
Is there a way to know these metrics?
Hi @IsaacChavez ,
You can check the CRU logs in job/backend/audit logs :
Reporting completion of CRU:{"context":{"type":"JOB_MAIN_PROCESS","authIdentifier":"dhidalgo","projectKey":"MYPROJECT","jobId":"Scenario_build_preprocessed"},"type":"LOCAL_PROCESS","id":"cfmcpwHMsxNqoGje","startTime":1707136651006,"localProcess":{"pid":2948620,"commandName":"java","cpuUserTimeMS":10850,"cpuSystemTimeMS":700,"cpuChildrenUserTimeMS":4210,"cpuChildrenSystemTimeMS":3190,"cpuTotalMS":18950,"cpuCurrent":0.11118832522585129,"cpuAverageOverPast60Seconds":0.0,"vmSizeMB":8312,"vmRSSMB":790,"vmHWMMB":790,"vmRSSAnonMB":742,"vmDataMB":2475,"vmSizePeakMB":8376,"vmRSSPeakMB":790,"vmRSSTotalMBS":32015,"majorFaults":0,"childrenMajorFaults":0}}
Note these are available in Audit logs and there is a business solution on how to leverage them:
https://knowledge.dataiku.com/latest/solutions/governance/solution-cru.html
What each of things actually mean:
- cpuCurrent is the index of the CPU used ;
- pid: id of the process ;
Memory:
- vmSizeMB is related to vmsize (but in MB), the amount of virtual memory used by the process ;
- vmSizePeakMB is vmPeak (peak virtual memory size), in MB - the peak of of virtual memory ;
- vmDataMB is related to vmData (but in MB). It is, roughly, the size of the memory heap of the process ;
- vmRSS is the use of physical RAM. So, vmRSSPeakMB is the peak use of such memory and vmRSSMB is the current physical RAM used (in MB) ;
- vmHWMMB relates to VmHWM (with a factor 1024, as it is in MB while the original metric is in KB) - the peak resident set size (HWM stands for High Water Mark) ;
- vmRSSAnonMB is related to rssanon, in MB. It is the used physical RAM that is no backed by a mapped file ;
CPU:
- cpuUserTimeMS is related to utome (with a 1000 / 100 Hz factor) and is the amount of time the process was scheduled in user mode ;
- cpuSystemTimeMS is related to stime (with a 1000 / 100 Hz factor) and is the amount of time the process was scheduled in kernel mode ;
- cpuTotalMS is the sum of utime+stime+cutime+stime (with a 1000 / 100 Hz factor) and is the amount of time the process was scheduled in user or kernel mode, or waited for a child in user or kernel mod ;
- cpuChildrenUserTimeMS is related to cutime (with a 1000 / 100 Hz factor) and is the amount of time the process waited for a child process scheduled in user mode ;
- cpuChildrenSystemTimeMS is related to cstime (with a 1000 / 100 Hz factor) and is the amount of time the process waited for a child process scheduled in kernel mode ;
Kind Regards,