Check out the first Dataiku 8 Deep Dive focusing on Productivity on October 29th Read More

using an EKS backend, can I suppress the "Reporting update of resource usage" status messages?

Level 2
using an EKS backend, can I suppress the "Reporting update of resource usage" status messages?

These get reported every minute and on a long job (>1 hour) means it is quite hard to see my own print/status messages? can I adjust the frequency - e.g. every 5 mins? or see these in a dashboard/graph somewhere? is there any documentation on what all the values mean? 

Here is an example:

[07:03:33] [INFO] [dku.usage.computeresource.jek]  - Reporting update of resource usage: {"context":{"type":"JOB_ACTIVITY","authIdentifier":"108008065","projectKey":"GE90_ENGINE_REMOVAL_SIMULATION","jobId":"Build_engine_removal_predictions_2020-10-05T11-01-01.315","activityId":"compute_engine_removal_predictions_NP","activityType":"recipe","recipeType":"pyspark","recipeName":"compute_engine_removal_predictions"},"type":"LOCAL_PROCESS","id":"JkqpIjflMvXKEFZu","startTime":1601895693822,"localProcess":{"pid":22157,"commandName":"/u01/dataiku/dataiku-dss-8.0.2/spark-standalone-home/bin/spark-submit","cpuUserTimeMS":29880,"cpuSystemTimeMS":2070,"cpuChildrenUserTimeMS":230,"cpuChildrenSystemTimeMS":160,"cpuTotalMS":32340,"cpuCurrent":0.01199760047990402,"vmSizeMB":19579,"vmRSSMB":1058,"vmHWMMB":1058,"vmRSSAnonMB":998,"vmDataMB":19366,"vmSizePeakMB":19579,"vmRSSPeakMB":1058,"vmRSSTotalMBS":85920,"majorFaults":0,"childrenMajorFaults":0}}
[07:03:33] [DEBUG] [dku.resource]  - Process stats for pid 22157: {"pid":22157,"commandName":"/u01/dataiku/dataiku-dss-8.0.2/spark-standalone-home/bin/spark-submit","cpuUserTimeMS":29880,"cpuSystemTimeMS":2070,"cpuChildrenUserTimeMS":230,"cpuChildrenSystemTimeMS":160,"cpuTotalMS":32340,"cpuCurrent":0.01199760047990402,"vmSizeMB":19579,"vmRSSMB":1058,"vmHWMMB":1058,"vmRSSAnonMB":998,"vmDataMB":19366,"vmSizePeakMB":19579,"vmRSSPeakMB":1058,"vmRSSTotalMBS":85920,"majorFaults":0,"childrenMajorFaults":0}
[07:04:33] [INFO] [dku.usage.computeresource.jek]  - Reporting update of resource usage: {"context":{"type":"JOB_ACTIVITY","authIdentifier":"108008065","projectKey":"GE90_ENGINE_REMOVAL_SIMULATION","jobId":"Build_engine_removal_predictions_2020-10-05T11-01-01.315","activityId":"compute_engine_removal_predictions_NP","activityType":"recipe","recipeType":"pyspark","recipeName":"compute_engine_removal_predictions"},"type":"LOCAL_PROCESS","id":"JkqpIjflMvXKEFZu","startTime":1601895693822,"localProcess":{"pid":22157,"commandName":"/u01/dataiku/dataiku-dss-8.0.2/spark-standalone-home/bin/spark-submit","cpuUserTimeMS":30650,"cpuSystemTimeMS":2110,"cpuChildrenUserTimeMS":230,"cpuChildrenSystemTimeMS":160,"cpuTotalMS":33150,"cpuCurrent":0.001999600079984003,"vmSizeMB":19712,"vmRSSMB":1060,"vmHWMMB":1060,"vmRSSAnonMB":1000,"vmDataMB":19499,"vmSizePeakMB":19712,"vmRSSPeakMB":1060,"vmRSSTotalMBS":149465,"majorFaults":0,"childrenMajorFaults":0}}
[07:04:33] [DEBUG] [dku.resource]  - Process stats for pid 22157: {"pid":22157,"commandName":"/u01/dataiku/dataiku-dss-8.0.2/spark-standalone-home/bin/spark-submit","cpuUserTimeMS":30650,"cpuSystemTimeMS":2110,"cpuChildrenUserTimeMS":230,"cpuChildrenSystemTimeMS":160,"cpuTotalMS":33150,"cpuCurrent":0.001999600079984003,"vmSizeMB":19712,"vmRSSMB":1060,"vmHWMMB":1060,"vmRSSAnonMB":1000,"vmDataMB":19499,"vmSizePeakMB":19712,"vmRSSPeakMB":1060,"vmRSSTotalMBS":149465,"majorFaults":0,"childrenMajorFaults":0}

 

0 Kudos
0 Replies