How to get metrics (jmx, prometheus, etc) for Dataiku DSS ?
From what I can get from the documentation at
The documentation does not mention what metrics are actually exported either, so for me it's hard to tell if it's even worth it to go all the trouble of setting something up that talks the graphite protocol and send to my already existing monitoring solution on ELK.
Anybody knows what metrics are actually exported beside cpu and vm usage (which I suppose are there)?
Operating system used: Amazon Linux 2
Answers
-
In my DSS instance there are 8252 distict metrics so I can't list all of them here but here is a list of the ones I found more interesting to track in my monitoring
dss.UUID.server.dku.objectsCounts.global.datasets 2324 1731601973 dss.UUID.server.dku.objectsCounts.global.recipes 1994 1731601973 dss.UUID.server.dku.scenarios.activeTriggers.global 4 1731601973 dss.UUID.server.dku.collaboration.activeUsers 2 1731601973 dss.UUID.server.dku.collaboration.connectedUsers 2 1731601973 dss.UUID.server.dku.collaboration.activeTrackingSessions 3 1731601973 dss.UUID.server.dku.collaboration.trackingSessions 3 1731601973 dss.UUID.server.dku.jobs.activities.possiblyRunning 0 1731601973 dss.UUID.server.dku.jobs.activities.queued 0 1731601973 dss.UUID.server.dku.jobs.activities.waitingForPermit 0 1731601973 dss.UUID.server.jvm.memory.heap.committed 1251999744 1731601973 dss.UUID.server.jvm.memory.heap.init 1042284544 1731601973 dss.UUID.server.jvm.memory.heap.max 8589934592 1731601973 dss.UUID.server.jvm.memory.heap.usage 0.11 1731601973 dss.UUID.server.jvm.memory.heap.used 906292560 1731601973 dss.UUID.server.jvm.memory.non-heap.committed 256507904 1731601973 dss.UUID.server.jvm.memory.non-heap.init 7667712 1731601973 dss.UUID.server.jvm.memory.non-heap.max -1 1731601973 dss.UUID.server.jvm.memory.non-heap.usage -245757056.00 1731601973 dss.UUID.server.jvm.memory.non-heap.used 245757056 1731601973 dss.UUID.server.dku.notifications.websocket.send.activeThreads 0 1731601973 dss.UUID.server.dku.notifications.websocket.send.pooledThreads 2 1731601973 dss.UUID.server.jvm.threads.blocked.count 0 1731601973 dss.UUID.server.jvm.threads.count 111 1731601973 dss.UUID.server.jvm.threads.daemon.count 38 1731601973 dss.UUID.server.jvm.threads.deadlock.count 0 1731601973 dss.UUID.server.jvm.threads.new.count 0 1731601973 dss.UUID.server.jvm.threads.runnable.count 19 1731601973 dss.UUID.server.jvm.threads.terminated.count 0 1731601973 dss.UUID.server.jvm.threads.timed_waiting.count 64 1731601973 dss.UUID.server.jvm.threads.waiting.count 28 1731601973 dss.UUID.server.dku.scenarios.triggerRuns.global.max 5.68 1731601973 dss.UUID.server.dku.scenarios.triggerRuns.global.mean 1.54 1731601973 dss.UUID.server.dku.scenarios.triggerRuns.global.min 1.35 1731601973 dss.UUID.server.dku.scenarios.triggerRuns.global.stddev 0.15 1731601973
so far I think I can configure metricbeat graphite module
to receive these metrics and send it to elasticsearch. -
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,123 Neuron
What port is this running on? Do you need a login for JMX?
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,123 Neuron
OK I got something working using Jolokia. As their Github page says Jolokia is a fresh way to access JMX MBeans remotely using REST. I downloaded the JVM-Agent from the downloads page and then used the second option of the JVM Agent to attach it to the com.dataiku.dip.DSSBackendMain process ID (9015 in my case) using OpenJDK 20 for MacOS (I used openjdk-20.0.2_macos-aarch64_bin.tar.gz as I am on Apple silicon):
[…]/jdk-20.0.2.jdk/Contents/Home/bin/java -jar ./jolokia-agent-jvm-2.1.1-javaagent.jar start --port 7778 9015
After doing that I could see the Jolokia HTTP server up on http://127.0.0.1:7778/jolokia/
I then used this sample curl query to pull the one of the metrics:
curl -s http://127.0.0.1:7778/jolokia/read/metrics:name=dku.jobs.running/Count | jq .
And here is the response:
{ "request": { "mbean": "metrics:name=dku.jobs.running", "attribute": "Count", "type": "read" }, "value": 1, "status": 200, "timestamp": 1731710330 }
To see what's available you can launch […]/jdk-20.0.2.jdk/Contents/Home/bin/jconsole. Then you attach it to the com.dataiku.dip.DSSBackendMain process:
Then select the MBeans tab, exapand the metrics and you should see everything that's available. Some of the metrics don't show up initially so may need to have some activity first for them to show:
-
What I did is to use the Administration > Other > Misc > Graphite Reporting
- Server: localhost:9109
- Frequency: 1
if you start a
nc -l -p 9109
you will see that Dataiku sends the metrics via tcp to that port.nc -l -p 9109
dss.xxxxx.server.dku.objectsCounts.global.datasets 2324 1731601973 dss.xxxxx.server.dku.objectsCounts.global.recipes 1994 1731601973 dss.xxxxx.server.dku.scenarios.activeTriggers.global 4 1731601973 …The I used metricbeat graphite module to receive the metrics, filter them (I'm only interested on 20 metrics of the +8000 available), then I send those to eleasticsearch.
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,123 Neuron
Thank you!