Monitor users

Solved!
reneege
Level 3
Monitor users

We have just deployed a couple of DSS projects to our automation node, and given Reader access to many customers who will be viewing the dashboards or running webApps produced by those projects.

We are wondering what is the best way to monitor the usages from those users. Is there a way we can get a report on who has logged on and what activities during last week/month without digging into log files?

We are on DSS 6.0.1. Any information would be appreciated!

1 Solution
Clรฉment_Stenac

Hi,

We do not have a reference of the audit log.

However, as a quick start, each JSON record in the audit log looks like:

 

 

{"severity":"INFO","logger":"dku.audit.generic","message":{"callPath":"/api/admin/logs/get-content","msgType":"admin-log-get","authSource":"USER_FROM_UI","clientIP":"127.0.0.1","name":"audit/audit.log","authUser":"admin","originalIP":"90.3.190.212"},"mdc":{"apiCall":"/api/admin/logs/get-content","user":"admin"},"callTime":2,"timestamp":"2020-04-19T04:39:40.340-0400"}

 

 

  • You want to focus the ones where "logger" is "dku.audit.generic".
  • Generally speaking, you can ignore "severity", "callTime", "mdc" and "message.callPath" which are fairly internal details

  • Then, the most important pieces of information are:timestamp, message.authUser and message.msgType

These three items make up the "when", "who" and "what" of the activity. Then, depending on "msgType", you will have additional details within "message".

The granularity of audit event is very fine and each page view will trigger multiple audit events corresponding to the underlying API calls. You'll thus indeed want to focus on some specific audit events.

Most events will at least contain a "projectKey" element in message, and will contain information about the current object. For example, all dataset-related events will contain "datasetName" and so on.

Here are some "interesting" audit events "msgType"

  • application-open: DSS was open in a browser tab
  • login/logout: self-explanatory
  • dataset-read-data-sample: A dataset's Explore was open
  • flow-job-start / flow-job-done: A job was started/completed
  • flow-object-build-start / flow-object-build-failed / flow-object-build-success: Within a job, a dataset was built
  • scenario-run: A scenario was run manually
  • scenario-fire-trigger: A scenario was run automatically

View solution in original post

8 Replies
ATsao
Dataiker

Hi there,

This information will indeed be contained in the audit logs, which can be found in the "run" folder of your DSS data directory. However, instead of parsing them manually, you can always import these files as datasets in DSS and use DSS to perform the analysis you need! A bit meta but I hope that this helps or at least gets you pointed in the right direction!

Best,

Andrew

reneege
Level 3
Author

Thank you for your info!  My understanding that this tool is for monitoring DSS services and system resource usages. Our interest is user logins and activities in a summarized report.

Marlan

Hi Andrew,

We have in fact done this (analyze the most recent audit log by connecting to it in DSS). However, making sense of the audit log has been a challenge. Is there any reference to how to interpret data in the audit logs? We are interested in something similar - logins and key activities (building datasets and running scenarios would be a start). The basic question is who is using DSS, how much are using it, and for what.

Thanks,

Marlan 

Clรฉment_Stenac

Hi,

We do not have a reference of the audit log.

However, as a quick start, each JSON record in the audit log looks like:

 

 

{"severity":"INFO","logger":"dku.audit.generic","message":{"callPath":"/api/admin/logs/get-content","msgType":"admin-log-get","authSource":"USER_FROM_UI","clientIP":"127.0.0.1","name":"audit/audit.log","authUser":"admin","originalIP":"90.3.190.212"},"mdc":{"apiCall":"/api/admin/logs/get-content","user":"admin"},"callTime":2,"timestamp":"2020-04-19T04:39:40.340-0400"}

 

 

  • You want to focus the ones where "logger" is "dku.audit.generic".
  • Generally speaking, you can ignore "severity", "callTime", "mdc" and "message.callPath" which are fairly internal details

  • Then, the most important pieces of information are:timestamp, message.authUser and message.msgType

These three items make up the "when", "who" and "what" of the activity. Then, depending on "msgType", you will have additional details within "message".

The granularity of audit event is very fine and each page view will trigger multiple audit events corresponding to the underlying API calls. You'll thus indeed want to focus on some specific audit events.

Most events will at least contain a "projectKey" element in message, and will contain information about the current object. For example, all dataset-related events will contain "datasetName" and so on.

Here are some "interesting" audit events "msgType"

  • application-open: DSS was open in a browser tab
  • login/logout: self-explanatory
  • dataset-read-data-sample: A dataset's Explore was open
  • flow-job-start / flow-job-done: A job was started/completed
  • flow-object-build-start / flow-object-build-failed / flow-object-build-success: Within a job, a dataset was built
  • scenario-run: A scenario was run manually
  • scenario-fire-trigger: A scenario was run automatically

Hi @Clรฉment_Stenac ,

This is super helpful. I had made some progress in understanding the log but your notes will really in putting together some helpful monitoring. Appreciate you taking the time to write these up. 

Thank you!

Marlan

CoreyS
Dataiker Alumni

We're happy you find this helpful @Marlan !

Looking for more resources to help you use Dataiku effectively and upskill your knowledge? Check out these great resources: Dataiku Academy | Documentation | Knowledge Base

A reply answered your question? Mark as โ€˜Accepted Solutionโ€™ to help others like you!
0 Kudos
reneege
Level 3
Author

Thank you much for this info, which helps us to understand the audit log details better!  Hope Dataiku will make an enhancement in near future to add a summary dashboard/report for user activities per projects under 'Administration -> Maintenance".

Turribeach

I second this request, Dataiku should provide better data on how users are using the app. Audit logs are not really suited for that although they can be used.