Dataiku monitoring

dtk-explorer

hello community !

I hope this message finds you well.

I am currently working on a project that involves integrating Dataiku DSS (Data Science Studio) with Azure's monitoring tools, specifically Azure Monitor and Azure Log Analytics Workspace. Our goal is to ensure comprehensive monitoring and logging for our Dataiku environment to enhance our observability and troubleshooting capabilities.

I would greatly appreciate your insights and recommendations on the following points:

1. Methods for Extracting/Push Logs from Dataiku to Azure:

   • What are the best practices for extracting logs from Dataiku DSS?
   • Are there any specific tools or plugins that facilitate the integration with Azure Monitor or Azure Log Analytics Workspace?
   • How can we automate the process of pushing these logs to Azure?

2. Types of Logs for Monitoring Dataiku DSS:

   • What types of logs are most useful for monitoring the performance and health of Dataiku DSS?
   • Are there specific log categories (e.g., application logs, system logs, user activity logs) that we should focus on?
   • How can we leverage these logs to proactively identify and resolve issues within our DSS environment?

Any guidance, documentation, or examples you can provide would be incredibly helpful. If anyone has experience with a similar setup or can point me toward relevant resources or tutorials, I would be very grateful.

Thank you in advance !

Turribeach

There are no specific integrations with those Azure tools that I am aware of. For an overall guidance on logging and monitoring you should read this reply I wrote last year:

https://community.dataiku.com/t5/Using-Dataiku/Tracking-User-Activity/m-p/38319/highlight/true#M14204

The post should cover pretty much all your points. There is also an excellent and very complete Dataiku Python API which provides many methods that can be used for monotoring. I suggest you start by reviewing all the documentation and deciding what you want to monitor and what logs you want to load. Then start new threads with more specific questions if you have any as this post has too many questions and it's too wide to recieve answers to each of your points.