ZS Associates - Audit Logging Framework to Capture Detailed Job Status for Every Dataset Build

Name:

Ranjan Relan, Strategy & Architecture Manager
Sonal Singh, Business Technology Associate Consultant

Title: Associate Consultant

Country: India

Organization: ZS Associates

ZS Associates is a management consulting and professional services firm focusing on consulting, software, and technology, headquartered in Evanston, Illinois that provides services for clients in healthcare, private equity, and technology.

Awards Categories:

  • Partner Acceleration

 

Business Challenge:

Loggings should be maintained in a tabular format for easy readability of every job run.

 

Business Solution:

Created audit logging framework using Python and SQL to capture the detailed Job status for every dataset build. This framework is incorporated with Dataiku scenarios and executed through intermediate code steps.

 

Business Area Enhanced: Risk, Compliance, Legal, Internal Audit. 

Use Case Stage: In Production

 

Value Generated:

  • This has enabled easy history maintenance for job runs with the detailed logs. 
  • This also captures the run time for every job execution which can be leveraged in future integration of job lineage dashboards.
  • This has created a Logging table in Snowflake which can be queried easily using SQL and get the required information for the job run.

 

Value Brought by Dataiku:

  • Dataiku provides ease of integration with Snowflake, which has been used to store the logging details in the Snowflake table.
  • Dataiku provides the code reusability functionality using Dataiku Libraries which helped us in implementing the same logging framework with multiple jobs.

Value Type:

  • Improve customer/employee satisfaction
  • Reduce risk
  • Save time
  • Increase trust
Share:
Version history
Publication date:
07-09-2022 10:11 AM
Version history
Last update:
‎09-07-2022 12:11 PM
Updated by: