ZS Associates - Audit Logging Framework to Capture Detailed Job Status for Every Dataset Build

sonal_18

Name:

Ranjan Relan, Strategy & Architecture Manager
Sonal Singh, Business Technology Associate Consultant

Title: Associate Consultant

Country: India

Organization: ZS Associates

ZS Associates is a management consulting and professional services firm focusing on consulting, software, and technology, headquartered in Evanston, Illinois that provides services for clients in healthcare, private equity, and technology.

Awards Categories:

• Partner Acceleration

Business Challenge:

Loggings should be maintained in a tabular format for easy readability of every job run.

Business Solution:

Created audit logging framework using Python and SQL to capture the detailed Job status for every dataset build. This framework is incorporated with Dataiku scenarios and executed through intermediate code steps.

Business Area Enhanced: Risk, Compliance, Legal, Internal Audit.

Use Case Stage: In Production

Value Generated:

• This has enabled easy history maintenance for job runs with the detailed logs.
• This also captures the run time for every job execution which can be leveraged in future integration of job lineage dashboards.
• This has created a Logging table in Snowflake which can be queried easily using SQL and get the required information for the job run.

Value Brought by Dataiku:

• Dataiku provides ease of integration with Snowflake, which has been used to store the logging details in the Snowflake table.
• Dataiku provides the code reusability functionality using Dataiku Libraries which helped us in implementing the same logging framework with multiple jobs.

Value Type:

• Improve customer/employee satisfaction
• Reduce risk
• Save time
• Increase trust