ZS Associates is a management consulting and professional services firm focusing on consulting, software, and technology, headquartered in Evanston, Illinois that provides services for clients in healthcare, private equity, and technology.
Loggings should be maintained in a tabular format for easy readability of every job run.
Created audit logging framework using Python and SQL to capture the detailed Job status for every dataset build. This framework is incorporated with Dataiku scenarios and executed through intermediate code steps.
Business Area Enhanced: Risk, Compliance, Legal, Internal Audit.
Use Case Stage: In Production
This has enabled easy history maintenance for job runs with the detailed logs.
This also captures the run time for every job execution which can be leveraged in future integration of job lineage dashboards.
This has created a Logging table in Snowflake which can be queried easily using SQL and get the required information for the job run.
Value Brought by Dataiku:
Dataiku provides ease of integration with Snowflake, which has been used to store the logging details in the Snowflake table.
Dataiku provides the code reusability functionality using Dataiku Libraries which helped us in implementing the same logging framework with multiple jobs.