Recommend Way for Comparing Metrics between Multiple Flow Datasets in a Check

Hi,

To achieve this I believe you will need to create a custom check using python. You can retrieve metric values for various datasets in your project using the dataikuapi. Below is a starter code (for example to use in a notebook or recipe) that shows how to use the API to compare metrics values between a source and target datasets:

import dataiku
from dataiku import pandasutils as pdu
import pandas as pd
import dataikuapi

##Context
client = dataiku.api_client()
proj = client.get_project(dataiku.default_project_key())
source_dataset = 'tweets_scored_aggregate'
target_dataset = 'tweets_scored_aggregate_prepared'
##Metrics ID
record_count = 'records:COUNT_RECORDS'
unique_values = 'col_stats:COUNT_DISTINCT:text'

##Retrieve Metrics Last Value and Compute Difference
source_metrics = proj.get_dataset(source_dataset).get_last_metric_values()
source_record_count = source_metrics.get_metric_by_id(record_count)['lastValues'][0]['value']
print(source_record_count)

target_metrics = proj.get_dataset(target_dataset).get_last_metric_values()
target_record_count = target_metrics.get_metric_by_id(record_count)['lastValues'][0]['value']
print(target_record_count)

gap = float(source_record_count) - float(target_record_count)
gap

Then in your custom check you can set the business rule for the Check outcome based on the 'gap' value

Hope this helps