I'm currently trying to keep track of metrics on a partitioned S3 dataset. I've used the "Autocompute after build" option, but the metrics are not computed on dataset change. I've also tried to compute the metrics using the api (here I'm interested in the record count) :
client = dataiku.api_client()
current_project = client.get_project('project')
sales_s3 = current_project.get_dataset('dataset')
But I get the following error :
DataikuException: java.lang.IllegalArgumentException: For `records:COUNT_RECORDS': Invalid partition identifier, has 1 dimensions, expected 2
Is there a workaround?
Here is what the partition screen looks like :
Ah! This is an interesting topic. There was one small thing missing in your code 🙂
When working on partitioned datasets, the compute_metrics method expects to know which partitions to work on. Hence the correct syntax is:
Note the use of  around metric_ids. It has to be a list, which means you can compute several metrics in one go for a given partition. To get the current list of partitions, you should use:
If you wanted to compute the metric for the whole dataset, then simply pass the partition keyword "ALL"
Pro-tip: when prototyping code inside a Jupyter notebook, the shortcut Shift+Tab will open a tooltip box with the documentation of the classes and method you are using. There are many useful tricks in Jupyter, have a look at https://www.cheatography.com/weidadeyue/cheat-sheets/jupyter-notebook/pdf_bw/.