How to programmatically refresh input dataset partitions with Snowflake?
Tanguy
Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2023 Posts: 137 Neuron
Hi,
I’m working with a Snowflake-partitioned dataset that serves as an input in my project flow. I’d like to automate the refresh of the partition listing, which is normally done manually using the "REFRESH PARTITIONS" button in the Metrics tab.
We previously managed to do this with S3 using the .synchronize_hive_metastore() method, but that no longer seems to work.
Does anyone have suggestions or an alternative approach?
Operating system used: RHEL 8.10
Operating system used: RHEL 8.10
Operating system used: RHEL 8.10
Best Answer
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,285 DataikerHi,
You can use tcompute_metrics with "ALL" partitions
https://developer.dataiku.com/latest/api-reference/python/datasets.html#dataikuapi.dss.dataset.DSSDataset.compute_metrics
Thanks