Can Data Collections be automatically propagated from Designer to Automation in Dataiku?
Hello everyone,
I would like to understand whether Data Collections in Dataiku can be propagated from a Designer node to an Automation node as part of a standard project deployment, without having to recreate them manually on Automation.
Context
In the case of the Feature Store, when a project bundle is deployed from Designer to Automation and a dataset is already published to the Feature Store on Designer, that publication is also reflected automatically on the Automation node.
Question
Is there a similar mechanism for Data Collections?
Goal
The objective is to avoid manual setup on Automation and make Data Collection availability part of the normal deployment workflow, similar to how Feature Store assets behave.
Any guidance or recommended pattern would be appreciated.
Thank you.
Comments
-
Miguel Angel Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 120 DataikerHi,
Data Collections are instance-level catalog objects, not project assets. As such, they are not propagated across nodes
-
Hi @Pankaj925,
Thanks for the feedback and an interesting idea!
Maybe it's a noob question. I'm just wondering why do you want to have data collection also in the automation node?
The reason I ask is that from my perspective, data collection is mostly used during the building phase to find building blocks when you work on a project.
I'm wondering why do you want to have it in automation node?
Thanks,
Han
-
Hi @Han_Han, thanks for asking this. So, the way my current mlops pipeline work, it only does monitoring for datasets from data management project. So, for datasets which are not part of data management, i am publishing them to data collection first and then fetching from there in automation to do monitoring.
Hope it answers
-
HI @Miguel Angel , thanks for your response.
-
@Miguel Angel: But, even feature store is also instance level objects.
