Can Data Collections be automatically propagated from Designer to Automation in Dataiku?

Pankaj925

Hello everyone,

I would like to understand whether Data Collections in Dataiku can be propagated from a Designer node to an Automation node as part of a standard project deployment, without having to recreate them manually on Automation.

Context

In the case of the Feature Store, when a project bundle is deployed from Designer to Automation and a dataset is already published to the Feature Store on Designer, that publication is also reflected automatically on the Automation node.

Question

Is there a similar mechanism for Data Collections?

Goal

The objective is to avoid manual setup on Automation and make Data Collection availability part of the normal deployment workflow, similar to how Feature Store assets behave.

Any guidance or recommended pattern would be appreciated.

Thank you.

Miguel Angel

Hi,

Data Collections are instance-level catalog objects, not project assets. As such, they are not propagated across nodes

Han_Han

Hi @Pankaj925,

Thanks for the feedback and an interesting idea!

Maybe it's a noob question. I'm just wondering why do you want to have data collection also in the automation node?

The reason I ask is that from my perspective, data collection is mostly used during the building phase to find building blocks when you work on a project.

I'm wondering why do you want to have it in automation node?

Thanks,

Han

Pankaj925

Hi @Han_Han, thanks for asking this. So, the way my current mlops pipeline work, it only does monitoring for datasets from data management project. So, for datasets which are not part of data management, i am publishing them to data collection first and then fetching from there in automation to do monitoring.

Hope it answers

Pankaj925

HI @Miguel Angel , thanks for your response.

Pankaj925

https://community.dataiku.com/discussion/comment/47585#Comment_47585

@Miguel Angel: But, even feature store is also instance level objects.