Automation with a changing schema

Graham_E
Level 2
Automation with a changing schema
I can't find an explicit description of this use case in the docs, so I am asking here. If there is something in the docs I missed, please let me know.

I am using a custom plugin to read data from an external service (via a REST api). This is working great. I am also using a scenario to build the project on request. The caveat is that in the future the schema of the data will change. We have columns of the form "entry_1", "entry_2" ... "entry_n" etc. and the max value of n will most likely change in the future. Ideally, the automation setup we would use would allow the data sets schema change to be propagated when the project is re-built. Is this possible? I would like the flow of data to be as hands off as possible.

Thanks
0 Kudos
2 Replies
Clรฉment_Stenac
Hi,

It is not currently possible to automate schema propagation changes along the Flow.
0 Kudos
Graham_E
Level 2
Author
Thanks for the quick reply Clรฉment .

Is there a work around we could use in the meantime? Like is it possible to pre-define a schema in conjunction with the custom plugin? If so we could pad the schema so that the value "n" for "entry_n" is arbitrarily large.
0 Kudos

Labels

?
Labels (2)
A banner prompting to get Dataiku