Survey banner
The Dataiku Community is moving to a new home! Some short term disruption starting next week: LEARN MORE

propagate schema fails at partitioning sync recipe

propagate schema fails at partitioning sync recipe

I added a column to the start of an SQL pipeline and needed to ensure that the schema change was propagated to subsequent tables. Thankflully, DSS "propage schema" functionality allowed me to automate this task.


dataiku's convenient functionality to propagate upstream changes in schema

However, at some point I encountered failures during the rebuilding process.


at some point, the added column was not propagated

It appears that the sync recipe responsible for partitioning did not update the schema of the output partitioned table, resulting in the errors observed.


The step in the flow where the schema change stopped being propagated



 the culprit

It would be nice if the user had the possibility to force the schema update, including in partitioning recipes (even if this results in dropping all existing partitions, as the message warns). This option could be added to the existing ones in the propagate schema launching box.

0 Replies


Labels (2)

Setup info

A banner prompting to get Dataiku