Survey banner
The Dataiku Community is moving to a new home! Some short term disruption starting next week: LEARN MORE

propagate schema fails at partitioning sync recipe

tanguy
propagate schema fails at partitioning sync recipe

I added a column to the start of an SQL pipeline and needed to ensure that the schema change was propagated to subsequent tables. Thankflully, DSS "propage schema" functionality allowed me to automate this task.

Capture_x.JPG

dataiku's convenient functionality to propagate upstream changes in schema

However, at some point I encountered failures during the rebuilding process.

Capture_fail.JPG

at some point, the added column was not propagated

It appears that the sync recipe responsible for partitioning did not update the schema of the output partitioned table, resulting in the errors observed.

Capture_propagate.JPG

The step in the flow where the schema change stopped being propagated

 

Capture_culprit.JPG

 the culprit

It would be nice if the user had the possibility to force the schema update, including in partitioning recipes (even if this results in dropping all existing partitions, as the message warns). This option could be added to the existing ones in the propagate schema launching box.

0 Replies

Labels

?
Labels (2)

Setup info

?
A banner prompting to get Dataiku