Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
I added a column to the start of an SQL pipeline and needed to ensure that the schema change was propagated to subsequent tables. Thankflully, DSS "propage schema" functionality allowed me to automate this task.
dataiku's convenient functionality to propagate upstream changes in schema
However, at some point I encountered failures during the rebuilding process.
at some point, the added column was not propagated
It appears that the sync recipe responsible for partitioning did not update the schema of the output partitioned table, resulting in the errors observed.
The step in the flow where the schema change stopped being propagated
the culprit
It would be nice if the user had the possibility to force the schema update, including in partitioning recipes (even if this results in dropping all existing partitions, as the message warns). This option could be added to the existing ones in the propagate schema launching box.
