propagate schema fails at partitioning sync recipe

Options
Tanguy
Tanguy Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2023 Posts: 112 Neuron

I added a column to the start of an SQL pipeline and needed to ensure that the schema change was propagated to subsequent tables. Thankflully, DSS "propage schema" functionality allowed me to automate this task.

Capture_x.JPG

dataiku's convenient functionality to propagate upstream changes in schema

However, at some point I encountered failures during the rebuilding process.

Capture_fail.JPG

at some point, the added column was not propagated

It appears that the sync recipe responsible for partitioning did not update the schema of the output partitioned table, resulting in the errors observed.

Capture_propagate.JPG

The step in the flow where the schema change stopped being propagated

Capture_culprit.JPG

the culprit

It would be nice if the user had the possibility to force the schema update, including in partitioning recipes (even if this results in dropping all existing partitions, as the message warns). This option could be added to the existing ones in the propagate schema launching box.

Setup Info
    Tags
      Help me…