Sync recipe following a pivot recipe

sseveur

Hi,

I have a sync recipe following a pivot recipe.

The pivot recipe recomputes the schema at each run but there isn't that kind of parameter for the sync recipe.

Will my flow fail if the schema from the outputted dataset of the pivot recipe changes ?

Greetings,

Steven

fchataigner2

Hi,

the sync recipe doesn't automatically propagate schema changes, so it will simply fill columns in its output dataset that exist in the input.

But actually, pivot is an outlier here. The overwhelming majority of recipes in DSS don't change their output schema. And that's because automatically updating schemas of datasets is a complicated operation, and very hard to get right.

If you need simple propagation, like what you get when you right-click on a dataset and do "propagate schema changes from here", you can use a scenario:

- start by building the output of the pivot recipe with a Build/train step, in mode "build only this dataset"

- use a propagate schema step

- build the rest of the flow with a Build/train step

sseveur

Hi,

Thanks, didn't know the propagate schema has been added to the scenario. Didn't want to use the API.

Cheers