Announcing the winners & finalists of the Dataiku Frontrunner Awards 2021! Read their inspiring stories

Sync recipe following a pivot recipe

Solved!
sseveur
Level 3
Level 3
Sync recipe following a pivot recipe

Hi, 

I have a sync recipe following a pivot recipe. 

The pivot recipe recomputes the schema at each run but there isn't that kind of parameter for the sync recipe. 

Will my flow fail if the schema from the outputted dataset of the pivot recipe changes ? 

Greetings,

Steven 

 

0 Kudos
1 Solution
fchataigner2
Dataiker
Dataiker

Hi,

the sync recipe doesn't automatically propagate schema changes, so it will simply fill columns in its output dataset that exist in the input.

But actually, pivot is an outlier here. The overwhelming majority of recipes in DSS don't change their output schema. And that's because automatically updating schemas of datasets is a complicated operation, and very hard to get right.

If you need simple propagation, like what you get when you right-click on a dataset and do "propagate schema changes from here", you can use a scenario:

- start by building the output of the pivot recipe with a Build/train step, in mode "build only this dataset"

- use a propagate schema step

- build the rest of the flow with a Build/train step

View solution in original post

0 Kudos
2 Replies
fchataigner2
Dataiker
Dataiker

Hi,

the sync recipe doesn't automatically propagate schema changes, so it will simply fill columns in its output dataset that exist in the input.

But actually, pivot is an outlier here. The overwhelming majority of recipes in DSS don't change their output schema. And that's because automatically updating schemas of datasets is a complicated operation, and very hard to get right.

If you need simple propagation, like what you get when you right-click on a dataset and do "propagate schema changes from here", you can use a scenario:

- start by building the output of the pivot recipe with a Build/train step, in mode "build only this dataset"

- use a propagate schema step

- build the rest of the flow with a Build/train step

View solution in original post

0 Kudos
sseveur
Level 3
Level 3
Author

Hi,

Thanks, didn't know the propagate schema has been added to the scenario. Didn't want to use the API. 

Cheers 

0 Kudos
A banner prompting to get Dataiku DSS