Sync recipe following a pivot recipe

Partner, L2 Designer, Registered Posts: 16 Partner

Hi,

I have a sync recipe following a pivot recipe.

The pivot recipe recomputes the schema at each run but there isn't that kind of parameter for the sync recipe.

Will my flow fail if the schema from the outputted dataset of the pivot recipe changes ?

Greetings,

Steven

Best Answer

  • Dataiker Posts: 355 Dataiker
    Answer ✓

    Hi,

    the sync recipe doesn't automatically propagate schema changes, so it will simply fill columns in its output dataset that exist in the input.

    But actually, pivot is an outlier here. The overwhelming majority of recipes in DSS don't change their output schema. And that's because automatically updating schemas of datasets is a complicated operation, and very hard to get right.

    If you need simple propagation, like what you get when you right-click on a dataset and do "propagate schema changes from here", you can use a scenario:

    - start by building the output of the pivot recipe with a Build/train step, in mode "build only this dataset"

    - use a propagate schema step

    - build the rest of the flow with a Build/train step

Answers

Welcome!

It looks like you're new here. Sign in or register to get started.

Welcome!

It looks like you're new here. Sign in or register to get started.