Sync recipe following a pivot recipe

Options
sseveur
sseveur Partner, L2 Designer, Registered Posts: 16 Partner

Hi,

I have a sync recipe following a pivot recipe.

The pivot recipe recomputes the schema at each run but there isn't that kind of parameter for the sync recipe.

Will my flow fail if the schema from the outputted dataset of the pivot recipe changes ?

Greetings,

Steven

Best Answer

  • fchataigner2
    fchataigner2 Dataiker Posts: 355 Dataiker
    Answer ✓
    Options

    Hi,

    the sync recipe doesn't automatically propagate schema changes, so it will simply fill columns in its output dataset that exist in the input.

    But actually, pivot is an outlier here. The overwhelming majority of recipes in DSS don't change their output schema. And that's because automatically updating schemas of datasets is a complicated operation, and very hard to get right.

    If you need simple propagation, like what you get when you right-click on a dataset and do "propagate schema changes from here", you can use a scenario:

    - start by building the output of the pivot recipe with a Build/train step, in mode "build only this dataset"

    - use a propagate schema step

    - build the rest of the flow with a Build/train step

Answers

Setup Info
    Tags
      Help me…