schema propagation problem

Martine

In the dataset explore, I can define 'description' using 'edit column schema'.

And I can propagate the schema to the following 'flow' using 'schema propagation'.

However, sometimes it is not inherited (if there is a 'prepare recipe' in the middle)

I want to know how to inherit it normally.

Alexandru

Hi @Martine ,

Schema propagation behaves differently in a Prepare recipe compared to other recipes as explained https://doc.dataiku.com/dss/latest/schemas/data-preparation.html#schema-in-prepare-recipe

The Prepare recipe enforces "separation" between input and output schema to handle transformations safely. This results in this behavior

• At Creation: All types from the input dataset are copied to the output dataset only when the recipe is first created.
  
• New Columns: When a column is created (or renamed), it is treated as "new." Its type is inferred safely based on the values, rather than inherited.
  
• No Propagation: Subsequent type changes in the input dataset are not propagated to the output because of the independence between the input and output schemas.

To handle this, you can set the types you need to change by modifying the type in the prepared recipe or in the output dataset schema of the prepare recipe.