Dataset overwritten instead of error

Registered Posts: 47 ✭✭✭✭
edited July 2024 in Using Dataiku

When building datasets I have seen that, on changes to the schema given by the recipe, the dataset is fully overwritten, data and all. This means that, when a recipe suddenly does not return the correct schema, all previous data is lost…

Previously we did get an error message if this was the case and we would not lose any data unexpectedly but after updating to 12.3 (from 8.0) this is suddenly not done anymore. Is there a setting we would need to change to resolve this? It is becoming quite a problem as some 'dynamic' steps like pivots/folds can quikly return a somewhat different schema based on what input is given which causes big problems for the flow as well as historical data.

When knowingly changing the schema inside of a recipe and saving the recipe we do get an error message like the one below. But if we do not save explicitly and rather run the recipe instead, it gets saved anyway and the output dataset is dropped and recreated. Previously we would see this message pop up when running the recipe as well, is this possible to achieve?

Answers

Welcome!

It looks like you're new here. Sign in or register to get started.

Welcome!

It looks like you're new here. Sign in or register to get started.