Dataset overwritten instead of error

SanderVW

When building datasets I have seen that, on changes to the schema given by the recipe, the dataset is fully overwritten, data and all. This means that, when a recipe suddenly does not return the correct schema, all previous data is lost…

Previously we did get an error message if this was the case and we would not lose any data unexpectedly but after updating to 12.3 (from 8.0) this is suddenly not done anymore. Is there a setting we would need to change to resolve this? It is becoming quite a problem as some 'dynamic' steps like pivots/folds can quikly return a somewhat different schema based on what input is given which causes big problems for the flow as well as historical data.

When knowingly changing the schema inside of a recipe and saving the recipe we do get an error message like the one below. But if we do not save explicitly and rather run the recipe instead, it gets saved anyway and the output dataset is dropped and recreated. Previously we would see this message pop up when running the recipe as well, is this possible to achieve?

Turribeach

Administration ⇒ Settings ⇒ Other ⇒ Misc. ⇒ Schema incompatibilities ⇒ Auto-accept schema changes at end of flow

SanderVW

Turribeach

Jul 30, 2024

Administration ⇒ Settings ⇒ Other ⇒ Misc. ⇒ Schema incompatibilities ⇒ Auto-accept schema changes at end of flow

For me it's under Administration⇒Settings⇒Engines & Flow⇒Flow build⇒Schema incompatibilities⇒Auto-accept schema changes at end of flow. I am on 12.3.1 if that matters.

It was disabled originally but enabling it did not change anything unfortunately. Should DSS be restarted after changes to this? Whether enabled or disabled it still overwrites the data without giving an error message.