Auto-regenerate schema on recipe step "Unnest object"
casper
Registered Posts: 42 ✭✭✭✭
I've made a recipe using the DSS recipe engine with the step "Unnest object in values".
This properly puts the object keys to columns and its values as their values. However, this only seems to save new keys when I run it in the designer and save the changes, which prompts me to accept the new schema changes.
When I run this recipe from the flow diagram with a changed set of columns due to the unnesting, no new columns are added to the output dataset. Have I missed an option?
This properly puts the object keys to columns and its values as their values. However, this only seems to save new keys when I run it in the designer and save the changes, which prompts me to accept the new schema changes.
When I run this recipe from the flow diagram with a changed set of columns due to the unnesting, no new columns are added to the output dataset. Have I missed an option?
Tagged:
Answers
-
Hello, The un-nest processor in a prepare visual recipe updates the schema only at design time. If your data changes and new keys appear, you will need to go again in the recipe design view. So the behaviour you describe is expected, as we enforce controls over the dataset schemas when you build from the flow. If you want to have a change in schema every time you run a recipe, you will need to write it in Python or R or Scala. There are examples of how to do it online, for instance https://stackoverflow.com/questions/38231591/splitting-dictionary-list-inside-a-pandas-column-into-separate-columns. Cheers, Alexandre
