append table
Hello,
I have an output table in my dataiky flow where results are appended after each build.
We had a data problem and we would like to relaunch the last run while removing the results of the previous one from the table and replacing it by the new one?
How can we do this without revert to a previous version because we have other changes we would like to keep.
Thank you in advance for your help
Operating system used: windows
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,987 Neuron
There is no built-in functionality for what you want to do. Furthermore I would warn you that append recipes are risky thing to use in Dataiku if you depend on your historical data. The default functionality in Dataiku is that when there are schema changes and changes are propagated is to drop and recreate the table, irrespective of you having set the "Append instead of override" check box in any recipe.
-
Thank you for your response !
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,987 Neuron
A better way of doing this will be to take the output of the Dataiku flow and do deltas outside of Dataiku so that you can manage your historical table outside of Dataiku. If you need to read your historical table in Dataiku you can add it to the flow as "unmanaged/external" dataset which means Dataiku will not be responsible for managing the schema of that table and you will need to manage it yourself (ie adding/deleting columns, etc).