Append data from a non-SQL step to a SQL table without dropping the SQL table
![echaoui](https://us.v-cdn.net/6038231/uploads/defaultavatar/nBZFMHJY3ZB3C.jpg)
Hello,
I'm trying to append data to my SQL table, but using the DSS recipe (Sync, Prepare), the SQL table is dropped and overwritten. Is there a way to only append?
Thanks
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,876 Neuron
Set the Append instead of Overwrite option. Depending on the recipe type this may be in the Configuration tab, Input / Output tab or Advanced tab. However note this is not a full guarantee Dataiku will not drop the table. Any schema changes will still cause Dataiku to drop the table. You can't have it both ways, if Dataiku manages the output table then it will need to drop it at times.
-
Thanks for the feedback. I tried using this option, but even though the schema is the same, it drops my table. Do you think there is a solution using Python or an SQL recipe? The table to be appended comes from non-SQL steps and needs to be appended to an SQL table.
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,876 Neuron
This option works so I think you are not looking at it properly. Rerun your final recipe (the one in append mode) twice. Do a row count on a SQL Notebook before and after running your recipe. You should see the row count increase.