Append data from a non-SQL step to a SQL table without dropping the SQL table

echaoui

Hello,

I'm trying to append data to my SQL table, but using the DSS recipe (Sync, Prepare), the SQL table is dropped and overwritten. Is there a way to only append?

Thanks

Turribeach

Set the Append instead of Overwrite option. Depending on the recipe type this may be in the Configuration tab, Input / Output tab or Advanced tab. However note this is not a full guarantee Dataiku will not drop the table. Any schema changes will still cause Dataiku to drop the table. You can't have it both ways, if Dataiku manages the output table then it will need to drop it at times.

echaoui

Thanks for the feedback. I tried using this option, but even though the schema is the same, it drops my table. Do you think there is a solution using Python or an SQL recipe? The table to be appended comes from non-SQL steps and needs to be appended to an SQL table.

Turribeach

This option works so I think you are not looking at it properly. Rerun your final recipe (the one in append mode) twice. Do a row count on a SQL Notebook before and after running your recipe. You should see the row count increase.