DATAIKU - append instead of override
append instead of override is not working
Planning to run the flow for every 5 mins.
Flow description:
Whenever there is a new record in SRC_DATA.
SQL recipe will do the diff between SRC_DATA table and TGT_DATA table and load into -> INTERM_SQL
Python recipe I do some count records and loads INTERM_SQL data to TGT_DATA.
once TGT_DATA loaded, using post write statement I am deleting data from the INTERM_SQL table
For INTERM_SQL & TGT_DATA table - I have enabled append instead of an override in their respective SQL & Python recipe.
tables get overridden instead of appending whenever Flow builds.
Answers
-
Hi I tested out a similar setup and for me it is working.
Also note> you dont need to do truncate on the last table post write, just change the first recipe and uncheck "append instead overwrite". Therefore when the next time you run the whole flow, the first temporary table is overwritten (with new data only), and then the dataset is appended to a target table.
I run a simple flow with 45 records 4 times and the target table has now 135 records.
-
I have one flow where I load CSV to the table in that the append feature is working properly.
I tried disabling the append option initially then
I thought of trying to enable append for all the tables in the flow. So I enabled append instead of override for my interm_sql table and deleted it on post write.
Weird with this flow
