Survey banner
The Dataiku Community is moving to a new home! Some short term disruption starting next week: LEARN MORE

DATAIKU - append instead of override

Level 2
DATAIKU - append instead of override

append instead of override is not working 

Planning to run the flow for every 5 mins.

Flow description:

Whenever there is a new record in SRC_DATA.

SQL recipe will do the diff between SRC_DATA table and TGT_DATA table and load into -> INTERM_SQL

Python recipe I do some count records and loads INTERM_SQL data to TGT_DATA.

once TGT_DATA  loaded, using post write statement I am deleting data from the INTERM_SQL table

For INTERM_SQL & TGT_DATA  table  - I have enabled append instead of an override in their respective SQL & Python recipe.

tables get overridden instead of appending whenever Flow builds.




0 Kudos
2 Replies
Level 5

Hi I tested out a similar setup and for me it is working. 


Also note> you dont need to do truncate on the last table post write, just change the first recipe and uncheck "append instead overwrite". Therefore when the next time you run the whole flow, the first temporary table is overwritten (with new data only), and then the dataset is appended to a target table.

I run a simple flow with 45 records 4 times and the target table has now 135 records.



Level 2

I have one flow where I load CSV to the table in that the append feature is working properly.

I tried disabling the append option initially then 

I thought of trying to enable append for all the tables in the flow. So I enabled append instead of override for my interm_sql table and deleted it on post write.

Weird with this flow

0 Kudos