Join us at the Everyday AI Conference in London, New York & Bengaluru! REGISTER NOW

DATAIKU - append instead of override

muthu11
Level 2
DATAIKU - append instead of override

append instead of override is not working 

Planning to run the flow for every 5 mins.

Flow description:

Whenever there is a new record in SRC_DATA.

SQL recipe will do the diff between SRC_DATA table and TGT_DATA table and load into -> INTERM_SQL

Python recipe I do some count records and loads INTERM_SQL data to TGT_DATA.

once TGT_DATA  loaded, using post write statement I am deleting data from the INTERM_SQL table

For INTERM_SQL & TGT_DATA  table  - I have enabled append instead of an override in their respective SQL & Python recipe.

tables get overridden instead of appending whenever Flow builds.

 

 
 

dataiku.jpg

0 Kudos
2 Replies
tomas
Neuron
Neuron

Hi I tested out a similar setup and for me it is working. 

 

Also note> you dont need to do truncate on the last table post write, just change the first recipe and uncheck "append instead overwrite". Therefore when the next time you run the whole flow, the first temporary table is overwritten (with new data only), and then the dataset is appended to a target table.

I run a simple flow with 45 records 4 times and the target table has now 135 records.

 

 

muthu11
Level 2
Author

I have one flow where I load CSV to the table in that the append feature is working properly.

I tried disabling the append option initially then 

I thought of trying to enable append for all the tables in the flow. So I enabled append instead of override for my interm_sql table and deleted it on post write.

Weird with this flow

0 Kudos