Sync Recipe - Append instead of overwrite

camshaft

Greetings,

I have a Sync recipe that uses an SQL query as the Input to load the past 6 months of data thru the previous calendar day to an Output table. Going forward, all I want is to load is the previous calendar day's data. Is this as simple as placing a check mark in the Append instead of overwrite checkbox shown beneath the Output? I'm looking for a way to load only the previous day's data without having to manually change the start date in the Input SQL query. In other words, is the Sync recipe smart enough to know to only load new records and ignore records already loaded to the Output target table? I'm using Dataiku 13.1.

Thanks in advance.

Turribeach

Yes, it's as simple as that. And you are lucky you are in v13.1 as Dataiku has finally fixed the issue that when the input schema changed it would drop and recreate the output table with the new schema. Of course that would catch people all the time as they were treating the Append mode as an historical table.

https://doc.dataiku.com/dss/latest/release_notes/13.html#behavior-change-handling-of-schema-mismatch-on-sql-datasets

camshaft

Thank you, Turribeach! That was the answer I was hoping for.