Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Is there a 'simple' way to use a sync recipe or other recipe to transfer only the new records from a source table?
I haven't fleshed out this entire thought yet but my table does have an identity column so that makes things slightly easier:
Let me know if there is another way that I'm not aware of to accomplish the same.
An interesting question.
You seem to be focusing on the part of the problem that is about adding new records. And that is an important part of many scenarios.
However, when working through similar problems I also find myself also thinking about the questions of deleted records and modified records. There are some situations like financial records and logs where existing records are not likely to change. However, there are lots of scenarios like Customer Directories where existing records are both modified and deleted. In those cases, an incremental addition can work.
Before going down the road you are laying out, I invite you to consider if either of these two additional aspects will be part of your situation?
You should look into the partitions functionality, which enables you to process just certain chunks of data. In your case, you would need your new records to be identified by a unique partition id, perhaps the insert date.
Watch this video from the academy to get started on partitions:
I hope this helps.