Can I use "append instead of overwrite" on a filter recipe to store results from a Twitter stream?

Alex_Combessie Posts: 539 ✭✭✭✭✭✭✭✭✭

March 2016 in Using Dataiku

I run my filter recipe periodically. I would like to use append instead of overwrite to store only new tweets (not what is already stored with the recipe output). Is it possible?

Tagged:

• Visual recipes
• Filtering

0 · Share on Facebook

Answers

• 

  jrouquie Posts: 87 ✭✭✭✭✭✭✭

  March 2016

  Hi,

  this looks like a use case for partitioning. Note this is an advanced concept of DSS.

  Imagine one uses "append", discovers a bug, fixes it, runs the flow again: the output dataset would then contain both the buggy and the correct data.

  0 · Share on Facebook
• 

  DvMg Posts: 10 ✭✭✭✭

  April 2019

  Do you have an example of how can do that?

  0 · Share on Facebook