Fill in gaps - incremental load

Registered Posts: 1

We have requirement which requires full historical data, however we are using incremental load, is there any recipe for this action?

Answers

  • Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,309 Neuron

    All recipes support the "Append instead of overwrite" option which is usually in the recipe's Input/Output tab but might be elsewhere in some recipes. Be careful with this option though, it does not guarantee that Dataiku will not drop the historical table if the schema changes which is how Dataiku will push schema changes. So in general you will be best to do your incremental load in Dataiku and then push those rows to the historical table outside of Dataiku so you can have full control of your historical table. With power comes responsability so you will need to manage your historical table accordingly (ie what to do with new columns are added, etc).

  • Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 14

    Hi @raf_sia
    ,Do you solve this situation?

Welcome!

It looks like you're new here. Sign in or register to get started.

Welcome!

It looks like you're new here. Sign in or register to get started.