how can we append the different build of same file?

KK12 Registered Posts: 2

Hi Dataiku community ,

I have created one python recipe which takes 3 different datasets as input and produces one output after processing some data transformation steps.

Now this output needs to be build every day and the one that runs on month end need to get appended in a other output which is basically created to keep the historically data.

Is there any way we can do this in DATAIKU ? or by using Python recipe ?

Please help

Best Answer

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,706 Neuron
    Answer ✓

    From your output daily dataset, whch you will build with a daily scenario, create a sync recipe (which basically copies the data from input to output). Have that sync recipe only run once a month in a separate scenario. You can configure the sync recipe to use the "Append instead of overwrite" but be warned, Dataiku will drop your historical dataset if the schema changes so you will be best keeping the historical data managed outside of Dataiku.


Setup Info
      Help me…