how can we append the different build of same file?
Hi Dataiku community ,
I have created one python recipe which takes 3 different datasets as input and produces one output after processing some data transformation steps.
Now this output needs to be build every day and the one that runs on month end need to get appended in a other output which is basically created to keep the historically data.
Is there any way we can do this in DATAIKU ? or by using Python recipe ?
Please help
Best Answer
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,757 NeuronOptionsFrom your output daily dataset, whch you will build with a daily scenario, create a sync recipe (which basically copies the data from input to output). Have that sync recipe only run once a month in a separate scenario. You can configure the sync recipe to use the "Append instead of overwrite" but be warned, Dataiku will drop your historical dataset if the schema changes so you will be best keeping the historical data managed outside of Dataiku.
Answers
-
Thank you Turribeach that worked .