Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi Dataiku community ,
I have created one python recipe which takes 3 different datasets as input and produces one output after processing some data transformation steps.
Now this output needs to be build every day and the one that runs on month end need to get appended in a other output which is basically created to keep the historically data.
Is there any way we can do this in DATAIKU ? or by using Python recipe ?
Please help
From your output daily dataset, whch you will build with a daily scenario, create a sync recipe (which basically copies the data from input to output). Have that sync recipe only run once a month in a separate scenario. You can configure the sync recipe to use the "Append instead of overwrite" but be warned, Dataiku will drop your historical dataset if the schema changes so you will be best keeping the historical data managed outside of Dataiku.
From your output daily dataset, whch you will build with a daily scenario, create a sync recipe (which basically copies the data from input to output). Have that sync recipe only run once a month in a separate scenario. You can configure the sync recipe to use the "Append instead of overwrite" but be warned, Dataiku will drop your historical dataset if the schema changes so you will be best keeping the historical data managed outside of Dataiku.
Thank you Turribeach that worked .