automatic updating of data with the scenario
Hi,
I use an api to get new data every day. I would like to stack those data on a unique dataset.
I have made this flow, but I can't build it recursively as it is a cycle and will produce stack overflow.
I think my solution is a bit tricky, is there a more straightforward way to do it?
- My flow first get data from api, then stack it with existing data and use sync recipe because it wasn't ok to stack directly the output of the stack recipe as an input.
- My scenario build each of the three dataset of the flow.
Best Answer
-
Hi,
In your case, it might be interesting to partition your dataset by dates and retrieve the latest data everyday with a scenario.
Please find herewith dedicated resources :
- Dataiku's academy on partitioning
- A tutorial to build latest partitions with a scenario
Answers
-
jtobelem Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Registered, Neuron 2022 Posts: 19 ✭✭✭✭✭Options
Hi Marine,
Thank you for your answer on the partitioning, it seems to be exactly what I was looking for! Unfortunately my licence does not work with partitioning, so I will try when we upgrade our plan.