automatic updating of data with the scenario

jtobelem
jtobelem Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Registered, Neuron 2022 Posts: 19 ✭✭✭✭✭

Hi,

I use an api to get new data every day. I would like to stack those data on a unique dataset.

I have made this flow, but I can't build it recursively as it is a cycle and will produce stack overflow.

I think my solution is a bit tricky, is there a more straightforward way to do it?

- My flow first get data from api, then stack it with existing data and use sync recipe because it wasn't ok to stack directly the output of the stack recipe as an input.

flow.png

- My scenario build each of the three dataset of the flow. 

scenario.png

Best Answer

Answers

  • jtobelem
    jtobelem Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Registered, Neuron 2022 Posts: 19 ✭✭✭✭✭

    Hi Marine,

    Thank you for your answer on the partitioning, it seems to be exactly what I was looking for! Unfortunately my licence does not work with partitioning, so I will try when we upgrade our plan.

Setup Info
    Tags
      Help me…