Announcing the winners & finalists of the Dataiku Frontrunner Awards 2021! Read their inspiring stories

automatic updating of data with the scenario

Solved!
jtobelem
Level 2
Level 2
automatic updating of data with the scenario

Hi,

I use an api to get new data every day. I would like to stack those data on a unique dataset.

I have made this flow, but I can't build it recursively as it is a cycle and will produce stack overflow.

I think my solution is a bit tricky, is there a more straightforward way to do it?

 

- My flow first get data from api, then stack it with existing data and use sync recipe because it wasn't ok to stack directly the output of the stack recipe as an input.

flow.png

- My scenario build each of the three dataset of the flow. 

scenario.png

 

0 Kudos
1 Solution
Marine
Dataiker
Dataiker

Hi, 

In your case, it might be interesting to partition your dataset by dates and retrieve the latest data everyday with a scenario. 

Please find herewith dedicated resources : 

- Dataiku's academy on partitioning

- A tutorial to build latest partitions with a scenario

 

 

View solution in original post

0 Kudos
2 Replies
Marine
Dataiker
Dataiker

Hi, 

In your case, it might be interesting to partition your dataset by dates and retrieve the latest data everyday with a scenario. 

Please find herewith dedicated resources : 

- Dataiku's academy on partitioning

- A tutorial to build latest partitions with a scenario

 

 

View solution in original post

0 Kudos
jtobelem
Level 2
Level 2
Author

Hi Marine,

Thank you for your answer on the partitioning, it seems to be exactly what I was looking for! Unfortunately my licence does not work with partitioning, so I will try when we upgrade our plan.

Labels (2)
A banner prompting to get Dataiku DSS