How to implement a feedback loop on a dataset ?

Highlighted
tomtom
Level 2
How to implement a feedback loop on a dataset ?

Hello,



Each month, I have to compute a dataset that takes the previous month's dataset (M-1) and add some stuff in it.

I wonder how I could to it in Dataiku as for the recipe, I should take the last output dataset (M-1) as the input.



I don't think it is currently possible to produce a feedback-loop in Dataiku: do you confirm ?

How could I achieve my computation with Dataiku ? The "append-only" feature is not a good answer, because before writing anything, I should read the (last month) output dataset to know what will be new in the (current month) output.



 



Best regards.

1 Reply
Liev
Level 3
Re: How to implement a feedback loop on a dataset ?
Hi tomtom,

The flow interface won't like circular references, which sounds like what you're describing here.

Hence, if you have something like this: Dataset_A -> Recipe -> Dataset_B, one solution to your problem is to define Dataset_C by 'pointing it' to Dataset_B. You can do this in the flow by adding a new Dataset and matching the location (for example SQL table) of Dataset_B. This way you can use as input to your recipe both Dataset_A and Dataset_C (which is in fact the same as Dataset_B).

I hope this is not too confusing!
0 Kudos
Labels (2)