Queuing Mechanism

[altaf_ansari]

Hello Dataiku Community,

I am working on a data flow where I require the following functionality:

1. I have an initial dataset, let's call it "A," which contains 7 rows of data.

2. I have a second dataset, "B," connected to dataset A. My goal is to move each row from dataset A to dataset B.

3. Each time a row is written to dataset B, I need a predefined scenario to be triggered detecting the change in dataset B . Once this scenario is successfully completed, the next row from dataset A should be processed and fed into dataset B, creating a queuing mechanism.

4. It's crucial that when the scenario runs, dataset B should be available for use within the scenario flows.

5. I anticipate that each scenario may take up to 2 days for completion. My concern is whether the workflow will wait for this extended duration before progressing to the next iteration.

Essentially, I'm looking to implement a queuing mechanism.I would appreciate any guidance or insights on how to implement this functionality effectively.

Thank you for your assistance.

[Turribeach]

Have a look at the Scenario trigger on dataset change and the Scenario trigger after another scenario.