Queuing Mechanism

altaf_ansari Registered Posts: 3

Hello Dataiku Community,

I am working on a data flow where I require the following functionality:

  1. I have an initial dataset, let's call it "A," which contains 7 rows of data.

  2. I have a second dataset, "B," connected to dataset A. My goal is to move each row from dataset A to dataset B.

  3. Each time a row is written to dataset B, I need a predefined scenario to be triggered detecting the change in dataset B . Once this scenario is successfully completed, the next row from dataset A should be processed and fed into dataset B, creating a queuing mechanism.

  4. It's crucial that when the scenario runs, dataset B should be available for use within the scenario flows.

  5. I anticipate that each scenario may take up to 2 days for completion. My concern is whether the workflow will wait for this extended duration before progressing to the next iteration.

Essentially, I'm looking to implement a queuing mechanism.I would appreciate any guidance or insights on how to implement this functionality effectively.

Thank you for your assistance.


  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,717 Neuron

    Have a look at the Scenario trigger on dataset change and the Scenario trigger after another scenario.

Setup Info
      Help me…