Senario Halting when no new data is available

Options
tgb417
tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron

I've got a data pipeline that runs on scenarios.

There may or may not be new data every time the scenario runs. I have been doing some inefficient items each time the scenario runs.

1. I have been guaranteeing that I get at least one old record every time the scenario runs, in order to ensure a stable schema. I have discovered if I have empty dataset the schema can get unsettled and has broken the pipeline.

Are their best practices for recognizing that a scenario is going to produce no-data and then terminate the scenario.

2. I run multiple scenarios back to back because I need to update multiple data sets. If one terminates a scenario in the above case. But still needs up update the datasets in a particular order. And just because the first scenario gets canceled because there is no new data in that dataset. How are folks insuring that the other datasets actually attempt to refresh.

Thoughts, Best Practices?

Thanks for your help.


Operating system used: Mac OS Ventura 13.4

Setup Info
    Tags
      Help me…