I've got a data pipeline that runs on scenarios.
There may or may not be new data every time the scenario runs. I have been doing some inefficient items each time the scenario runs.
1. I have been guaranteeing that I get at least one old record every time the scenario runs, in order to ensure a stable schema. I have discovered if I have empty dataset the schema can get unsettled and has broken the pipeline.
Are their best practices for recognizing that a scenario is going to produce no-data and then terminate the scenario.
2. I run multiple scenarios back to back because I need to update multiple data sets. If one terminates a scenario in the above case. But still needs up update the datasets in a particular order. And just because the first scenario gets canceled because there is no new data in that dataset. How are folks insuring that the other datasets actually attempt to refresh.
Thoughts, Best Practices?
Thanks for your help.
Operating system used: Mac OS Ventura 13.4
--Tom
