Scenarios can't recursively call themselves nor can their be multiple executions of the same scenario at the same time. Personally I would modify your plugin to be able to handle a range of dates as how would you handle the situation when the data load fails for a few days and you need to catch up and load multiple days?
Further more I wouldn't use project variables because these can be left in an incorrect state and cause data to be duplicated. In DSS, there are instance-level variables, project-level variables, and scenario-level variables. The scenario-level variables are only visible for the duration of the scenario run, and are defined either programatically (in Python) or using a Define variables Scenario step. All 3 types of variables are available to recipes of jobs started in a scenario run. If you are loading an historical table the right approach will be to check the last date loaded in the historical table at runtime as part of your Scenario run and then load all dates from that date till today using a range date Scenario variable that the recipe/plugin can read. This will make the plugin have an auto-catch up capability and the range date Scenario variable will always get the correct value as it will be calculated at runtime.
Here is how you define a scenario variable:
from dataiku.scenario import Scenario
scenario = Scenario()
all_scenario_vars = scenario.get_all_variables()
all_scenario_vars['new_scenario_var'] = "new value"
scenario.set_scenario_variables(**all_scenario_vars)
You can run this on a Custom Python step in your Scenario.
Finally while you can't run scnearios recursively there is one trick you can use. There is scenario trigger called "Trigger after scenario" which basically runs after another scenario runs. You could easily have two scenarios which are set to "Trigger after scenario" like this:
- Scenario 1 "Trigger after scenario" Scenario 2
- Scenario 2 "Trigger after scenario" Scenario 1
And obviously if you trigger Scenario 1 it will then trigger Scenario 2 after it finishes which then will trigger Scenario 1 ad infinitum. However you will need to be able to find a way to break the recursive call as it will never ever stop.
