Document undocumented scenario catch-up fuctionality

Turribeach

I have always assumed Dataiku will not run any missed scenarios when DSS is shutdown. Turns out I was wrong, Dataiku has a built-in scenario catch-up capability. This idea is to document the scenario catch-up fuctionality as it is critical feature that Dataiku Administrators and users should know and Support have confirmed it is currently undocumented. This fuctionality can't currently be disabled.

So here is how I understand this fuctionality works. When Dataiku is restarted again it will calculate the “lateness” of all missed scenarios with time-based triggers. If the lateness is 10% or less compared to normal delay between runs then Dataiku will run the missed scenario time-based trigger, otherwise it will not.

For instance for a daily time-based trigger set to run every 24hs, the time-based trigger will only catch up (ie run even after Dataiku restarts) if it the “lateness” was within 2 hours and 24 minutes of its initially planned running time (2h:24mi is 10% of 24hs).

Let's have a practical example. Scenario X has time-based trigger set to run every day at 10am. On Sat 9am Dataiku is shutdown for maintenance. Dataiku is then restarted at 11:30am. Dataiku then calculates the Scenario X “lateness” as 1h:30mi (11:30am - 10:00am). It then calculates the normal delay (24hs) and 10% of that (2h:24mi). As 1h:30mi < 2h:24mi the scenario will catch up/run after Dataiku is restarted.

Turribeach

It's not just a DSS restart that can result on the scenario catch up functionality being used but also disabling the Other => Misc. => Scenario Triggers =>Disable Automatic Triggers option (from a prior enabled state) will cause DSS to evaluate missed scenarios.

To provide a few examples:

1. Daily Time-based scenario that starts 6:00 PM - disableAutomaticTriggers is set to true
2. Shut off DSS at 5:59 PM
3. Start DSS at 6:01 PM - disableAutomaticTriggers still true
4. Set disableAutomaticTriggers to false at 6:03, the scenario that should have fired at 6:00 still fires at 6:04 PM

Second scenario:

1. Daily Time-based scenario starts at 6:07 - disableAutomaticTriggers -> true
2. 6:09 set disableAutomaticTriggers to false, scenario that should have fired at 6:07 fires at 6:09. No DSS restart needed ​in this test.