Queuing Scenarios with Webapp

chrisfeagles

Hello,

I have a former colleague who developed a code (JS, Python, HTML, CSS) webapp that allows non-Dataiku users in our organization to run scenarios from my teams projects with different input parameters and receive the analysis output via email when the scenario completes. One item we've recently struggled with is the growing user base of our analysis tools and issues with multiple users using the webapp at once to run the same scenario. At the moment, the code "queues" a scenario by requiring the user to maintain their browser window with a pop-up that says "please wait, don't close", this checks when a currently running scenario completes and starts the next. We're finding that people are not following the directions and closing this window, which seems to result in this "queued" scenario not receiving the parameters for their run and the scenario ultimately fails.

I'm curious if anyone has found a better way to handle this situation? Is there a concept of queuing a scenario behind another with custom parameters with the Python API?

Sarina

Hi @chrisfeagles
,

I think that this might still lead to the same issues, because if the template scenario is paused waiting for the other scenarios to complete, then that template scenario won't be able to be re-triggered until it completes. It's a bit of a circular problem! Thinking through your use case it does seem like Applications might be an overall better solution, I should have mentioned this earlier. One of the main purposes of applications is to allow for concurrent builds of the flow, by allowing users to have their own instantiation of the application. So I think that moving over to an Application might be a more scalable and simpler option.

Thanks,
Sarina

Sarina

Hi @chrisfeagles
!

Others might have additional ideas, but I have one other idea here that might work. The idea is to create a new scenario each time a user uses the webapp, run the scenario with the user's specific parameters and then delete the scenario upon completion. This way, you could have many simultaneous scenarios running at once for each webapp that is up.

It's a little awkward, but I think it accomplishes what you'd want to see. For a rough idea of how this would work (I'm happy to pass along a more specific example if you are interested in this approach):
(1) You would have a base scenario that you'll use to generate all of the temp generated scenarios from. You'll never actually run the base scenario though, it will now simply be used as a template for your run scenarios.
(2) When a user "runs the scenario" your code will now get your template scenario and get the definition from the template scenario. It will then use create_scenario() to create a new temporary scenario based off of the base scenario template definition and the user's inputted parameters. You can use a random hash or something similar to add to the new temp scenario name to ensure that many different scenarios can be created at once. Then the webapp triggers the temp scenario.
(3) Once the scenario completes, the webapp can destroy the scenario. If it seems like that might cause similar issues to your current ones, you could instead have a job that runs ~hourly and cleans up all completed temp scenarios or something along those lines.

If users navigate away from the webapp in this case there shouldn't be an issue, since the temp scenario will trigger right away with the correct parameters.

I would be happy to build out a full example if this seems like an approach you'd be interested in trying.

Thanks,
Sarina

chrisfeagles

Hi Sarina,

Sounds interesting, I am wondering how this would work when the scenarios are building part of the flow?

Chris

Sarina

Hi @chrisfeagles
,

Ah indeed this will not work as the flow cannot be built concurrently. In that case, I think that switching from using a webapp to using DSS Applications could be a better option. This will allow users to instantiate their own instance of the application and run the scenario from the application instance, allowing for full concurrency when triggering a scenario with parameters.

If this seems like a good approach, you can reach out to your customer team to help walk through any of the details of your use case as well.

Thanks,
Sarina

chrisfeagles

Your original suggestion got me thinking. What if I add a “pause” and check if other scenarios are running to the start of the template scenario. This would still allow for “queuing” outside of the web app. Then I could just add logic to the first scenario step instead. Of course having some timeout would be necessary.