I am trying to pass a parameter from a dash web app to a scenario to use in a flow.

Mattdb

This is what I have tried
trigger_fire = scenario.run()

    scenario.set_scenario_variables(run_id=run_trigger.id, File = "your_value")

    scenario_run = trigger_fire.wait_for_scenario_run()

but I am getting this error  'DSSScenario' object has no attribute 'set_scenario_variables'

Alexandru

Hi Matt,
You cannot pass a scenario variable outside of a scenario itself.
You can leverage project variables instead as explained here :
https://community.dataiku.com/discussion/3687/scenario-with-parameters

Thanks

Turribeach

@Alexandru I hate to defy the official authority but I believe this statement is not correct. Custom Scenario variables are available in recipes outside the scenario itself as long as the recipe is running as part of the scenario and you use the right method to fetch them. Let me shown it with an example. @Mattdb read along since this subject is somehow more complicated that it needs to be.

Here I defined two variables in a Scenario with a Define variables step:

Note that I intentionally overwrote a project variable to see how they interact with Custom Scenario variables. Here are my project variables:

OK now how can we see / interact with these Custom Scenario Variables? One way is to create a Custom Python scenario step which basically access them in Python code as part of the scenario execution:

If we now execute this scenario:

And look at the scenario logs we can see the values of the scenario variables:

Note how the project_variable is being overwritten by the Scenario variable of the same name as the variable value is "overwritten project value" rather than "project value" as defined in the project variables. So keep this in mind, Scenario variables take precedence over Project or Intance Global variables.

Can I access these Scenario Variables from a recipe?

Yes! Let's see what happens if we add the same Python code in a recipe and run the recipe manually:

So it fails as it is not part of a Scenario run so it has no scenario object to get. What if I run the recipe as part of the Scenario run then? Let's add the dummy_ds as a Build step and run the recipe to execute the recipe as part of a scenario run:

It fails again! So what's going on here? What's happening here is that the Python recipe is executing in a different context than of the scenario run and as such it won't be able to access the Scenario variables directly using a scenario handle which only applies when running Python code in a scenario step/script. To access the Scenario variables in a recipe we must use the dataiku.get_custom_variables() method as follows (line 14):

Now we can execute the scenario again and we get a successful run. We then look at the sched_build job logs to see if we can see the Custom Scenario variables:

And we can see that not only our custom Scenario variable is present but also the Project variable value is overwritten as expected. One final issue, if you wish to access Scenario variables in recipes you need to make sure they exist before you use them. Let's see what happens if we now run the updated Python recipe where we read the Scenario variables using dataiku.get_custom_variables() outside a Scenario run:

We now get a scenario_variable is not defined error but note that there is no error on the project_variable. This is because the project_variable exists in the project scope so when the recipe runs outside the scenario it will still be able to use the project variable. If we look at the job logs e can see that actually the project variable is not using the scenario overwritten value but the project one, proving the value didn't get assigned since this recipe ran outside the scenario run which overrides the project_variable:

But the scenario_variable does not exist so the recipe fails. So how can we get around this error? Well the simplest solution is to use the get method when we retrieve the scenario variable value which won't raise an exception if the variable (key) doesn't exist or it has not been set:

Now it works as expected. If we look at the logs we see that the scenario_variable Is None as it is not defined as expected for a recipe run outside a scenario:

But obviously in addition to handle the exception you will also need to think how you expect the recipe to behave when it runs outside a scenario run and the scenario variables are not set. One way to deal with this issue would be to define all your scenario variables as project variables as well in which case the project variables will act as a default value if you run a recipe manually outside an scenario and then get set properly at runtime when the recipe runs as part of a scenario run.

But then if you do that why use scenario variables at all? Why not just use project variables directly given that you have to define scenario variable as project variables to avoid recipe errors when running recipes outside a scenario? Well in this design you will always get to keep your project variables default values "static" as your scenario variables will
overwrite the values at runtime during the scenario run, so the project variables will not change with the scenario execution even though difterent values may be used at runtime by the scenario. That is probably the key advantage of using Scenario variables in that any manual recipe executions will run with expected default values rather than variables that may have changed due to the last execution of a scenario. In other words Scenario variables are ephimeral and only exist while the scenario is running while project variables exist for ever.

So the next question is how can I access these Scenario Variables from a visual recipe?

This is easy as these custom Scenario variables are now available to all recipes. Let's create a Prepare recipe and use project_variable:

If we just run the recipe we can see that the value of the new column is "project value":

This is expected since it's using the project variable value. Now let's add this dataset to the scenario and run it from there:

Success. If we now visit the dataset with the Project/Scenario variable we can see the value was overwritten by the scenario as expected:

Now all of this doesn't really address @Mattdb actual requirement but the above introduction was required to be able to understand how Scenario variables work. In @Mattdb case we want to pass some variables from a Dash webapp itself, rather than fetching them in the Scenario. For this we now need to use an even darker feature called Scenario parameters. First let's define how to execute a scenario and pass some parameters to it:

import dataiku

client = dataiku.api_client()
project = client.get_project('CT_TEST')
scenario = project.get_scenario('SCENARIO_VARIABLES_TEST')

# Execute synchronously with scenario.run_and_wait()
scenario_run = scenario.run_and_wait(params={'param1': 'value1', 'param2': 2})
print(scenario_run.outcome)

So now I am calling my scenario and passing to parameters, param1 and param2. How can we see these parameters? I now modified my scenario custom Python step to show all variables using pprint:

If we look at the step log we can see our custom parameters being passed:

However as you can see these Scenario parameters are not passed exactly by their name but are passed with a "scenarioTriggerParam_" preffix and are also in the scenarioTriggerParams dictionary. So let's now extract the Scenario parameter and push it to a Scenario variable so we can use it in recipes in the flow. I do this in my scenario Custom Python step:

On line 10 you can see how I extract the Scenario Parameter and on line 11 I push it to our existing project_variable Project/Scenario variable. Now let's run the scenario using the API code and passing the parameter. We can see how I get the value of the parameter in my log output:

And finally if we look at the dataset using the formula with "${project_variable}" it no shows value1 which is the value passed from the Scenario parameter:

So there you go, this is a massive post to explain something that really should be much more easy to do in Dataiku. Hopefully Dataiku can improve this is in the future but at the moment this is the only way of passing Scenario parameters from outside Dataiku to the Dataiku flow.