Passing global variables to application-as-recipe parameters
I'm trying to develop an application-as-recipe to create a consistent data extract. My plan is to have this application-as-recipe to function for both building a training dataset and for a daily scoring process. The process for building the dataset would use fixed start and end dates but the daily scoring process would use a 7 day moving window. My thought would be to create a scenario where the first step is a custom python script that updates the project's global variables relative to the current date. Then, I would pass the global variable to the application-as-recipe settings page.
Unfortunately it appears that the appilication-as-recipe will not accept global variables here. Any thoughts on how I can pass global variables or otherwise update the dates in an automated method?
Answers
-
Hi,
the variables of the project are indeed not used for the values of fields of the recipe. But you can have a first "Execute Python code" step in the scenario defining your app-as-recipe task, where you fetch the variables of the project from which the recipe is launched, and adjust the variables in the current project as needed, like
from dataiku.scenario import Scenario # get the project from which the app-as-recipe is launched source_project_key = Scenario().get_all_variables()['scenarioTriggerParam_projectKey'] # get its variables, with the public API import dataiku source_project = dataiku.api_client().get_project(source_project_key) source_variables = source_project.get_variables() all_source_variables = dict(source_variables['standard']) all_source_variables.update(dict(source_variables['local'])) # then set some variables for the rest of the scenario execution Scenario().set_scenario_variables(foo=all_source_variables['bar], ...)
-
What I ended up doing was adding a function to all my python recipes within the application to check if the user passed a valid date string or if they passed the name of the global variable storing the date string. It's not ideal but gets the job done.
-
Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 317 NeuronOptionsHello,
As it may be helpful to others, I'm posting code below that takes @fchataigner2
's suggestion a bit further. This code replaces calling project variable references in the app-as-recipe input fields with their calling project values.For example if the variable ${beginDate} is defined in the calling project, you can specify that variable at the Begin Date prompt in an app-as-recipe and it will be replaced with the value of that variable as defined in the calling project and passed into app-as-recipe process.
Marlan
import dataiku from dataiku.scenario import Scenario app_scenario_variables = Scenario().get_all_variables() # project variables plus scenario variables # Get the project from which the app-as-recipe is launched # (If running from within application project, returns application project key so step doesn't fail) call_project_key = app_scenario_variables.get('scenarioTriggerParam_projectKey', dataiku.default_project_key()) # Get calling project variables via the public API call_project_handle = dataiku.api_client().get_project(call_project_key) call_project_variable_sets = call_project_handle.get_variables() call_project_variables = dict(call_project_variable_sets['standard']) call_project_variables.update(dict(call_project_variable_sets['local'])) # Resolve any app variables that have a ${varname} reference in their value updated_app_scenario_variables = {} for app_name, app_value in app_scenario_variables.items(): # Only resolve variables with string values (variable reference can't be int # and not handling nested variables) if isinstance(app_value, str): # Check all calling variables for ${calling_variable_name} references in this app variable value app_value_updated = False for call_name, call_value in call_project_variables.items(): varref = '${'+call_name+'}' if varref in app_value: app_value_updated = True app_value = app_value.replace(varref, call_value) if app_value_updated: updated_app_scenario_variables[app_name] = app_value # Add calling project key (probably will want to define this in app project with default value) updated_app_scenario_variables['callProjectKey'] = call_project_key # Update app scenario variables for the rest of the scenario execution Scenario().set_scenario_variables(**updated_app_scenario_variables)
