Passing global variables to application-as-recipe parameters

sylvyr3

I'm trying to develop an application-as-recipe to create a consistent data extract. My plan is to have this application-as-recipe to function for both building a training dataset and for a daily scoring process. The process for building the dataset would use fixed start and end dates but the daily scoring process would use a 7 day moving window. My thought would be to create a scenario where the first step is a custom python script that updates the project's global variables relative to the current date. Then, I would pass the global variable to the application-as-recipe settings page.

Unfortunately it appears that the appilication-as-recipe will not accept global variables here. Any thoughts on how I can pass global variables or otherwise update the dates in an automated method?

fchataigner2

Hi,

the variables of the project are indeed not used for the values of fields of the recipe. But you can have a first "Execute Python code" step in the scenario defining your app-as-recipe task, where you fetch the variables of the project from which the recipe is launched, and adjust the variables in the current project as needed, like

from dataiku.scenario import Scenario
# get the project from which the app-as-recipe is launched
source_project_key = Scenario().get_all_variables()['scenarioTriggerParam_projectKey']
# get its variables, with the public API
import dataiku
source_project = dataiku.api_client().get_project(source_project_key)
source_variables = source_project.get_variables()
all_source_variables = dict(source_variables['standard'])
all_source_variables.update(dict(source_variables['local']))
# then set some variables for the rest of the scenario execution
Scenario().set_scenario_variables(foo=all_source_variables['bar], ...)

sylvyr3

What I ended up doing was adding a function to all my python recipes within the application to check if the user passed a valid date string or if they passed the name of the global variable storing the date string. It's not ideal but gets the job done.

Marlan

Hello,

As it may be helpful to others, I'm posting code below that takes @fchataigner2
's suggestion a bit further. This code replaces calling project variable references in the app-as-recipe input fields with their calling project values.

For example if the variable ${beginDate} is defined in the calling project, you can specify that variable at the Begin Date prompt in an app-as-recipe and it will be replaced with the value of that variable as defined in the calling project and passed into app-as-recipe process.

Marlan

import dataiku
from dataiku.scenario import Scenario

app_scenario_variables = Scenario().get_all_variables() # project variables plus scenario variables

# Get the project from which the app-as-recipe is launched 
# (If running from within application project, returns application project key so step doesn't fail)
call_project_key = app_scenario_variables.get('scenarioTriggerParam_projectKey', dataiku.default_project_key())


# Get calling project variables via the public API
call_project_handle = dataiku.api_client().get_project(call_project_key)

call_project_variable_sets = call_project_handle.get_variables()
call_project_variables = dict(call_project_variable_sets['standard'])
call_project_variables.update(dict(call_project_variable_sets['local']))

# Resolve any app variables that have a ${varname} reference in their value
updated_app_scenario_variables = {}
for app_name, app_value in app_scenario_variables.items():

    # Only resolve variables with string values (variable reference can't be int 
    # and not handling nested variables)
    if isinstance(app_value, str):

        # Check all calling variables for ${calling_variable_name} references in this app variable value
        app_value_updated = False
        for call_name, call_value in call_project_variables.items():
            varref = '${'+call_name+'}'
            if varref in app_value:
                app_value_updated = True
                app_value = app_value.replace(varref, call_value)

        if app_value_updated:
            updated_app_scenario_variables[app_name] = app_value


# Add calling project key (probably will want to define this in app project with default value)
updated_app_scenario_variables['callProjectKey'] = call_project_key

# Update app scenario variables for the rest of the scenario execution
Scenario().set_scenario_variables(**updated_app_scenario_variables)

CoreyS

Thanks for sharing @Marlan
!