Passing global variables to application-as-recipe parameters

sylvyr3 Registered Posts: 21 ✭✭✭✭

I'm trying to develop an application-as-recipe to create a consistent data extract. My plan is to have this application-as-recipe to function for both building a training dataset and for a daily scoring process. The process for building the dataset would use fixed start and end dates but the daily scoring process would use a 7 day moving window. My thought would be to create a scenario where the first step is a custom python script that updates the project's global variables relative to the current date. Then, I would pass the global variable to the application-as-recipe settings page.

Screen Shot 2021-05-25 at 8.43.21 PM.png

Unfortunately it appears that the appilication-as-recipe will not accept global variables here. Any thoughts on how I can pass global variables or otherwise update the dates in an automated method?


  • fchataigner2
    fchataigner2 Dataiker Posts: 355 Dataiker
    edited July 17


    the variables of the project are indeed not used for the values of fields of the recipe. But you can have a first "Execute Python code" step in the scenario defining your app-as-recipe task, where you fetch the variables of the project from which the recipe is launched, and adjust the variables in the current project as needed, like

    from dataiku.scenario import Scenario
    # get the project from which the app-as-recipe is launched
    source_project_key = Scenario().get_all_variables()['scenarioTriggerParam_projectKey']
    # get its variables, with the public API
    import dataiku
    source_project = dataiku.api_client().get_project(source_project_key)
    source_variables = source_project.get_variables()
    all_source_variables = dict(source_variables['standard'])
    # then set some variables for the rest of the scenario execution
    Scenario().set_scenario_variables(foo=all_source_variables['bar], ...)
  • sylvyr3
    sylvyr3 Registered Posts: 21 ✭✭✭✭

    What I ended up doing was adding a function to all my python recipes within the application to check if the user passed a valid date string or if they passed the name of the global variable storing the date string. It's not ideal but gets the job done.

  • Marlan
    Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 317 Neuron
    edited July 17


    As it may be helpful to others, I'm posting code below that takes @fchataigner2
    's suggestion a bit further. This code replaces calling project variable references in the app-as-recipe input fields with their calling project values.

    For example if the variable ${beginDate} is defined in the calling project, you can specify that variable at the Begin Date prompt in an app-as-recipe and it will be replaced with the value of that variable as defined in the calling project and passed into app-as-recipe process.


    import dataiku
    from dataiku.scenario import Scenario
    app_scenario_variables = Scenario().get_all_variables() # project variables plus scenario variables
    # Get the project from which the app-as-recipe is launched 
    # (If running from within application project, returns application project key so step doesn't fail)
    call_project_key = app_scenario_variables.get('scenarioTriggerParam_projectKey', dataiku.default_project_key())
    # Get calling project variables via the public API
    call_project_handle = dataiku.api_client().get_project(call_project_key)
    call_project_variable_sets = call_project_handle.get_variables()
    call_project_variables = dict(call_project_variable_sets['standard'])
    # Resolve any app variables that have a ${varname} reference in their value
    updated_app_scenario_variables = {}
    for app_name, app_value in app_scenario_variables.items():
        # Only resolve variables with string values (variable reference can't be int 
        # and not handling nested variables)
        if isinstance(app_value, str):
            # Check all calling variables for ${calling_variable_name} references in this app variable value
            app_value_updated = False
            for call_name, call_value in call_project_variables.items():
                varref = '${'+call_name+'}'
                if varref in app_value:
                    app_value_updated = True
                    app_value = app_value.replace(varref, call_value)
            if app_value_updated:
                updated_app_scenario_variables[app_name] = app_value
    # Add calling project key (probably will want to define this in app project with default value)
    updated_app_scenario_variables['callProjectKey'] = call_project_key
    # Update app scenario variables for the rest of the scenario execution

  • CoreyS
    CoreyS Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 1,150 ✭✭✭✭✭✭✭✭✭

    Thanks for sharing @Marlan

Setup Info
      Help me…