Is there a list of all the builtin variables like ${projectKey}, ${dssUserLogin}, etc

ecerulm
ecerulm Registered Posts: 45 ✭✭✭✭✭

I noticed the user of some builtin variables in the documentation like:

  • projectKey jobProjectKey
  • dssUserLogin
  • jobProjectKey
  • jobId
  • activityId

but it's there a list somewhere with all the possibilities, and where they can be used?

For example is not clear to me if I can use ${jobProjectKey} in the Administration > Settings > Compute & Scaling > Conteinerized execution > Containerized execution configs > Kubernetes namespace. it just say that "Variable expansion is supported" but what variables can be used there exactly?

I know I can use dsUserLogin since it's mention explicitly in https://doc.dataiku.com/dss/latest/containers/namespaces.html but what other variables are available?

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,225 Dataiker
    edited July 17

    Hi @ecerulm
    ,

    There isn't a single centralized list of which built-in variables will depend on the context ( job, scenario, webapp, notebook, etc.)

    You can see variables available in specific recipes/scenarios e.g:
    Screenshot 2023-09-09 at 8.36.08 AM.png

    Priority of variables is explained in more detail here when variables expansion is supported: https://doc.dataiku.com/dss/latest/connecting/sql/snowflake.html#how-to-set-it-up
    Recipe > Scenario > Project > User > Global ( instance)

    You can always print all other variables available. You can use the datiaku API to see the difference from a notebook/recipe/scenario.

    # get client
    import dataiku
    client = dataiku.api_client() 
    
    #When running a job print available variables( excludes recipe variables)
    print(dataiku.get_custom_variables())
    
    # print instance global variables
    global_variables = client.get_global_variables()
    print(global_variables)
    
    
    #project variables
    project = client.get_project("PROJECT_KEY")
    #there are standard + local project variables 
    project_variables = project.get_variables()
    print(project_variables)
    
    #user properties 
    user = client.get_user("user_id")
    print(user.get_settings().get_raw()['userProperties'])
    

    For a scenario, you can use the following to print all variables:

    from dataiku.scenario import Scenario
    import json
    print(json.dumps(Scenario().get_all_variables(), indent=2))


    For the case of the namespace, while all variables are expanded the built-in variables available will depend on the context.

    You should opt for variables available in all contexts, including notebooks and recipes. Otherwise, the containerized execution could not be both recipes and notebooks. The built-in variables jobProjectKey / jobId are NOT available in notebooks for example

    The built-in variables you can use safely are projectKey and dssUserLogin. You can also use global variables user properties or project variables.

    Thanks,

  • ecerulm
    ecerulm Registered Posts: 45 ✭✭✭✭✭

    so running your script in a notebook it only shows the following variables

    • dku.install.dir
    • projectKey
    • dip.home

    I don't see dssUserLogin there.

    So I would really like to see a list of contexts and the variables available in each of those contexts. LIke

    Notebooks

    • ${dku.install.dir}
    • ${dip.home}
    • ${projectKey}
    • dssUserLogin not available here ??
    • project variables, user variables and global variables

    Jobs

    • projectKey (??)
    • dssUserLogin
    • jobProjectKey (how is this different from projectKey?)
    • jobId
    • activityId ??
    • project variables, user variables and global variables
    • other ??

    the other contexts

    I think having such reference would be a good addition to the dataiku documentation. I'm not particularly keen on having to run code to find out what variables exists in each context, I think is way better to have them documented.

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,043 Neuron

    @info-rchitect
    This could be a very clunky way to figure out the context where the code is running as it seems there are different variables exposed at each context.

    https://community.dataiku.com/t5/Using-Dataiku/Understanding-if-Python-code-is-running-in-a-recipe-scenario-web/m-p/37493#M13842

  • info-rchitect
    info-rchitect Registered Posts: 180 ✭✭✭✭✭✭

    @Turribeach
    I will code something up and share it with the community if it works out.

  • SebasBer
    SebasBer Registered Posts: 7 ✭✭✭

    I agree with this, having the variables documented is preferable to having to run code to see what variables are available.

Setup Info
    Tags
      Help me…