Survey banner
The Dataiku Community is moving to a new home! We are temporary in read only mode: LEARN MORE

Tracing Variables

Solved!
nharbour
Level 1
Tracing Variables

Is there a feature to trace the use of variables in Dataiku DSS? It would be very useful to be able to automate a search through Projects to identify where it's been used, so we can evaluate the impact of any change to that variable.

0 Kudos
1 Solution
Turribeach

Hi, there are no built-in features for this but you can search inside recipe code using the Python API: 

 

 

import dataiku

client_handle = dataiku.api_client()
variables_to_search = ['var1', 'var2']
project_handle = client_handle.get_project('some project key')

python_recipes = [i for i in project_handle.list_recipes() if i['type'] in ['python']]

for python_recipe in python_recipes:
    recipe_name = python_recipe['name']
    recipe_handle = project_handle.get_recipe(recipe_name)
    recipe_script = recipe_handle.get_settings().get_payload().lower()
    if recipe_script:
        for var in variables_to_search:
            if var.lower() in recipe_script:
                print(f'Found variable {var} in recipe {recipe_name}')

 

 


This being a text search will likely lead to false positives depending on how unique your variables names are. The same code can be used to loop through all projects in your DSS instance but this may take a while to run depending on your instance size.

A less advanced and more basic way will be to export the project, download the export, unzip it and then use Notepad++ Find in Files functionality to search for variables in all the project data files.

View solution in original post

0 Kudos
2 Replies
Turribeach

Hi, there are no built-in features for this but you can search inside recipe code using the Python API: 

 

 

import dataiku

client_handle = dataiku.api_client()
variables_to_search = ['var1', 'var2']
project_handle = client_handle.get_project('some project key')

python_recipes = [i for i in project_handle.list_recipes() if i['type'] in ['python']]

for python_recipe in python_recipes:
    recipe_name = python_recipe['name']
    recipe_handle = project_handle.get_recipe(recipe_name)
    recipe_script = recipe_handle.get_settings().get_payload().lower()
    if recipe_script:
        for var in variables_to_search:
            if var.lower() in recipe_script:
                print(f'Found variable {var} in recipe {recipe_name}')

 

 


This being a text search will likely lead to false positives depending on how unique your variables names are. The same code can be used to loop through all projects in your DSS instance but this may take a while to run depending on your instance size.

A less advanced and more basic way will be to export the project, download the export, unzip it and then use Notepad++ Find in Files functionality to search for variables in all the project data files.

0 Kudos
nharbour
Level 1
Author

Thank you for your quick response. That's very useful.

0 Kudos