How I can access recipe's SQL output using python api client

Options
Gustavo_Brian
Gustavo_Brian Registered Posts: 10 ✭✭✭✭

I'd like to access SQL output generated by a recipe programatically using Python API client.

Is that possible?

UPDATE:

I see this information is returned by /dip/api/flow/recipes/generic/get-status

Tagged:

Answers

  • UserBird
    UserBird Dataiker, Alpha Tester Posts: 535 Dataiker
    edited 11:38AM
    Options

    Hi,

    You find below a small code snippet from which you can inspire to apply to your use case:



    # HELPER FUNCTION
    CODE_RECIPES = ['sql_script', 'sql_query', 'impala', 'hive', 'spark_sql']
    def get_recipe_sql(self, project_handle, recipe_name, recipe_type):
    recipe_handle = project_handle.get_recipe(recipe_name)
    if recipe_type in CODE_RECIPES:
    dap = recipe_handle.get_definition_and_payload()
    return dap.get_payload()
    else:
    status = recipe_handle.get_status()
    return status.data.get('sql')


    # GO
    client = dataiku.api_client()
    project_handle = client.get_project('MY_PROJECT')
    recipes = project_handle.list_recipes()

    #loop through every recipe of the project and print out the SQL if possible
    for recipe in recipes:
    recipe_name = recipe.get('name')
    recipe_type = recipe.get('type')
    try:
    sql = get_recipe_sql(project_handle, recipe_name, recipe_type)
    except Exception as e:
    message = "Could not retrieve sql code from recipe {} in project {}: {}"
    print(message.format(recipe_name, project, e))
    continue

    Cheers,

  • Gustavo_Brian
    Gustavo_Brian Registered Posts: 10 ✭✭✭✭
    Options
    So close!! Most of the recipes are returning the SQL, thanks.
    Sadly the one at the screenshot is failing with:

    {u'allMessagesForFrontend': {u'anyMessage': True,
    u'error': True,
    u'maxSeverity': u'ERROR',
    u'messages': [{u'code': u'ERR_RECIPE_VALIDATION_FAILED',
    u'details': u'Failed to compute recipe status: NullPointerException',
    u'message': u'Validation failed: Failed to compute recipe status: NullPointerException',
    u'severity': u'ERROR',
    u'title': u'Validation failed'}],
    u'success': False,
    u'warning': False},
    u'engines': [],
    u'output': {u'anyMessage': False,
    u'error': False,
    u'messages': [],
    u'success': False,
    u'warning': False},
    u'topLevelMessages': {u'anyMessage': True,
    u'error': True,
    u'maxSeverity': u'ERROR',
    u'messages': [{u'code': u'ERR_RECIPE_VALIDATION_FAILED',
    u'details': u'Failed to compute recipe status: NullPointerException',
    u'message': u'Validation failed: Failed to compute recipe status: NullPointerException',
    u'severity': u'ERROR',
    u'title': u'Validation failed'}],
    u'success': False,
    u'warning': False}}
Setup Info
    Tags
      Help me…