Capturing Python logging output in a file

Options
Marlan
Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 317 Neuron
Looking for a way to capture Python logging output from a recipe in a file. I explored directing output to a file in a managed folder but writing to a managed folder requires use of the API. Any other ideas?

Answers

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer, Registered Posts: 753 Dataiker
    Options
    Hi,

    If your managed folder is on a "Local filesystem" connection, you can write directly in it.

    * First, get the base path of the managed folder with dataiku.Folder("xxxxxx").get_path()
    * Then you can write into it using the regular Python file API

    So you can setup the logging module to log to a file whose path you have computed first from get_path()

    https://docs.python.org/2/howto/logging.html
  • Marlan
    Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 317 Neuron
    Options
    Thanks Clement!

    This worked as advertised for a recipe that already had a managed folder as an output but wanted to share some additional detail about other situations that may be useful to others.

    We are running our instance in multi-user-security mode and in this case write access on a managed folder is only granted when said managed folder is defined as output of the recipe. So the above situation worked fine. For other Python recipes however, a new managed folder must be created for each to store the output of the Python recipe.
Setup Info
    Tags
      Help me…