Jupyter Notebook and python logging in DSS

Hi,

Both storing in a dataset or in a managed folder make sense. If these are indeed logs (meaning stuff that is useful to understand what happened, but not required as long as everything is working OK), a dataset may be a slightly atypical idea.

We would indeed recommend the use of Python's "logging" library which is designed exactly for that: your program emits multiple log items with an associated "severity" or "level" (debug, info, warning, error), and then the library will emit these log items to various outputs, with possible severity filters.

The logging library has multiple outputs. The default one is "console", meaning that it does the equivalent of "print", so it will appear directly in the notebook. But you could indeed create a "file" appender in Python logging, that would write into a managed folder file.

Assuming that you are using a local managed folder, that could look like:

logging.basicConfig(level=logging.DEBUG)
managed_folder_path = dataiku.Folder("myfolder").get_path()
logging.root.addHandler(logging.FileHandler(os.path.join(managed_folder_path, "log.log"))

logging.info("hello") <-- This will print both in console and in log.log in the managed folder
    

You can also play with formats in logging in order to make sure to print timestamps, severities, ...

However, for a process that you are going to repeat multiple times a day, we would recommend that you use DSS jobs rather than notebooks. DSS will automatically store the "console" output of each job, without need for you to manage it (but you would still use the logging library to dispatch by severity)