Setup logging for remotely running python scripts

plamina · June 13

Hey there, I am trying to setup python logging, but am not able to configure the logging file saving using the Dataiku APIs.

What would be the equivalent for remote of this setup for locally run python processes:

import logging
logging.basicConfig(level=logging.INFO)
managed_folder_path = dataiku.Folder("logs").get_path()
logging.root.addHandler(logging.FileHandler(os.path.join(managed_folder_path, "log.log")))

Thanks!

Catalina · June 13

Hi @plamina
,

When you execute this code in a notebook or as a recipe on DSS, the code has access to the filesystem of the server and your FileHandler can access the server filesystem and write the logs there as they are coming in.

If you want to execute a similar code remotely, then your code will not have access to the filesystem of the server and even if you have the managed folder path you will not be able to open a file handler.

One solution to this is to write the logs to a tempfile and at the end of the script use the dataikuapi to upload the tempfile to the managed folder.