Setup logging for remotely running python scripts

plamina

Hey there, I am trying to setup python logging, but am not able to configure the logging file saving using the Dataiku APIs.

What would be the equivalent for remote of this setup for locally run python processes:

import logging
logging.basicConfig(level=logging.INFO)
managed_folder_path = dataiku.Folder("logs").get_path()
logging.root.addHandler(logging.FileHandler(os.path.join(managed_folder_path, "log.log")))

Thanks!

Catalina

Hi @plamina
,

When you execute this code in a notebook or as a recipe on DSS, the code has access to the filesystem of the server and your FileHandler can access the server filesystem and write the logs there as they are coming in.

If you want to execute a similar code remotely, then your code will not have access to the filesystem of the server and even if you have the managed folder path you will not be able to open a file handler.

One solution to this is to write the logs to a tempfile and at the end of the script use the dataikuapi to upload the tempfile to the managed folder.

Turribeach

This example shows how to create a temp file and then uploaded to the Folder using remote compatible methods:

https://knowledge.dataiku.com/latest/code/managed-folders/tutorial-managed-folders.html

This shows how to create a temporary folder:

https://developer.dataiku.com/12/concepts-and-examples/managed-folders.html

This article describes how to stream logging events from the API node and use them to create a feedback loop in your model:

https://knowledge.dataiku.com/latest/mlops-o16n/model-monitoring/tutorial-api-endpoint-monitoring-basics.html