Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hello, I am trying to get the printing of some messages to a log file (log.txt) in logfolder (a managed folder in DSS, connects to sharepoint folder).
Error ==>
Line 128: managed_folder_path = dataiku.Folder("logfolder").get_path()
Job failed: Error in Python process: At line 128: <class 'Exception'>: Folder is not on the local filesystem (uses fsprovider_sharepoint-server_sharepoint-server_shared-documents), cannot perform direct filesystem access. Use the read/write API instead.
My code ==>
import logging
import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu
from datetime import datetime
import os.path
logging.basicConfig(level=logging.DEBUG)
managed_folder_path = dataiku.Folder("logfolder").get_path()
logging.root.addHandler(logging.FileHandler(os.path.join(managed_folder_path, "log.txt")))
logging.info("hello")
Operating system used: Unix
Finally I am able to wrap my head around and create filesystem folder instead (after that I could write morecode to copy that file back to sharepoint):
- In Flow, + Dataset > Folder.
- Label = logfolder
- Store into = filesystem_folders
- Paritioning = Not partitioned
- Set output to logfolder from Python recipe
Code ==>
import logging
import dataiku
import os.path
logger = logging.getLogger("mylogger")
if logger.hasHandlers():
logger.handlers = []
logger.setLevel(logging.DEBUG)
#create File for Log
handler = logging.FileHandler(os.path.join(managed_folder_path, "log.txt"))
handler.setLevel(logging.DEBUG)
#log format
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
#adding the handler to Logging System
logger.addHandler(handler)
logger.error("hello")
Hi,
The 'get_path' function can only be used in the specific case of local folders. Information about this function can be found on this article: https://doc.dataiku.com/dss/latest/connecting/managed_folders.html#local-vs-non-local
The error message we are getting is quite informative about it too:
Folder is not on the local filesystem (uses fsprovider_sharepoint-server_sharepoint-server_shared-documents), cannot perform direct filesystem access
Fortunately, it is also telling us what we can do:
Use the read/write API instead.
Information about the API can be found in the help: https://doc.dataiku.com/dss/latest/python-api/index.html#python-apis
Thanks on the reply. Could you point me to an API method that is able to return the path of a non-local folder?
On the other hand, maybe I can save locally to DSS and re-copy that log.txt file back to sharepoint?
Hi @bon2bone,
You are receiving this error ("Folder is not on the local filesystem... Use the read/write API instead") because the get_path() method can only be called for managed folders that are stored on the local filesystem of the DSS server. I recommend taking a look at our documentation on Local vs. Non-Local Managed Folders and instead using the various read/download and write/upload APIs to access your managed folder. Please see the code below for example:
folder = dataiku.Folder("enter-folder-id/name")
with open("local_file_to_upload") as f: folder.upload_stream("name_of_file_in_folder", f)
API reference documentation: https://doc.dataiku.com/dss/latest/python-api/managed_folders.html#managed-folders
If you have any questions, please let us know.
Thanks!
Jordan
Hello, thanks on the reply.
For the code below, what is the local file to upload? I am writing text into log.txt only...
Code ==>
with open("local_file_to_upload") as f:
Hi @bon2bone,
Apologies, the code I provided writes a file to the managed folder. If you want to write data to a file within a managed folder, you could do something like this:
import dataiku
my_str = "hello world"
my_str_as_bytes = str.encode(my_str)
logging_folder = dataiku.Folder("VYKX8FzU")
with logging_folder.get_writer("log.txt") as w:
w.write(my_str_as_bytes)
Please let me know if that works for you.
Thanks!
Jordan
Could I have the writer in append mode? E.g. When I rerun the script the 2nd time, log.txt file should have "hello world" 2 times:
hello world
hello world
Finally I am able to wrap my head around and create filesystem folder instead (after that I could write morecode to copy that file back to sharepoint):
- In Flow, + Dataset > Folder.
- Label = logfolder
- Store into = filesystem_folders
- Paritioning = Not partitioned
- Set output to logfolder from Python recipe
Code ==>
import logging
import dataiku
import os.path
logger = logging.getLogger("mylogger")
if logger.hasHandlers():
logger.handlers = []
logger.setLevel(logging.DEBUG)
#create File for Log
handler = logging.FileHandler(os.path.join(managed_folder_path, "log.txt"))
handler.setLevel(logging.DEBUG)
#log format
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
#adding the handler to Logging System
logger.addHandler(handler)
logger.error("hello")
Thanks for sharing your solution with the rest of the Community @bon2bone!