How do i read a pickle file from a managed folder
ssuhas76
Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 31 ✭
Hello there,
I am trying to read a pickle file from a managed folder in the dataiku api endpoint code.
client = dataikuapi.DSSClient(host, apiKey)
client._session.verify = False
project = client.get_project("xxxxxx")
variables = project.get_variables()['standard']
managedfolder = project.get_managed_folder("xxxxxxxxxxxxxxx")
modelid = 'babababababababababab'
with managedfolder.get_file(f"models/{modelid}.pkl") as fd:
loaded_model = pickle.load(open(f"models/{modelid}.pkl", 'rb'))
I am facing a issue saying the file doesn't exist. Please help
Best Answer
-
JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 296 Dataiker
Hi @ssuhas76
,If the pickle file is on a non-local managed folder (i.e., S3), please use the following for loading & saving:
import dataiku import pickle remote_folder = dataiku.Folder("pkl-models") # This is a managed folder on S3 connection # Save pickle with remote_folder.get_writer("/test-model.pkl") as writer: pickle.dump(clf, writer) # Assuming clf is a sklearn object # Load pickle with remote_folder.get_download_stream('test-model.pkl') as f: clf_loaded = pickle.load(f)
If the pickle file is on a local managed folder, please try this:
import pickle import dataiku folder = dataiku.Folder("folderID").get_path() model_path = folder + "/model.pkl" with open(model_path, 'rb') as file: clf = pickle.load(file)
Thanks!
Jordan