Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
I am trying to run a python recipe and have a model saved in a managed folder. I understand that I have to use get_download_stream() to read the data, but the python module that I need to use (FAISS) does not support reading the saved model as bytes. Is there a way that I can download the file and obtain a path, so that I can just feed in the path to the python module during the containerized execution?
Operating system used: Mac OS Monterey
Hi @Astrogurl,
The following code will download the file to a temporary directory first so that you can pass the path to FAISS:
import os.path
import shutil
import tempfile
import dataiku
folder = dataiku.Folder("FOLDER")
with tempfile.TemporaryDirectory() as temp_dir:
path = os.path.join(temp_dir, "my-file.txt")
# Download the remote file to `path`
with folder.get_download_stream("/my-file.txt") as download_stream:
with open(path, "wb") as local_file:
shutil.copyfileobj(download_stream, local_file)
# Do stuff with the temp file here
# It will be automatically deleted when the `temp_dir` block finishes
print(path)
Thanks,
Zach
Hi @Astrogurl,
The following code will download the file to a temporary directory first so that you can pass the path to FAISS:
import os.path
import shutil
import tempfile
import dataiku
folder = dataiku.Folder("FOLDER")
with tempfile.TemporaryDirectory() as temp_dir:
path = os.path.join(temp_dir, "my-file.txt")
# Download the remote file to `path`
with folder.get_download_stream("/my-file.txt") as download_stream:
with open(path, "wb") as local_file:
shutil.copyfileobj(download_stream, local_file)
# Do stuff with the temp file here
# It will be automatically deleted when the `temp_dir` block finishes
print(path)
Thanks,
Zach
Thank you @ZachM, this works!