Managed Folder with Container Execution

Astrogurl Registered Posts: 2

I am trying to run a python recipe and have a model saved in a managed folder. I understand that I have to use get_download_stream() to read the data, but the python module that I need to use (FAISS) does not support reading the saved model as bytes. Is there a way that I can download the file and obtain a path, so that I can just feed in the path to the python module during the containerized execution?

Operating system used: Mac OS Monterey


Best Answer

  • Zach
    Zach Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 153 Dataiker
    Answer ✓

    Hi @Astrogurl

    The following code will download the file to a temporary directory first so that you can pass the path to FAISS:

    import os.pathimport shutilimport tempfileimport dataikufolder = dataiku.Folder("FOLDER")with tempfile.TemporaryDirectory() as temp_dir:path = os.path.join(temp_dir, "my-file.txt")# Download the remote file to `path`with folder.get_download_stream("/my-file.txt") as download_stream:with open(path, "wb") as local_file:shutil.copyfileobj(download_stream, local_file)# Do stuff with the temp file here# It will be automatically deleted when the `temp_dir` block finishesprint(path)




Setup Info
      Help me…