Folder access error in API endpoint
In the API endpoint script, I get an error when trying to access a folder which is placed in the resources of my environment, it's a sentence transformer model which I need to access.
Can someone please suggest? 
Answers
-
In the API code I'm accessing this env in this way
sentence_transformer_home = os.getenv('SENTENCE_TRANSFORMERS_HOME') model_path = os.path.join(sentence_transformer_home, 'sentence-transformers_all-MiniLM-L6-v2') model = SentenceTransformer(model_path) -
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,757 NeuronOptions
Check the permissions of that folder / file and if exists in your API node. It looks like you may need to download the transformers model first before you can invoke it in your function. The API node is a different machine so it won’t have the same thing your Designer node has.
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,757 NeuronOptions
What is
SENTENCE_TRANSFORMERS_HOME
set to? And have you installed the transformers model in this folder?
-
It's
also yes, I have downloaded the model
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,757 NeuronOptions
As you can see by the error the code is looking for the model in a tmp directory Dataiku uses to deploy the service. This is not desirable. You should download the model in a generic location in your API node machine and then use an absolute path to reference it in your API service code. That way any service using the transformers model can use the same downloaded model.
-
That makes sense, thank you @Turribeach
, I'll try this approach.