Survey banner
Switching to Dataiku - a new area to help users who are transitioning from other tools and diving into Dataiku! CHECK IT OUT

Folder access error in API endpoint

sdp
Level 2
Folder access error in API endpoint

In the API endpoint script, I get an error when trying to access a folder which is placed in the resources of my environment, it's a sentence transformer model which I need to access.

Can someone please suggest? 
Screenshot 2024-04-30 161656.png

0 Kudos
6 Replies
sdp
Level 2
Author

In the API code I'm accessing this env in this way 

sentence_transformer_home = os.getenv('SENTENCE_TRANSFORMERS_HOME')
model_path = os.path.join(sentence_transformer_home, 'sentence-transformers_all-MiniLM-L6-v2')
model = SentenceTransformer(model_path)
0 Kudos
Turribeach

Check the permissions of that folder / file and if exists in your API node. It looks like you may need to download the transformers model first before you can invoke it in your function. The API node is a different machine so it won’t have the same thing your Designer node has. 

0 Kudos

What is 

SENTENCE_TRANSFORMERS_HOME

set to? And have you installed the transformers model in this folder?

0 Kudos
sdp
Level 2
Author
 

It's 
Screenshot 2024-05-02 101109.png

also yes, I have downloaded the model 
Screenshot 2024-05-02 101220.png

0 Kudos

As you can see by the error the code is looking for the model in a tmp directory Dataiku uses to deploy the service. This is not desirable. You should download the model in a generic location in your API node machine and then use an absolute path to reference it in your API service code. That way any service using the transformers model can use the same downloaded model. 

0 Kudos
sdp
Level 2
Author

That makes sense, thank you @Turribeach, I'll try this approach. 

0 Kudos