LLM

mebinjoy
Level 1
LLM

Hi ,

Is there a way to use LLM connections and models  as a API node?



Operating system used: Windows

0 Kudos
3 Replies
AlexT
Dataiker

Hi,
Currently, you can use the LLM APIs directly in a custom python API endpoint.
This may evolve in the future.

You would need to leverage set_remote_dss https://developer.dataiku.com/latest/getting-started/outside-usage.html as API nodes are outside of DSS

For the host/API key note you can store these env variables to avoid storing in code
A) on static API node you can set env variables 
B) Using Resource init script for the code env for API endpoints deployed on K8s.

from dataiku.code_env_resources import set_env_var
set_env_var(“DKU_API_KEY”, “xxxx”)
set_env_var(“DKU_DSS_URL”, “https://xxxxxxx”)

Then use read the env variables an use LLM APIs: https://developer.dataiku.com/latest/concepts-and-examples/llm-mesh.html

0 Kudos
mebinjoy
Level 1
Author

Hi Alex , I'm getting this error in the api designer :

 No connection adapters were found for '127.0.0.1:51442/dip/publicapi/projects/TUT_REAL_TIME/llms/completions'

0 Kudos
AlexT
Dataiker

Hi @mebinjoy,

Can you open a support ticket and share the Instance diagnostics and a zip of the api package from the version tab of API Designer?

Thanks 

0 Kudos