LLM

[mebinjoy]

Hi ,

Is there a way to use LLM connections and models  as a API node?

---

---

Operating system used: Windows

[AlexT]

Hi,
Currently, you can use the LLM APIs directly in a custom python API endpoint.
This may evolve in the future.

You would need to leverage set_remote_dss https://developer.dataiku.com/latest/getting-started/outside-usage.html as API nodes are outside of DSS

For the host/API key note you can store these env variables to avoid storing in code
A) on static API node you can set env variables 
B) Using Resource init script for the code env for API endpoints deployed on K8s.

from dataiku.code_env_resources import set_env_var
set_env_var(“DKU_API_KEY”, “xxxx”)
set_env_var(“DKU_DSS_URL”, “https://xxxxxxx”)

Then use read the env variables an use LLM APIs: https://developer.dataiku.com/latest/concepts-and-examples/llm-mesh.html

[mebinjoy]

Hi Alex , I'm getting this error in the api designer :

 No connection adapters were found for '127.0.0.1:51442/dip/publicapi/projects/TUT_REAL_TIME/llms/completions'

[AlexT]

Hi @mebinjoy,

Can you open a support ticket and share the Instance diagnostics and a zip of the api package from the version tab of API Designer?

Thanks