LLM

mebinjoy

Hi ,

Is there a way to use LLM connections and models as a API node?

---

---

Operating system used: Windows

Alexandru

Hi,
Currently, you can use the LLM APIs directly in a custom python API endpoint.
This may evolve in the future.

You would need to leverage set_remote_dss https://developer.dataiku.com/latest/getting-started/outside-usage.html as API nodes are outside of DSS

For the host/API key note you can store these env variables to avoid storing in code
A) on static API node you can set env variables
Using Resource init script for the code env for API endpoints deployed on K8s.

from dataiku.code_env_resources import set_env_var
set_env_var(“DKU_API_KEY”, “xxxx”)
set_env_var(“DKU_DSS_URL”, “https://xxxxxxx”)

Then use read the env variables an use LLM APIs: https://developer.dataiku.com/latest/concepts-and-examples/llm-mesh.html

mebinjoy

Hi Alex , I'm getting this error in the api designer :

No connection adapters were found for '127.0.0.1:51442/dip/publicapi/projects/TUT_REAL_TIME/llms/completions'

Alexandru

Hi @mebinjoy
,

Can you open a support ticket and share the Instance diagnostics and a zip of the api package from the version tab of API Designer?

Thanks

TH99

Hi @mebinjoy
,

Were you able to solve it? I have the exact same error.