Possible to use LLM Mesh connection profiles in (python) APIs?

[Antal]

I'm playing around with the new LLM Mesh connection profiles functionality.

A lot of LLM use cases we have require the data product to be deployed as an API, so in those cases it's not an option to use LLM recipes.

It would however be really beneficial to use the LLM connection profiles in those APIs to make sharing/maintaining/securing LLM connections much easier.

Is it possible to 'import' those connection profiles in python API code to easily reuse pre-defined LLM connections even in custom APIs?

---

Operating system used: AWS Linux

[importthepandas]

I've a similar query here: https://community.dataiku.com/t5/Plugins-Extending-Dataiku/LLM-Service-Plugin/td-p/39018

Probably a product idea!

[Marlan]

Hi @Antal,

I believe calling a LLM connection through the Dataiku API is possible. 

The Project class now has a get_llm method. The returned object should let you use the connection programmatically. There appear to be a couple of layers of objects to get to the point of getting a completion from the LLM. I say "should" here because while I thought I had worked out how to do this, it doesn't yet work. I suspect I'm not using the right name/ID when calling the get_llm method. 

In any case, we are also highly interested in this capability and hope to have it worked out soon.

Marlan

cc @importthepandas 

[Antal]

Interesting @Marlan 

Yes, I tried something similar. But indeed it tells me it's not a valid llm_id. So how do we find the llm_id of our defined llm connections? And how do we tell it what deployment to use (davinci vs gpt-turbo for example)

Also, is the project class available when executing in a python API endpoint on one of the API nodes?

code example:

llm_conn = project.get_llm('LLM connection name')

llm_comp = llm_conn.new_completion()

llm_comp.cq = {
    'messages': [
        {'role': 'system', 'content': 'You are a helpful assistant'},
        {'role': 'user', 'content': 'Who is Mahatma Gandhi?'}
    ],
    'settings': {}
}

llm_comp.execute()

[Marlan]

Hi @Antal, looks like we tried pretty much the same thing and ran into the same error. We just need to figure out what the magic llm_id is. I tried looking in logs for a hint but no luck there.

Marlan

[Antal]

I got a bit further with this and got it to work in the Design node at least.

But I don't think this will work from an API node. I'll give it a try.

[EDIT] No, as the project in question doesn't exist as such on an API node list_llms() fails due to no connections found

import dataiku

client = dataiku.api_client()

project = client.get_project(dataiku.default_project_key())

llm_list = project.list_llms()

llm_id = [llm for llm in llm_list if llm['connection'] == 'connection_name' and llm['deployment'] == 'deployment_name'][0]['id']

llm_conn = project.get_llm(llm_id)

llm_comp = llm_conn.new_completion()

llm_comp.cq = {
    'messages': [
        {'role': 'system', 'content': 'You are a helpful assistant'},
        {'role': 'user', 'content': 'Who is Mahatma Gandhi?'}
    ],
    'settings': {}
}

response = llm_comp.execute()

[Marlan]

Nice work @Antal! We aren't on the latest version yet so don't have the list_llms() in the API yet. I did get information from Dataiku that structure of LLM ID is: LLM type (lower case):connection name:deployment name. Example: "azureopenai:Azure_OpenAI:gpt-35-turbo". I assume that is what you are getting in your logic.

Marlan

[Antal]

I can indeed confirm that that is the case @Marlan.

Now if only we could just as easily import/load  and use these connections in an API endpoint. I guess we'll have to wait for LLM Mesh to go out of public preview for full documentation on functionality and such.