Possible to use LLM Mesh connection profiles in (python) APIs?

Options
Antal
Antal Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 85 Neuron

I'm playing around with the new LLM Mesh connection profiles functionality.

A lot of LLM use cases we have require the data product to be deployed as an API, so in those cases it's not an option to use LLM recipes.

It would however be really beneficial to use the LLM connection profiles in those APIs to make sharing/maintaining/securing LLM connections much easier.

Is it possible to 'import' those connection profiles in python API code to easily reuse pre-defined LLM connections even in custom APIs?


Operating system used: AWS Linux

Answers

  • importthepandas
    importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuron
    Options
  • Marlan
    Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 316 Neuron
    Options

    Hi @Antal
    ,

    I believe calling a LLM connection through the Dataiku API is possible.

    The Project class now has a get_llm method. The returned object should let you use the connection programmatically. There appear to be a couple of layers of objects to get to the point of getting a completion from the LLM. I say "should" here because while I thought I had worked out how to do this, it doesn't yet work. I suspect I'm not using the right name/ID when calling the get_llm method.

    In any case, we are also highly interested in this capability and hope to have it worked out soon.

    Marlan

    cc @importthepandas

  • Antal
    Antal Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 85 Neuron
    Options

    Interesting @Marlan

    Yes, I tried something similar. But indeed it tells me it's not a valid llm_id. So how do we find the llm_id of our defined llm connections? And how do we tell it what deployment to use (davinci vs gpt-turbo for example)

    Also, is the project class available when executing in a python API endpoint on one of the API nodes?

    code example:

    llm_conn = project.get_llm('LLM connection name')

    llm_comp = llm_conn.new_completion()

    llm_comp.cq = {
    'messages': [
    {'role': 'system', 'content': 'You are a helpful assistant'},
    {'role': 'user', 'content': 'Who is Mahatma Gandhi?'}
    ],
    'settings': {}
    }

    llm_comp.execute()

  • Marlan
    Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 316 Neuron
    Options

    Hi @Antal
    , looks like we tried pretty much the same thing and ran into the same error. We just need to figure out what the magic llm_id is. I tried looking in logs for a hint but no luck there.

    Marlan

  • Antal
    Antal Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 85 Neuron
    Options

    I got a bit further with this and got it to work in the Design node at least.

    But I don't think this will work from an API node. I'll give it a try.

    [EDIT] No, as the project in question doesn't exist as such on an API node list_llms() fails due to no connections found

    import dataiku

    client = dataiku.api_client()

    project = client.get_project(dataiku.default_project_key())

    llm_list = project.list_llms()

    llm_id = [llm for llm in llm_list if llm['connection'] == 'connection_name' and llm['deployment'] == 'deployment_name'][0]['id']

    llm_conn = project.get_llm(llm_id)

    llm_comp = llm_conn.new_completion()

    llm_comp.cq = {
    'messages': [
    {'role': 'system', 'content': 'You are a helpful assistant'},
    {'role': 'user', 'content': 'Who is Mahatma Gandhi?'}
    ],
    'settings': {}
    }

    response = llm_comp.execute()

  • Marlan
    Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 316 Neuron
    Options

    Nice work @Antal
    ! We aren't on the latest version yet so don't have the list_llms() in the API yet. I did get information from Dataiku that structure of LLM ID is: LLM type (lower case):connection name:deployment name. Example: "azureopenai:Azure_OpenAI:gpt-35-turbo". I assume that is what you are getting in your logic.

    Marlan

  • Antal
    Antal Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 85 Neuron
    Options

    I can indeed confirm that that is the case @Marlan
    .

    Now if only we could just as easily import/load and use these connections in an API endpoint. I guess we'll have to wait for LLM Mesh to go out of public preview for full documentation on functionality and such.

Setup Info
    Tags
      Help me…