How to leverage locally hosted llm using python scripts

areisdorf

Dataiku Version: 13.3.1

I have several LLMs that I have in my DSS cache from the HuggingFace Connection. I can leverage these models using the prompt recipe. However I am struggling to use them in custom python scripts.

For example if I get the LLM id of all the models I have a connection to using the following script, and then put the LLM id in for a model I have based on a cloud connection (OpenAI, Claude, etc). This works as expected.

project = dataiku.api_client().get_default_project()

completion_models = project.list_llms()



for model in completion_models:

    print(model['id'])

LLM_ID = "my-selected-LLM"

client = dataiku.api_client()

project = client.get_default_project()

llm = project.get_llm(LLM_ID)

completion = llm.new_completion()

completion.with_message("What is AI?")

resp = completion.execute()



if resp.success:

    print(resp.text)

However, if I use one of my HuggingFace model ids such as: "huggingfacelocal:myhf:meta-llama/Meta-Llama-3.1-8B-Instruct:TEXT_GENERATION_LLAMA_2:promptDriven=true".

Then I get the following error: indicating that the chat_template_content_format parameter is missing when the OpenAIServingChat class is being initialized.

Exception: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Failed to start HuggingFace LLM, caused by: ExecutionException: java.lang.RuntimeException: java.io.IOException: Failed to start HuggingFace LLM, caused by: RuntimeException: java.io.IOException: Failed to start HuggingFace LLM, caused by: IOException: Failed to start HuggingFace LLM, caused by: CodedException: Local kernel failed to initialize model, caused by: RequestFailedException: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'