How to leverage locally hosted llm using python scripts
Dataiku Version: 13.3.1
I have several LLMs that I have in my DSS cache from the HuggingFace Connection. I can leverage these models using the prompt recipe. However I am struggling to use them in custom python scripts.
For example if I get the LLM id of all the models I have a connection to using the following script, and then put the LLM id in for a model I have based on a cloud connection (OpenAI, Claude, etc). This works as expected.
project = dataiku.api_client().get_default_project()
completion_models = project.list_llms()
for model in completion_models:
print(model['id']) LLM_ID = "my-selected-LLM"
client = dataiku.api_client()
project = client.get_default_project()
llm = project.get_llm(LLM_ID)
completion = llm.new_completion()
completion.with_message("What is AI?")
resp = completion.execute()
if resp.success:
print(resp.text)
However, if I use one of my HuggingFace model ids such as: "huggingfacelocal:myhf:meta-llama/Meta-Llama-3.1-8B-Instruct:TEXT_GENERATION_LLAMA_2:promptDriven=true".
Then I get the following error: indicating that the chat_template_content_format parameter is missing when the OpenAIServingChat class is being initialized.
Exception: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Failed to start HuggingFace LLM, caused by: ExecutionException: java.lang.RuntimeException: java.io.IOException: Failed to start HuggingFace LLM, caused by: RuntimeException: java.io.IOException: Failed to start HuggingFace LLM, caused by: IOException: Failed to start HuggingFace LLM, caused by: CodedException: Local kernel failed to initialize model, caused by: RequestFailedException: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'