Connection to locally hosted LLM on Apple Silicon

ron_dk
Level 2
Connection to locally hosted LLM on Apple Silicon

Is there any way to setup a connection that connects to a locally running LLM? On Apple Silicon it is possible to run huggingface models locally with some different open source solutions, which make use of Metal for the GPU-compute. The solution I have running can be said to function as a server accessible as localhost. Would be great to be able to setup a connection to run AI recipes on that. 


Operating system used: MacOS

5 Replies
ron_dk
Level 2
Author

Thanks, Turri. But I think I was unclear. I do have metal running on MacOS, and I do have a local LLM running (via LM Studio). What I am looking for is a no-code way to setup an LLM connection (using the tools available under the "connections" tab of the admin section) to that local LLM. Have tried different workarounds (using the Azure connection, installing the deprecated GPT plugin) but none of them will let me direct the AI recipe to an LLM running locally (but outside of Dataiku).

Of course, this aveniue of inquiry begins with the fact that the Huggingface connection (also under the admin) won't run om my Mac, since the setup requires Nvidia GPUs to run. If there is a solution to get around that limitation, that would also be great. Is that maybe what you are suggesting could be achieved with the Apple PluggableDevice?

0 Kudos

OK I got what you want to do now. I am guessing Dataiku hasn't opened the local avenue yet. I haven't played much with the LLM Mesh but my simple understanding is that it lets you connect to different LLMs providers. I wonder if will be possible to duplicate one of the LLM provider APIs locally and fake the DNS using your local hosts file so that it connects to a local webserver that execute your local LLM.

0 Kudos

Hi @ron_dk,

I understand that a LLM Mesh connection plugin is in the works. This would enable creating custom connections for specific use cases (like yours). You may want to contact Dataiku directly to get more information.

We will be using this when it is available to connect to an API that we are putting in front of an LLM.

 

Marlan

0 Kudos
ron_dk
Level 2
Author

Thanks, Marlan. Would be a bit of revolution for us non-coders. Fingers crossed!

0 Kudos

Labels

?
Labels (1)
A banner prompting to get Dataiku