How to deploy a RAG model

Arunkarthick

I have built a flow in Dataiku that takes an image as input and extracts the text from the image using an LLM. The flow works correctly inside DSS, but I am unable to find clear guidance on how to deploy this solution as an API service. I looked through the documentation and community resources but couldn’t find examples or step-by-step instructions for deploying an LLM-based image-to-text workflow. How should this be deployed properly, and are there any recommended practices or limitations when using LLMs in a deployed API?

Dataiku version used: 14.2.1

Dataiku version used: 14.2.1

Mikio

Hey there, we don't have much in the way of documentation as we generally do not recommend using the API node to deploy LLM based applications. Not all LLM mesh capabilities are available from the API node and since the LLM Mesh itself lives on the design/automation node, this creates a dependency between your API node and the underlying node.

I would recommend using the advanced LLM mesh add on, if you have it, to create a code agent. This will allow your python logic to be treated as a native LLM provider that can be queried via the mesh API.

If you don't have the add-on, you can expose your logic via a backend Webapp. This runs on the Design Node and has native access to Mesh connections.

You can use the following structure as a starting point:

import dataiku
import base64

# Define this inside your Webapp or Code Agent
def extract_text(image_base64):
    # 1. Decode the image
    image_bytes = base64.b64decode(image_base64)
    
    # 2. Access LLM Mesh natively (Design/Automation Node context)
    client = dataiku.api_client()
    llm = client.get_llm("your_multimodal_llm_id")
    
    # 3. Perform the vision task
    res = llm.complete_visual_prompt(
        prompt="Extract all text from this image.",
        image_bytes=image_bytes,
        mime_type="image/jpeg"
    )
    
    return {"text": res.text}