From Stranded GPUs to Governed Inference: Dataiku and NVIDIA Run:ai for NVIDIA NIM at Scale
By Shashank Gaur, Solution Architect at Dataiku, Nate Kim, Field Engineer at Dataiku, Guy Salton, Solution Architect at NVIDIA & Vineeth Kalluru, Solution Architect at NVIDIA
From Benchmark to Production: The Last Mile for NVIDIA NIM Deployments
Recent benchmarking from NVIDIA and Nebius made one thing clear: fractional GPU scheduling with NVIDIA Run:ai isn't just an optimization technique — it's a foundational capability for production LLM (large language models) inference. In the NVIDIA/Nebius benchmark, which tested NVIDIA NIM microservices on fractional GPU allocations across H100 NVL and HGX B200 clusters, NIM delivered 86% of full-GPU concurrent user capacity at just 0.5 GPU, up to 2x more concurrent users on smaller models with 0.25 GPU fractions, and 3x more total system users when co-locating mixed workloads on shared GPUs — all with sub-second TTFT (time to first token).
The remaining question for most enterprises is operational: how do AI teams actually deploy and inference NIM in production every day without becoming Kubernetes specialists? That's where Dataiku comes in.
Dataiku and the AI Factory Accelerator: NIM as a Built-In Capability
Dataiku is the Platform for AI Success that enterprises trust for governed AI, and through its collaboration with NVIDIA, delivers the AI Factory Accelerator — a reference stack that brings NIM microservices, NVIDIA NeMo Guardrails, and NVIDIA Run:ai orchestration into a single, governed environment.
The NVIDIA NIM integration ships natively with Dataiku and exposes two complementary capabilities:
- a NIM LLM Mesh connection that surfaces any NIM endpoint — self-hosted via Dataiku, or running on external infrastructure, as a governed model alongside commercial APIs with full audit logging, cost tracking, streaming, and tool calling support, and
- a NIM Deployment Macro that lets administrators deploy, inspect, and manage the NVIDIA GPU Operator, NIM Operator, and individual NIM services directly from the Dataiku administrative console to any cloud or on-prem attached Kubernetes cluster using the NVIDIA NIM Operator under the hood
The combination means a Dataiku administrator can deploy a NIM Service to an attached Kubernetes cluster and immediately wire it into the LLM Mesh as a governed endpoint — all without writing YAML or leaving the Dataiku UI. Underneath that experience, every NIM Dataiku deploy runs as a containerized workload on Kubernetes, dispatched through NVIDIA Run:ai. The fractional GPU and dynamic scaling capabilities NVIDIA demonstrated in benchmarking become dropdown choices for the practitioner deploying the model.
Why Fractioning Matters
The NVIDIA/Nebius benchmark surfaces the operational reality enterprise IT has been navigating for years: most models don't fill a card. Qwen3-14B occupies ~35% of an H100 NVL. Phi-4-Mini fits comfortably in 25%. Embedding models like Qwen-Embeddings-0.6B can run in 12.5%. Allocating a full GPU to each leaves enormous capacity stranded — and forces teams to over-provision their fleet just to host the model catalog the business is asking for.
NVIDIA Run:ai changes the math by exposing two values per workload:
- Request — the guaranteed memory floor each NIM replica always gets.
- Limit — the burst ceiling each replica can dynamically expand into when the GPU has spare capacity.
This means a chat model reserved at 0.5 GPU can briefly grow to fill the card during traffic spikes, then contract back to its floor when an embeddings service on the same GPU needs its share. Memory isolation is enforced at runtime; compute cycles are distributed fairly, and the cluster breathes with demand. Combined with horizontal pod autoscaling, the NIM Operator makes fractional NIM deployment a production-grade pattern rather than an experiment.
How Dataiku Deploys NVIDIA NIM with Fractional GPUs and Dynamic Scaling
To register NVIDIA Run:ai as the cluster scheduler in Dataiku. Go to Administration → Cluster → Actions → Runai Scheduler → Select the Action → Run Macro (Fig 1). This ensures that all Dataiku Jobs on the selected cluster use the Runai Scheduler.
Fig 1 – NVIDIA Run:ai scheduler macro configuration tab in Dataiku. Running this macro will enable Dataiku pods to use runai scheduler on the cluster
To deploy NIM, go to Administration → Cluster → Actions → NVIDIA NIM → Select NIM Service: Add, select the NIM model from the drop-down, Enable Use Fraction GPUs flag and add the fraction to use and the dynamic scaling fraction limit. Click Run Macro.
Once deployed, the NIM Service: Inspect macro returns the model endpoint, which you can register in a NIM LLM Mesh connection. The NIM is immediately available in Prompt Studios, LLM-powered Visual Recipes, agents, and the Python and REST LLM Mesh APIs, all fully governed, with cost tracking, audit trails & dynamic scaling
Fig 2 - NIM Service Add Macro configuration Tab in Dataiku. Running this will deploy the selected NVIDIA NIM with the provided configuration in the Dataiku Attached Cluster
Fig 3 - NVIDIA Run:ai Dashboard, showing the running NIM Service with fraction GPUs deployed via Dataiku NVIDIA NIM Macro
The Bottom Line
The NVIDIA/Nebius benchmark proved that GPU fractioning and dynamic scaling deliver production-grade performance for NIM inference. Dataiku makes it the default deployment experience: a built-in NIM Deployment Macro for managing GPU Operator, NIM Operator, and NIM services from the admin console; a NIM LLM Mesh connection that turns every endpoint into a governed model; NVIDIA Run:ai scheduling that fractionalizes GPUs by default; and autoscaling that handles traffic without manual capacity planning.
Fractional NIM deployment stops being an infrastructure project. It becomes a built-in capability, governed by NVIDIA Run:ai, operationalized by Dataiku, and ready for the next agentic workload your business asks for.
Beyond NIM: Fractional GPUs Across the Dataiku Platform
The same NVIDIA Run:ai integration extends fractional GPU and dynamic scaling to every workload in Dataiku — code notebooks and recipes, Visual ML and Visual Deep Learning training, Hugging Face model deployment, Code Studios, and custom containerized recipes. One NVIDIA Run:ai integration. Every GPU workload in Dataiku.
Ready to deploy NVIDIA NIM on fractional GPUs in your environment? Connect with your Dataiku or NVIDIA account team to walk through the AI Factory Accelerator on your cluster.