The pillar that determines unit economics. This layer takes a model and
turns it into tokens efficiently. At NeoCloud scale, a 5% improvement in
tokens/sec/$ across thousands of GPUs is a structural cost advantage, not an
optimization. Tail latency (p95/p99 TTFT and ITL) matters as much as raw
throughput for enterprise interactive workloads.
This is the pillar with the least existing code and the biggest hiring need —
the expertise largely doesn't exist inside Mirantis or amazee today.
| Layer | Tool | Why | Status |
|---|---|---|---|
| Serving engine | vLLM | De-facto standard OSS engine: continuous batching, PagedAttention, prefix caching, speculative decoding, quantization (FP8/INT4), multi-LoRA serving, OpenAI-compatible server. Huge community, every relevant model lands here first. | 🚧 MVP week 1 |
| GPU enablement | NVIDIA GPU Operator | Drivers, device plugin, DCGM on any conformant k8s. We assume the cluster has it. MVP is NVIDIA-only (flagged to Ronan); vLLM CPU mode covers GPU-less dev/demo. | 📋 |
| Distributed serving | llm-d | Kubernetes-native distributed serving built around vLLM: disaggregated prefill/decode, KV-cache-aware routing via Gateway API Inference Extension. The strategy doc names "vLLM & llm-d or equivalent" explicitly. | 🔍 evaluating |
| Distributed serving (alt) | NVIDIA Dynamo | NVIDIA's disaggregated serving framework (Rust/Python, engine-agnostic: vLLM, TRT-LLM, SGLang). Strong NVIDIA alignment story for IREN's fleet; heavier operational footprint. | 🔍 evaluating |
The llm-d vs Dynamo question does not block anything. Both sit behind the
gateway and above vLLM. We ship single-node vLLM now, benchmark both
scale-out tracks against real IREN hardware, and decide with data. Track the
evaluation in the status board.
The interface itself is stable and unchanged: per model, an
OpenAI-compatible base URL and a served model name. The gateway is the
runtime's only consumer — nothing else talks to an engine directly. The
runtime must run on an NVIDIA GPU node (NVIDIA GPU Operator — the MVP is
NVIDIA-only) or in CPU mode (slow; for dev/demo without GPUs).
What changes is how those two strings are delivered, and there are two
horizons:
Today — the MVP is Helm-driven
(ADR-008). The runtime
chart serves N models from one models[] list, and the umbrella chart
wires the resulting endpoints into the gateway's model configuration. This
keeps the teams decoupled — each side is developed and tested standalone,
then composed in the umbrella chart — at the cost of the wiring being a
human step, and of helm upgrade being the only lifecycle verb.
Agreed long term — a CRD and controller own model lifecycle
(ADR-009). A
namespaced ModelDeployment resource becomes the single source of truth,
reconciled by a thin controller that renders plain vLLM Deployments. Then:
status, and theReady resources — no morestatus carries a capacity signal (queue depth, KV-cacheDivision of responsibility either way: the runtime renders workloads, caches
weights, reports readiness and emits capacity. It never proxies
/v1/chat/completions — that would add a hop and break SSE streaming for no
benefit. The gateway owns auth, rate limiting, token accounting and
model → backend routing.
ModelDeployment (ADR-009).topology value so the backend choice stays open.| Feature | Status | Editions | Ticket | Notes |
|---|---|---|---|---|
| Single-node vLLM serving | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | TKF-3 | One model served on a single node — an NVIDIA GPU, or CPU mode for dev/demo — behind an OpenAI-compatible endpoint (MVP engine — see ADR-001). |
| Helm-deployed engine with GPU scheduling | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | TKF-4 | Engine deployment fully managed by the umbrella chart: GPU resource requests, node selection, model config as values (tkf-runtime chart, runtime.enabled). |
| Gateway↔runtime shared-secret auth | 🟢 Shipped | Enterprise | TKF-52 | Static API key on the engine (tkf-runtime auth.static), umbrella-managed Secret consumed by both sides, gateway reference via os.environ — engine rejects unauthenticated calls, probes unaffected. |
| Multi-model serving (one values list) | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | TKF-59 | The runtime chart serves N models from a single list: per-model endpoint, weight cache and probes; add/remove a model without restarting the others. Fail-fast validation of the list, plus GPU time-slicing proven end to end. Helm-first per ADR-008 — superseded for the long term by the ModelDeployment CRD (ADR-009), but this is how the MVP ships. |
| Multi-model orchestration research (KServe / llm-d / Dynamo) | 🟢 Shipped | Enterprise, NeoCloud | TKF-61 | Should model lifecycle stay Helm-generated or move to an orchestrator? Resolved in ADR-009: a thin ModelDeployment CRD + controller of our own, rendering plain vLLM, with third-party orchestrators deferred to a separate decision made on need. Owning the CRD is what keeps that backend choice reversible. |
| ModelDeployment CRD + controller | ⚪ Planned | Enterprise, NeoCloud | TKF-78 | The agreed long-term model-management surface (ADR-009): a namespaced ModelDeployment resource is the single source of truth, reconciled by a thin controller rendering plain vLLM Deployments. Adds what a chart structurally cannot — per-model lifecycle independent of a release, admission-time validation against live cluster state, machine-readable status (endpoint + capacity signal), continuous reconciliation, and a shared content-addressed weight cache. Helm shrinks to installing CRDs, controller and RBAC. |
| Batch API / idle-GPU utilization (research) | 🟡 Evaluating | Enterprise, NeoCloud | TKF-54 | Queue relaxed-SLA work to keep GPUs busy: enterprise capex ROI off-hours, and NeoCloud provider margin (fill benched/idle capacity). vLLM has no batch API today — survey options, bring a recommendation. |
| Dedicated endpoints (reserved capacity) | 🟡 Evaluating | NeoCloud, Enterprise | TKF-56 | A model reserved on N GPUs, billed hourly with optional autoscaling — vs shared token endpoints. May map to a dedicated k0rdent cluster per reservation in the NeoCloud. |
| Elastic bench capacity (fluctuating fleets) | 🟡 Evaluating | NeoCloud | TKF-57 | Borrow over-provisioned/benched provider nodes with a short take-back warning — requires the serving fleet and gateway to tolerate node counts changing under them. |
| Capacity signaling (contract v2) | 🟡 Evaluating | NeoCloud, Enterprise | TKF-58 | Engines advertise headroom to the gateway (busy / free / send-batch), enabling idle-fill, elastic fleets and predictive model pre-warming — evolves the one-way runtime→gateway contract. |
| On-cluster weight caching | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | TKF-5 | vLLM Hugging Face cache on a PVC so replicas cold-start from local storage, not the internet — chart configuration, not a registry component. |
| Multi-GPU tensor parallelism | ⚪ Planned | Enterprise, NeoCloud | TKF-6 | Serve models larger than one GPU (vLLM tensor/pipeline parallel). |
| LoRA adapter hot-swapping | ⚪ Planned | Enterprise, NeoCloud | TKF-7 | Dynamically load/unload customer LoRA adapters against shared base models. |
| Perf tuning: prefix caching, speculative decoding, quantization | ⚪ Planned | NeoCloud | TKF-8 | The tokens/sec/$ program: continuous batching is free with vLLM; prefix caching, speculative decoding and FP8/INT4 quantization are tuned per model. |
| Distributed serving (llm-d) | 🟡 Evaluating | NeoCloud | TKF-9 | Kubernetes-native disaggregated prefill/decode and KV-cache-aware routing built on vLLM + Inference Gateway. |
| NVIDIA Dynamo evaluation | 🟡 Evaluating | NeoCloud | TKF-10 | Alternative disaggregated serving stack; evaluate against llm-d for the NeoCloud runtime track. |
| Follow-the-sun capacity: model warm pools & autoscaling | ⚪ Planned | NeoCloud | TKF-11 | Load models in/out of GPU memory as demand shifts across time zones; SLA-aware real-time provisioning. |
| Dynamic model loading on demand | ⚪ Planned | Enterprise, NeoCloud | TKF-63 | Load/unload models through an API without a Helm rollout. No longer gated on picking an orchestrator: per ADR-009 this becomes a feature of our own controller — create/delete a ModelDeployment. Note vLLM itself cannot swap base models in-process, so the lifecycle is pod-level; its sleep mode is a packing tool, not a substitute. |
| Capacity inventory for dynamic allocation | ⚪ Planned | Enterprise, NeoCloud | TKF-65 | Know total and free GPU capacity before placing a model — required once allocation moves from Helm to API/UI. Per ADR-009 the controller owns this: admission-time fit checks against live node and GPU state, plus a capacity signal published in ModelDeployment status for the gateway. |
| Model weight streaming from a registry | ⚪ Planned | Enterprise, NeoCloud | TKF-67 | Stream weights (e.g. from MSR, ADR-006) instead of full downloads for faster model starts. Becomes relevant with dynamic allocation in phase 3. |