Status: Accepted · Date: 2026-08-10 (meeting transcribed) · Deciders: Phil, Michael, Ryan, Dmitrii · Supersedes:
ADR-008 ·
Research: TKF-61
ADR-008 put model
management in Helm as a deliberate, temporary MVP choice, with the
orchestration question deferred to research
(TKF-61). That research is now
done, and this ADR records its outcome.
Helm did the job well. The chart is genuinely capable and none of what
follows is a complaint about it:
models[] list drives everything. Each entry renders its ownmodelDefaultsname, hfModel, revision, servedName.templates/_validate.tpl),hfModel, revisions that are not pinned commit SHAs,servedName, over-long generated Service names, andgpuMemoryUtilization budget.So the question this ADR answers is not "can Helm serve many models."
It can, and does. The question is whether a Helm chart is the right owner of
model lifecycle once the person choosing models stops being the person
running helm upgrade.
| Limitation | Why it matters |
|---|---|
| Lifecycle is release-scoped | Adding, changing or retiring one model is a helm upgrade of the whole release. Every other engine is re-reconciled at the same time, so an unrelated edit can disturb healthy models. |
| Validation is release-wide and render-time | One bad entry aborts the entire render — nine healthy models cannot deploy because the tenth has a typo. And template validation only sees values: it cannot check that weights fit the GPU's VRAM, that requests fit node allocatable, or that the storage class has room. |
| No status | Nothing machine-readable reports ready, endpoint or load. The gateway's model_list stays static configuration, updated by a human. |
| No reconciliation | Helm is one-shot. No drift correction, no ordered lifecycle, helm upgrade is the only verb. |
| NodePorts and resources are hand-assigned | Deliberate in the chart — a derived port would renumber surviving models when the list changes — but it means every model needs a human-chosen port plus a matching load-balancer target edit, and hand-computed CPU/memory for every model/node pairing. |
| No API a UI can safely target | An admin UI would have to generate values.yaml and shell out to Helm. That is exactly the second-source-of-truth risk ADR-008 set out to avoid. |
Four scenarios, and only one of them is ours:
| One GPU | Many GPUs | |
|---|---|---|
| One model | vLLM — solved, today's state | llm-d / NVIDIA Dynamo — distributed serving |
| Many models | NVIDIA GPU Operator — time-slicing, MPS, or MIG | This ADR — vLLM orchestration across a fleet |
The two axes are independent, and that is load-bearing for the decision:
nvidia.com/gpu. Packing several models onto oneOur target is the bottom-right cell: many models across a fleet, each on
whatever shape it needs.
ModelDeployment custom resource is the single source of truth forruntime.tokenfactory.mirantis.com,v1alpha1, namespaced.tkf-runtime-controller) reconciles it, renderingkubectl, or by a GitOps controllerExplicitly not decided here, and deliberately so: which distributed-serving
backend we would eventually use, and when. See
Deferred by design.
This is the heart of the decision, so it is worth being precise about it.
We are not building a controller because no existing tool can deploy
vLLM. Several can. We are building it because Token Factory needs an API
surface it controls.
The admin UI (TKF-53) and
tokenfactory-gateway both need to bind to something. Whatever they bind
to becomes a contract we must keep stable across years of product
evolution:
phase, conditions and capacity, and writesmodel_list from status.endpoint acrossReady resources, replacing today's hand-wired umbrella-chartIf those consumers bind to a third-party API, our product's stability
becomes that project's release policy.
KServe's LLMInferenceService is the closest external candidate. It is also
still alpha and recently broke: KServe
PR #4886 (merged 2026-01-14)
moved it from serving.kserve.io/v1alpha1 to v1alpha2 as an explicitly
labelled breaking change. The cause was not KServe's own instability — it
was an upstream break, where Gateway API Inference Extension v1.2.0
graduated its API, changed group names, and renamed InferenceModel to
InferenceObjective. KServe embeds those types, so it had to follow.
Had our admin UI been bound to LLMInferenceService, that dependency chain
would have reached our product surface. With our own CRD, the same event is
absorbed inside one reconcile branch.
Because ModelDeployment is the public API, changing what renders it is an
internal change:
LLMInferenceService later, if its lifecycle features earnIn every case the admin UI, the gateway contract and every existing
ModelDeployment are unaffected. Owning the CRD is what makes the
backend decision deferrable — which is exactly what lets us postpone it
today rather than guessing now.
Two categories of work belong to us no matter which backend renders the
pods, and both require a reconcile loop rather than a translation layer:
Cross-object invariants. Some rules span all models at once, so they
belong to something that watches the whole set:
servedName globally unique, so the gateway's model_list isA stateless translator could enforce these at write time, but it cannot
repair them — if a shared PVC or an engine is deleted out of band,
nothing notices. KServe reconciles its own objects; it does not know about
our invariants.
Status aggregation is continuous work, not a query. Producing the
gateway's endpoint list and capacity signal means merging what the backend
reports with metrics scraped from every vLLM pod, then normalising the
result into one shape that is identical whether the backend is raw vLLM,
KServe or llm-d. Doing that per request, on demand, is slow, uncacheable,
and fans out to every pod on every poll. Doing it in a reconcile loop and
writing the result to status makes it a cheap read for any number of
consumers — and keeps the shape stable while backends change underneath.
A REST facade was considered and rejected for now. The Kubernetes API is
already a REST API, and using it directly gives us authentication, RBAC,
admission validation, audit logging, watch/streaming, optimistic
concurrency and generated clients in every language for free — with CRDs
supplying a typed, versioned, schema-validated resource on top. For
consumers running in-cluster, a ServiceAccount token is all that is
required.
Building a facade means another service to build, deploy, authenticate,
version, monitor and secure, and it would still need the reconcile loop
underneath. It also keeps all state in etcd rather than introducing a
database with its own migration and backup story.
There is prior art: KServe itself has no management REST API. Its
control plane is CRDs only; kubectl and client libraries are the
interface. Its data plane is REST, but that is inference traffic.
If a facade later becomes necessary — cross-cluster aggregation, exposing
model management beyond the cluster's trust boundary where security review
will not permit reaching the API server, or authorisation rules finer than
RBAC's verb-plus-resource model — that facade is the gateway, not a new
runtime component.
INSTALL TIME (once) RUNTIME (per model, via Kubernetes API)
------------------- ---------------------------------------
helm install tkf-runtime admin UI kubectl GitOps
| | | |
| CRDs + controller + RBAC +---------+---------+
| |
v v
+--------------------+ +------------------------------+
| tkf-runtime | watches | ModelDeployment (CRD) |
| controller |----------->| spec <- desired state |
| - validating hook | | status <- observed state |
| - shared weights | writes | |
| - nodePort alloc |----------->| |
| - status/capacity | +------------------------------+
+--------------------+ |
| renders | (1) reads status.endpoint
v v
Deployment + PVC + Svc +----------------------+
| | tokenfactory-gateway |
| +----------------------+
| |
+------------> vLLM pods <-----------------+
(2) proxies inference traffic
Helm is NOT in the model-lifecycle path.
apiVersion: runtime.tokenfactory.mirantis.com/v1alpha1
kind: ModelDeployment
metadata:
name: qwen2-5-7b
namespace: tkf
spec:
model:
repo: Qwen/Qwen2.5-7B-Instruct
revision: a09a35458c702b33eeacc393d103063234e8bc28 # required, pinned
servedName: qwen2.5-7b-instruct
topology: single # single | disaggregated
replicas: 1
# Resource configuration is optional: the controller derives defaults from
# model size and the target node's GPU, and refuses the resource if it
# cannot be scheduled.
Four required fields for the common case. Everything else is derived, which
removes the per-model/per-node hand-tuning the chart requires.
topology exists from the start even though only single is implemented.
It is the seam that keeps the distributed-serving decision reversible: a
405 B model differs from a 7 B one by a topology value and a parallelism
block, and which backend renders it is operator-level configuration, not
part of the per-model API.
Rejecting bad configuration at kubectl apply time, rather than at CUDA
init five minutes later, is one of the larger practical wins over Helm —
and it is only possible because a webhook can read live cluster state,
which a template cannot.
| Check | Rejects |
|---|---|
| Fit | Estimated weights + KV cache exceed the target GPU's VRAM at the requested utilisation |
| Revision pinned | Empty spec.model.revision — a mutable ref would let a redeploy silently serve different weights |
| Served-name uniqueness | Two ModelDeployments advertising the same servedName, which would make the gateway's model_list ambiguous |
| Node capacity | Derived CPU/memory requests exceed allocatable on any node matching the selector |
| Storage | Requested weight-cache size exceeds remaining capacity on the target storage class |
| Topology support | A topology requested with no backend configured to render it |
status:
phase: Ready # Pending | Downloading | Loading | Ready | Degraded | Failed
observedGeneration: 3
endpoint:
baseURL: http://tkf.example.com:30800/v1
servedName: qwen2.5-7b-instruct
replicas:
desired: 1
ready: 1
weightCache:
pvc: weights-qwen2-5-7b-a09a354
shared: true # refcounted across ModelDeployments
capacity: # shape defined by TKF-58
...
conditions:
- type: WeightsAvailable
status: "True"
- type: EngineReady
status: "True"
The capacity block should be shaped as a superset of what a Gateway API
Inference Extension endpoint picker consumes — queue depth, KV-cache
utilisation, model residency. Those are the same signals the wider ecosystem
standardises on. Doing this costs nothing now and means that if we ever
adopt that layer, our contract is already compatible rather than needing
migration.
This refines, and does not replace, the contract on the
Inference Runtime pillar:
the runtime still produces an OpenAI-compatible base URL and served model
name per model. What changes is that they are now published in resource
status and consumed programmatically, instead of being wired by hand in
the umbrella chart.
/v1/chat/completions — inserting a hop would add latency and break SSEreplicas 0↔1. This is whyRejected. It is the status quo, and the six structural limitations above are
the reason. Most decisively, it cannot give an admin UI anything safe to
talk to, which is the requirement that motivated
TKF-53 in the first place.
The cheapest possible alternative: keep the chart, deliver it properly with
Flux's helm-controller, one HelmRelease per model, weight-cache claim
overridable so identical models share a PVC.
It genuinely fixes a lot: reconciliation and drift correction, a CRD the UI
can CRUD instead of shelling out to Helm, per-model blast radius, and free
rollback on failed upgrades.
Rejected because it cannot fix the domain layer:
HelmRelease.status has a closed shapeReady=True means the release applied — it cannot mean "the endpoint ismodel_list and thespec.values is opaque_validate.tpl can only check servedName and NodePort uniquenessFlux remains complementary and desirable — it is a good way to install
the operator and to apply ModelDeployment manifests from Git. It is just
not a substitute for the controller.
LLMInferenceService)The strongest external candidate, and the one we expect to revisit. One CRD
covers single-node, multi-node and prefill/decode-disaggregated serving; it
composes officially with llm-d; it is converging on upstream Gateway API
Inference Extension primitives; and it has genuine lifecycle features we do
not (revisions, canary, pluggable storage initialisers for S3/GCS/OCI).
Governance is real: maintainer seats are spread across Red Hat, Bloomberg,
Nutanix, SAP, Cloudera and Ideas2IT, with release authority outside Red Hat.
Not now, because:
LLMInferenceService is alpha and churning — two alpha versions deep,odh-model-controller) for platform glue: routes, authDeferring costs us little precisely because the CRD makes delegation a
later, internal change.
The most feature-complete open-source control plane for vLLM specifically,
developed under the vLLM project itself, and the closest thing to a direct
competitor for what we are building. It spans both control plane and
disaggregated data plane: a self-service console, a production
OpenAI-compatible batch API, multi-engine support, prefill/decode
disaggregation with prompt-length bucketing and latency-predictive routing,
mixed-GPU heterogeneous serving, and LoRA management.
Rejected, for two reasons:
tokenfactory-gateway.Worth mining for design, particularly its batch API as prior art for
TKF-54 and its console for
TKF-53. Not adopted.
Both solve a different problem: one model across many GPUs, via
prefill/decode disaggregation, KV-cache-aware routing and multi-node
parallelism. Neither does model lifecycle — llm-d's own per-model layer is
a Helm chart, which is the abstraction we are moving away from — so
neither is an alternative to this decision. They are a future backend for
topology: disaggregated.
Not needed yet: models up to ~70 B fit one GPU or one node with tensor
parallelism, llm-d is CNCF Sandbox at v0.5, and disaggregation's KV-cache
transfer wants a fast fabric to pay off.
Red Hat OpenShift AI ships the layered architecture this ADR assumes:
KServe for lifecycle, llm-d for distributed serving, vLLM executing, and
authentication in the gateway layer ahead of vLLM — the same split between
gateway-owned auth and runtime-owned serving that we propose. Most
directly relevant: Red Hat models topology as a per-model runtime
selection with an admin-level default, which is structurally identical to
our spec.topology field plus operator-level backend configuration. That is
good evidence the resource shape here is the right one. What Red Hat does
not provide, and we would still build, is the Token Factory-specific
contract: model_list generation, our capacity signal, weight-cache policy.
Nutanix Enterprise AI is positioned as a centralised AI control plane
unifying cloud-hosted and private models behind one secure inference
endpoint, leading on token economics, multi-tenancy and governance rather
than serving internals. That is essentially Token Factory's market position
from an adjacent infrastructure vendor, and it confirms the framing of the
runtime as a component beneath a gateway rather than the product surface
itself.
Decided boundaries, so these do not get relitigated:
spec.topology reserves the seam.ModelDeployment is namespaced, and one runtime instance servesv1alpha1model_list becomes generated rather than hand-wired.status.capacity) and anvidia.com/gpu to request; whether the device is time-sliced iskubectl and GitOps become first-class management paths, notWhat we still need to decide later when the need comes:
LLMInferenceService stabilises past alpha and we