Status: Accepted · Date: 2026-08-19 ·
Deciders: Michael, Dmitrii (runtime-team review on
inference-runtime PR #22)
TKF-95 decided that external
OpenAI-compatible providers get their own custom resource —
ExternalModelProvider, provider-shaped: one resource is one connection
(base URL + credentials Secret) carrying the models it offers. The first
implementation put the kind in the inference-runtime repo, in
ModelDeployment's API group, on the theory that co-locating both kinds
would let one team bless the API surface and one webhook enforce
servedName uniqueness across them.
The runtime team's review accepted the shape but rejected the home, with
an argument that holds up:
- The runtime's scope is hardware utilization — weights onto GPUs,
serving them efficiently. An external provider has no workload, no
weights, no placement, no capacity signal. Nothing for the runtime to
reconcile.
- The resource exists only so the gateway can route to the provider.
In LiteLLM terms it is already just a model_list entry
(api_base + api_key) — a runtime-owned CR would round-trip through
the runtime to end up back in gateway configuration.
- The ecosystem precedent agrees: KServe has no external-model concept
at all; Envoy AI Gateway carries it (AIServiceBackend,
BackendSecurityPolicy, route-level aliasing) in front of the serving
layer.
The split is: the runtime serves; the gateway routes and holds provider
credentials.
ExternalModelProvider lives in the tokenfactory-gateway repo under
its own API group, gateway.tokenfactory.mirantis.com/v1alpha1
(shortName emp). The shape is exactly what the runtime team reviewed;
only the home and group changed. The gateway chart ships the CRD, so it
arrives with the gateway deployment.- There is no standalone controller. The gateway-model-sync watcher
(TKF-91) — the component that
already reconciles desired models into LiteLLM — is this kind's
reconciler and status writer. That upgrades the status semantics:
Ready means registered and routable in the gateway, not merely
"the credentials Secret exists". Credentials are still read only via
SecretKeyRef and registered with LiteLLM by value (encrypted in its DB
with the master key), which is what keeps provider adds restart-free.
- servedName uniqueness needs no webhook. ModelDeployments and
ExternalModelProviders meet at exactly one point — the watcher's merge
boundary — which refuses ambiguous names loudly, registering neither
claimant. The cross-kind admission question from TKF-95 dissolves.
- Attribution follows the group: the portal stamps
gateway.tokenfactory.mirantis.com/requested-by on ExternalModelProviders
(ModelDeployments keep their runtime-domain key).
- Repo ownership matches team ownership: the runtime team's API surface
stays purely about serving; the platform team owns the gateway group
end to end (CRD, watcher, portal Providers screen —
TKF-96).
- The Helm
externalModels list (TKF-62)
still retires at the TKF-92
cutover — that plan is unchanged; only the CR's home moved.
- Deferred, deliberately: an external provider is arguably a global
fact (the same provider key is valid from every region), while a
namespaced CR is per-cluster — N clusters would hold N copies of the
credential. For the single-cluster MVP this changes nothing. When
multi-region NeoCloud is real, the likely answer is the control plane
distributing provider entries to per-cluster gateways; that gets its own
decision then.
- A reachability probe (feeding a
Degraded phase) remains a follow-up;
the phase enum reserves the value but nothing produces it yet.