Goal: demonstrate that an enterprise can register users and direct them to
a local model — each user gets a chat link and an API key — with the simplest
stack that still feels like managing a real system. The management loop is
the product; serving performance is not (yet) the point.
Success looks like: an admin creates a user → the user opens a chat link,
talks to a local model → copies an API key → calls the same model from code —
all governed through one gateway.
Timeline agreed 2026-07-23
The platform side of this stack runs end to end from the umbrella chart on
the dev rig today. Exit criteria sharpened at the alignment meeting: fully
deployed on AWS by 2026-08-14 (TKF-50)
with a customer installation guide,
ready for design-partner walkthroughs from mid-August. The chat surface gains
a portal entry point (TKF-48); one
consolidated login story is the goal (TKF-24,
ADR-007).
Phase 1 — done. Single node, single model, single GPU: the full
management loop (users → chat → keys → governed gateway → real GPU
serving) live on AWS with TLS and chat SSO, ahead of the 2026-08-14
target.
Phase 2 — current. Multi-model and multi-GPU, with everything still
managed through Helm (ADR-008):
serve several local models from one list
(TKF-59 /
TKF-60), register external
OpenAI-compatible backends
(TKF-62) — and changing one
model never disrupts the rest.
Phase 3 — in progress. Dynamic management through custom resources,
on two tracks. Local models: the
ModelDeployment CRD
(the TKF-61 research outcome) with a thin runtime controller owning model
lifecycle (TKF-78, with the
runtime team). External providers: gateway-owned ExternalModelProviders
per ADR-011 —
already live on the demo environment: the gateway-model-sync watcher
(TKF-91) registers CRs into
the gateway restart-free and writes routing status back. The admin
section in the portal
(ADR-010) is
shipped — scaffolding TKF-94,
Kubernetes access TKF-93,
provider management TKF-96 —
with the product screens
(TKF-66,
TKF-53) and the Helm→CR
cutover (TKF-92) remaining.
The whole phase-3 set is labelled edition-mvp on the board.
The scope doc's four components — with two deliberate upgrades: the engine
is vLLM, not Ollama
(ADR-001), and the management
layer is the shipped Portal + Control Plane instead of any custom page
(ADR-005 — the "My Keys"
surface already exists in production). Everything above the gateway is exactly
what it would be over the production engine, so the NeoCloud runtime
(llm-d/Dynamo, multi-node) is a backend swap.
| Layer | Choice | Notes |
|---|---|---|
| Engine | vLLM single-node | OpenAI-compatible; real production engine from day one. Default model: any small 7–8B open-weight model (Gemma/Qwen class — runtime team's pick, swappable via values). NVIDIA GPU, or CPU mode for GPU-less dev/demo |
| Gateway | LiteLLM | The irreducible core — per-user virtual keys and governable access. Non-negotiable: never point the chat UI straight at the engine. |
| Chat | LibreChat | Off-the-shelf; OIDC-capable when SSO lands. (Open WebUI is the fallback; its license added branding clauses — check before switching.) |
| Management | Portal + Control Plane (shipped) | Same stack as production; the self-contained (de-AWS) profile is the packaging work — controlplane#7 |
| Identity | Keycloak | Required for Portal login (ADR-004 amended); toggleable off for gateway-only installs |
| Custom surface | None | Resolved by ADR-005: the Portal's shipped key self-service covers "My Keys" |
| Packaging | One umbrella Helm chart | helm install tokenfactory -f values-single-node.yaml against any k8s with a GPU. No k0rdent (ADR-003) |
The split between the runtime team and the platform side is one interface
(details on the Serving Engine page):
the runtime chart outputs an OpenAI-compatible base URL + model name
(auth TBD); the umbrella chart passes them into the gateway's model
configuration. Each side develops and tests standalone, then composes.
Today the contract is one-way: the runtime hands over an endpoint, the
gateway routes to it. The discussed evolution
(TKF-58) inverts part of it:
engines signal capacity ("fully busy" / "headroom — send batch or
real-time work"), enabling idle-fill (TKF-54), elastic fleets (TKF-57) and
predictive pre-warming of newly released models. Not MVP scope — recorded
so the contract's direction is visible to both teams.
Inference optimization tuning · model registry & automated deploys ·
observability dashboards · chargeback & analytics · SLO/priority classes · MCP
tool governance · guardrails (Llama-Guard/PII) · multi-tenant hierarchy ·
twelve-screen console · multi-tenant IdP brokering.
Each deferred item maps to a capability on the status board — the
MVP cuts the flesh, keeps the governance spine, and the spine is the same one
the NeoCloud edition runs on.
curl end-to-end.