This is the target architecture for the full Token Factory, annotated with the
concrete open-source components. The Enterprise MVP
is a strict subset of this picture; the NeoCloud edition
is the whole thing.
Reading guide: components marked ✅ run in production today (amazee.ai);
🚧 are being built; 📋 are planned; 🔍 are under evaluation.
Everything a customer touches speaks the OpenAI API against the gateway —
never directly against an engine. That single rule is what makes governance
real: no key, no tokens.
The Control Plane
(tokenfactory-controlplane, formerly the amazee.ai backend) ✅ is a FastAPI
service that owns tenancy (users/teams), key lifecycle, budgets and spend
caps, region orchestration, products/billing, and audit. It provisions into
each region's gateway over the LiteLLM admin API and reads spend back from it.
Identity
(tokenfactory-identity, formerly moad-auth) ✅ is Keycloak: OIDC for portals
and chat, custom themes, magic links, and — for enterprises — brokering to the
customer's own IdP 📋.
LiteLLM ✅ is the per-region gateway:
virtual keys, team budgets, rate limits, model aliases/allow-lists, spend
logging, and OpenAI-compatible routing to whatever serves the model — local
vLLM or upstream providers (hybrid). See
ADR-002.
vLLM 🚧 is the serving engine —
continuous batching, paged attention, quantization, LoRA serving, prefix
caching (ADR-001 covers why not
Ollama, even for the MVP). Scale-out beyond single replicas goes through
llm-d 🔍 or NVIDIA Dynamo 🔍
— disaggregated prefill/decode, KV-cache-aware routing. That evaluation is a
NeoCloud-track decision; nothing in the MVP depends on it.
Model weights come from a model cache 🚧: Hugging Face snapshots on PVCs
first, evolving toward an OCI/content-addressed registry with P2P distribution
(Model Registry pillar).
Prometheus + Grafana ✅ for the control plane today; extended with vLLM
runtime metrics 📋 (TTFT/ITL/queue depth) and DCGM GPU telemetry 📋. Spend
metering flows through LiteLLM's logs into the Control Plane ✅ — that's the number we
bill on. Customer-facing OTel export 📋 comes later.
(Observability pillar).
| Plane | Enterprise MVP (4 wk) | Enterprise | NeoCloud |
|---|---|---|---|
| Consumption | LibreChat + Portal key self-service | + SSO chat | Full Portal |
| Control plane | Control Plane + Portal, self-contained profile (ADR-005) | Control Plane + Portal (slim profile) | Control Plane, full (billing, products) |
| Gateway | LiteLLM, single instance | LiteLLM | LiteLLM per region |
| Runtime | vLLM single-node | vLLM multi-GPU | vLLM + llm-d/Dynamo |
| Telemetry | Gateway defaults only | Prometheus/Grafana/DCGM | Full, incl. SLO + forecasting |
Details: Editions & Packaging.