One word: Gateway
Early strategy documents called this pillar the "Inference Mesh". We now use
Gateway everywhere — for the per-region data plane and for the
multi-region fabric it grows into. Only the repo name
tokenfactory-inference-meshkeeps the old word.
The pillar that turns disparate endpoints into an enterprise-grade fabric —
and the one where we start from the strongest position: it ships today as the
amazee.ai Private AI Gateway, with paying production customers. The gateway
is how Token Factory delivers governable access now and 5-nines later.
The shipped architecture is a two-layer system:
| Feature | Status | Editions | Ticket | Notes |
|---|---|---|---|---|
| OpenAI-compatible endpoint | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | — | Single /v1 endpoint any OpenAI SDK can point at; routes to local or upstream models. |
| Per-user / per-team virtual keys | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | — | Key issuance scoped to users and teams with metadata, expiry and aliases (LiteLLM /key/generate driven by the control plane). |
| Budgets, rate limits & expiry per key | 🟢 Shipped | Enterprise, NeoCloud | — | max_budget, budget_duration and RPM limits attached at key creation; enforced in the gateway data plane in real time. |
| Spend caps with precedence (team / member / key) | 🟢 Shipped | Enterprise, NeoCloud | — | Layered caps where the strictest applicable gate wins; soft limits alert, hard limits stop traffic. |
| Model allow-lists per team | 🟢 Shipped | Enterprise, NeoCloud | — | Which teams may call which models, enforced at the gateway. |
| Machine-to-machine provisioning API | 🟢 Shipped | Enterprise, NeoCloud | — | Internal API-token-authenticated endpoints for programmatic key provisioning (used today by the Portal). |
| Email delivery (SMTP + test inbox) | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | TKF-98 | One smtp.* value shape (host, credentials, encryption) feeds Keycloak realm email (magic links, password resets) and the portal invite mailer; enterprises point it at their relay. Dev/demo installs enable the bundled Mailpit instead — an in-cluster catch-all inbox (basic-auth web UI at mail.) so every email the platform sends is visible without a provider account. |
| Audit logging | 🟢 Shipped | Enterprise, NeoCloud | — | Structured audit trail of control-plane actions. |
| External model backends (BYO frontier APIs) | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | TKF-62 | Register external OpenAI-compatible backends — base URL, model name, API key — as a repeatable list in the umbrella values. Adding/removing one never restarts the platform (gateway reload is fine); existing keys and config survive untouched. MVP phase 2. |
| Model upgrade & deprecation flow | ⚪ Planned | Enterprise, NeoCloud | TKF-64 | Replace model vX with vY while traffic shifts cleanly: gateway model aliases + deprecation flags, retirement without breaking clients. Post-MVP. |
| Gateway guardrails | ⚪ Planned | Enterprise, NeoCloud | TKF-68 | Surface the gateway’s built-in guardrail hooks (content filtering, PII) as Token Factory configuration — compliance teams want enforcement in transit, not only client-side. Post-MVP. |
| MCP governance (per-team MCP access) | ⚪ Planned | Enterprise, NeoCloud | TKF-69 | Operators control which MCP tools each team may use, via the gateway’s MCP support. Post-MVP. |
| Gateway audit log exposure | ⚪ Planned | Enterprise, NeoCloud | TKF-70 | The gateway already keeps an audit log — expose it (admin UI / export) so operators can answer who did what. Post-MVP. |
| Gateway model list from runtime status | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | TKF-91 | The gateway generates its model_list from Ready ModelDeployment status (ADR-009 pull model) — replacing the umbrella-wired configuration; evolves the proven wiring-job sync into a watcher. Built and merged (default off); goes live at the TKF-92 cutover. |
| External providers as custom resources (ExternalModelProvider) | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | TKF-95 | External OpenAI-compatible providers get their own CRD — one resource per provider connection (base URL + credentials secret + model list). Gateway-owned per ADR-011 (runtime serves; gateway routes and holds provider credentials): the kind lives in the gateway repo under gateway.tokenfactory.mirantis.com, and the gateway-sync watcher is its reconciler and status writer — Ready means routable. Live on the demo environment: create a CR and the model appears on the gateway restart-free; delete it and it leaves. Retires the Helm externalModels list at the TKF-92 cutover. |
| Feature | Status | Editions | Ticket | Notes |
|---|---|---|---|---|
| Region registry & orchestration | 🟢 Shipped | NeoCloud | — | Regions as first-class objects, each with its own gateway + DB cluster; validated on registration. |
| Cross-region key & resource provisioning | 🟢 Shipped | NeoCloud | — | Keys and teams provisioned into a chosen region from one control plane. |
| Dedicated (single-tenant) regions | 🟢 Shipped | NeoCloud | — | Private regions for regulated/sovereign customers. |
| Data-residency enforcement | ⚪ Planned | Enterprise, NeoCloud | TKF-19 | Region access control so requests, storage and logs stay inside declared jurisdictions. |
| Latency / capacity / cache-aware routing | ⚪ Planned | NeoCloud | TKF-20 | Route each request to the best site on latency, capacity, cost and cache locality. |
| Request-level failover & session-preserving HA | ⚪ Planned | NeoCloud | TKF-21 | Failover across replicas/sites without breaking the customer session; the 5-nines architecture. |
| Cross-NeoCloud federation ("surge to partner") | ⚪ Planned | NeoCloud | TKF-22 | Daisy-chained token factories: when one provider saturates, requests spill to a partner factory invisibly — open-weight models are commoditized, so responses are interchangeable (the "electrical grid" model). Needs identity, billing reconciliation, SLA enforcement across operators. |
Today, a client picks its region endpoint. The multi-region work moves that decision
into the platform:
| Capability | Mechanism | Tools | Status |
|---|---|---|---|
| Request-level routing | Route on latency, capacity, cost, cache locality | Envoy AI Gateway 🔍 / Gateway API Inference Extension 🔍 in front of / instead of parts of LiteLLM's router | 📋 |
| Residency enforcement | Requests, storage, logs pinned to declared jurisdictions with provable guarantees | region access control (in progress in amazee.ai backend) | 🚧 |
| Failover & HA | Session-preserving retry/failover across replicas and sites — the 5-nines architecture | gateway routing layer + health/metric signals from the runtime | 📋 |
| Federation | "Surge to partner": burst to other NeoClouds/operators with identity, metering and SLA continuity | long-term; standards-based APIs | 📋 |
The in-cluster half of routing overlaps with the
llm-d / Dynamo evaluation —
KV-cache-aware routing lives at the boundary between gateway and runtime, and we
will decide the split there deliberately, not by accident.