Status: Accepted (inherited from production) · Date: 2026-07
Every edition needs a gateway: virtual keys, budgets, rate limits, model
routing, one OpenAI-compatible endpoint. The amazee.ai Private AI Gateway has
run LiteLLM per region in production since early 2025, driven by a FastAPI
control plane, with paying customers.
LiteLLM remains the gateway data plane for all editions. The Token Factory
Control Plane (formerly the amazee.ai backend) remains the service that
provisions and governs it.
- It's shipped. Two years of production hardening, including workarounds
for real LiteLLM quirks (idempotency handling, membership budget bugs) that a
greenfield gateway would rediscover the hard way.
- The MVP scope doc independently picked LiteLLM — the two tracks agree.
- Budgets/keys/teams/spend-logs are exactly the governance spine both editions
sell; LiteLLM's admin API is the contract our control plane already speaks.
- Upstream-provider routing (Bedrock, Azure, OpenAI, Anthropic) is built in —
the Enterprise AI Gateway hybrid story is configuration.
- LiteLLM's router is per-instance; cross-region gateway routing is our work,
layered above or beside it. The Envoy AI Gateway / Gateway API Inference
Extension evaluation (🔍 on the status board) may take over the
in-cluster routing half — that would complement, not replace, LiteLLM's
key/budget layer, or absorb it only with a deliberate future ADR.
- We track LiteLLM's release cadence closely and pin versions in the chart;
the control plane's idempotent-error handling shields us from most upstream
API drift.