The pillar that lets both the operator and the customer trust the platform.
The non-negotiable is per-token metering accurate enough to bill on — enforced
in real time, because if usage visibility lags consumption, finance blocks the
platform. Everything else iterates.
Two surfaces, one shared pipeline (so they never drift):
Metering path (bill-grade): LiteLLM logs every request's tokens and cost;
the Control Plane aggregates into spend/daily-activity APIs, projects budgets back
onto the gateway for real-time enforcement, and snapshots spend periods for
history. This runs in production today.
Telemetry path (SLO-grade): Prometheus scrapes the runtime and GPUs; this
is what SLO alerting, capacity forecasting, and eventually gateway routing
decisions consume. This is the build-out.
| Concern | Tool | Status |
|---|---|---|
| Metrics store | Prometheus (per cluster; federation/long-term store decided at NeoCloud scale — Thanos/Mimir 🔍) | ✅ control plane, 📋 runtime |
| Dashboards | Grafana | ✅ basic, 📋 fleet |
| GPU telemetry | NVIDIA DCGM exporter (ships with GPU Operator) | 📋 |
| Runtime metrics | vLLM native /metrics |
📋 |
| Customer export | OpenTelemetry | 📋 |
| Billing source | LiteLLM spend logs → Control Plane ledger | ✅ |
| Feature | Status | Editions | Ticket | Notes |
|---|---|---|---|---|
| Per-token spend tracking (key / user / team) | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | — | Token counts and cost per request aggregated into daily activity and spend APIs. |
| Budget ledger: cycles, top-ups, rollover | 🟢 Shipped | NeoCloud | — | FIFO ledger of subscription allocations, top-ups and rollovers with expiry, projected onto gateway budgets. |
| Prepaid pool budgets | 🟢 Shipped | NeoCloud | — | Purchase-gated prepaid pools with expiry windows. |
| Stripe subscriptions, webhooks & products | 🟢 Shipped | NeoCloud | — | Full monetization loop: products, pricing tables, checkout, idempotent webhook pipeline, dunning. |
| Real-time budget enforcement | 🟢 Shipped | Enterprise, NeoCloud | — | Hard stops enforced in the data plane when budgets exhaust — not next-day reconciliation. |
| Operator margin / pricing configuration | ⚪ Planned | NeoCloud | TKF-12 | Cost-plus margin configuration per model for operator pricing. |
| Chargeback / cost-center reporting | ⚪ Planned | Enterprise | TKF-13 | Enterprise-internal chargeback: spend by department/team/project, exportable. |
| Feature | Status | Editions | Ticket | Notes |
|---|---|---|---|---|
| Control-plane metrics & dashboards | 🟢 Shipped | Enterprise, NeoCloud | — | Prometheus-instrumented control plane with Grafana dashboards. |
| Token-level usage analytics | 🟢 Shipped | Enterprise MVP, Enterprise, NeoCloud | — | Per-key/user/team daily activity surfaced to customers. |
| GPU fleet telemetry | 🟢 Shipped | Enterprise, NeoCloud | TKF-14 | DCGM exporter → Prometheus: utilization, memory, thermals per GPU. |
| Runtime serving metrics (TTFT, ITL, queue depth) | 🟢 Shipped | Enterprise, NeoCloud | TKF-15 | vLLM's native metrics endpoint scraped and dashboarded; the basis for SLOs and routing decisions. |
| Customer-side OTel export | ⚪ Planned | Enterprise, NeoCloud | TKF-16 | Tenant telemetry exportable into the customer's own observability stack. |
| SLO / burn-rate alerting | ⚪ Planned | NeoCloud | TKF-17 | p95/p99 TTFT and ITL SLOs with alerting for operators. |
| Capacity forecasting & anomaly detection | ⚪ Planned | NeoCloud | TKF-18 | Fleet-level forecasting to drive purchasing and placement. |