Token Cost Radar

1What happens when your agent has 5,000 tools?

Uber published October 2 details of its MCP Gateway after internal adoption expanded to roughly 800 MCP servers exposing about 5,000 tools. Rather than make clients manage that estate directly, Uber built a centralized gateway for server discovery, authentication, authorization, configuration, and access. The architecture is a concrete example of the tool-surface problem reaching enterprise scale: capability catalogs can grow far beyond what should be treated as permanently resident agent context.

2Open-weight inference carries an 81% median price discount

BenchLM's October 2 pricing statistics track 181 models and put the median API rate at $0.95 per million input tokens and $3.75 per million output tokens. At a 3:1 input-to-output mix, open-weight models have a median blended price of $0.50 per million tokens versus $2.63 for proprietary models, an 81% discount. The data reinforces the modelmaxxing case: workload routing has a wide enough price surface to matter even before caching and batch discounts enter the calculation.

3Frontier token prices hold at $2.13 per million blended tokens

The Token Price Index updated October 2 at $2.13 per million blended tokens, down 9.1% month over month but up 80.8% year over year. Claude Sonnet 5.5 replaced Sonnet 5 at unchanged $2 input and $10 output rates, while GPT-6.1 Sol replaced GPT-6 Sol after only ten days. The combination illustrates why AI budgets cannot assume a simple downward price curve: capability, model composition, and product turnover can move faster than procurement cycles.

4AI gateways move from routing traffic to governing the bill

CRN reports October 2 that CData's Connect AI Gateway is being positioned as a control point between enterprise agents and underlying IT systems as companies confront high AI costs, uncertain ROI, and limited governance. The product combines access and action controls with centralized agent connectivity. The larger trend is increasingly clear: the gateway layer is becoming where enterprises expect to govern both what agents can reach and how much those interactions cost.

Tool-Schema Compression Enables Agentic RAG Under Constrained Context Budgets

This study evaluates 28 tool definitions across 6,566 controlled API calls and finds that conservative tool-schema compression saves 44% to 50% of schema tokens. At an 8K context budget, ordinary JSON schemas overflowed the available window and produced 2.6% average exact match, while compressed schemas restored operation with a 20.5 percentage-point average lift. Scaling tests found ordinary schemas overflowing at roughly 494 tools while compressed representations remained operational beyond 800.

Why it matters: Tool-surface bloat is not merely an invoice problem. Schemas compete directly with task evidence for finite context, so reducing resident tool metadata can preserve both token budget and usable capability.

Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs

FreshCache treats semantic-cache reuse as a freshness-risk decision rather than relying on similarity alone. On an 8,072-query benchmark expanded to 31,201 paraphrased queries, its learned policy achieved 97% search API savings at a 24-hour evaluation window with 0.1% hash-based stale error. The authors estimate answer-affecting stale error at roughly 0.034% in a separate judged sample.

Why it matters: Semantic caching improves token yield only when reuse remains trustworthy. A risk gate lets applications avoid retrieval and inference work when cached knowledge is likely still valid while escalating cases where freshness matters.

C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

C2KV combines KV-cache compression with reusable non-prefix context representations designed to be concatenated across long-context workloads. Across multiple model families and long-context benchmarks, the authors report up to 17 times faster inference while preserving generation quality, with the design targeting both redundant prefill computation and the storage and transfer cost of large caches.

Why it matters: Repeated context has two prices: recomputing it and keeping its inference state around. Efficient reusable caches attack both, moving token economics below the visible prompt.

Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving

This serving study finds that 80% to 95% of requests in its production traces are short despite fleets commonly being configured for worst-case context. Routing requests by estimated token budget into separate short-context and long-context pools reduced GPU-hours by 31% to 42% on Azure and LMSYS traces and lowered preemption rates 5.4 times.

Why it matters: Token budgeting is becoming an infrastructure primitive. The predicted token footprint can determine which serving pool should handle a request before the first inference token is produced.