Token Cost Radar

1Can you stop an AI agent before it exceeds its budget?

Google Cloud announced new cost controls alongside its Gemini agent on October 8. According to CEO Thomas Kurian's announcement, organizations can establish project-level spending limits in Cloud Billing that monitor token consumption and sandbox costs. When a limit is reached, the affected project's agent pauses until an administrator resumes it. Google also introduced Smart Routing, which selects models according to workload requirements and cost, and supports departmental chargeback through project-level accounting. The announcement builds on Google's earlier Spend Caps initiative. The important development is the integration of routing, metering, and execution control within the agent platform rather than treating them as separate administrative functions.

2Uber's legal agent shows how selective context can improve useful output

Uber Engineering published the architecture of its Legal Redlining Agent on October 8. The system uses similarity search and metadata filtering to narrow thousands of historical legal interactions to approximately 20 candidates, applies a second relevance filter, and supplies selected examples and rules to its language models. Uber reports a reduction of more than 20% in average contract review time and 91% accuracy in AI-generated decisions. The engineering team also describes replacing potentially expensive reflection loops with a single targeted correction step, explicitly limiting token consumption and latency. These are Uber's reported application results, not a controlled token-savings benchmark, but they demonstrate how retrieval discipline and workflow design can improve the economics of useful work.

3Token Fabric targets the infrastructure cost behind inference

Reuters reports that Nvidia-backed Upscale AI launched Token Fabric on October 8, combining networking hardware and software to connect AI processors from different suppliers within data centers. The system is designed to identify infrastructure bottlenecks and reduce the time expensive accelerators spend waiting for data. Upscale plans an initial release during the fourth quarter of 2026, with additional components arriving through 2027. No independently verified token-cost reduction was reported. The significance is architectural: inference economics depends on processor utilization, communication overhead, and infrastructure efficiency as well as model-level token pricing.

Breaking the Tie: A Cluster-Aware Routing Framework for Large Language Models

Submitted October 5, this paper examines a weakness in model routing: several models may answer the same training question correctly, making it difficult to learn which model should receive future requests. The proposed Cluster-Aware Soft-Labeling Routing framework uses broader domain-level performance information to train a lightweight selector rather than treating every correct model as an interchangeable target. The authors report a 7.80% improvement in overall average benchmark performance relative to Llama-3.3-70B-Instruct and routing inference latency of 1.13 seconds. The study focuses on routing quality rather than directly demonstrating lower commercial API bills.

Why it matters: Cost-aware routing only works when the router reliably identifies sufficiently capable models. Better selection can reduce expensive escalations and failed executions, although routing overhead and actual task costs must still be measured.

KV²: A Self-Refining KV Cache

Submitted October 2, KV² introduces a two-stage approach to compressing reusable key-value caches for long-context inference. A lightweight scorer first identifies potentially important context tokens, after which a more expensive reconstruction step examines only the selected subset rather than the entire original prompt. On RULER at 16,000 tokens with a 2% KV-cache budget, the authors report an average-score improvement exceeding 40 percentage points over the next-best evaluated baseline. The method also achieved strong results across LongBench cache budgets from 2% to 10%, with lower compression-stage runtime and peak memory than full-context reconstruction.

Why it matters: Repeated context creates infrastructure costs even when API providers discount cached tokens. Selectively preserving useful inference state can reduce memory requirements without repeatedly processing the complete context.

Hybrid Semantic Tool Discovery for Enterprise MCP Gateway: Architecture and Implementation

This August paper remains important in light of this week's enterprise agent announcements and Uber's published tool-surface measurements. Researchers describe SCOUT, a production tool-discovery system at PayPal that exposes two MCP meta-tools instead of injecting an entire enterprise tool catalog into model context. The system combines BM25 retrieval, dense vector search, and Reciprocal Rank Fusion to select relevant tools from more than 2,000 indexed tools across more than 200 MCP servers. The authors report reducing tool-token consumption from 140,200 tokens, or 70.1% of context, to approximately 1,300 tokens, or 0.8%, representing a 99% reduction. These are the authors' reported production measurements.

Why it matters: The tool-surface tax can become a substantial recurring input cost. This production case demonstrates that large capability catalogs can remain accessible without requiring every tool schema to occupy the model's working context.