Token Cost Radar

1Why is AI spending up thirteenfold when the average token price is down 47%?

Vercel's October 15 AI Gateway Production Index reports that September spending across its gateway was approximately thirteen times December's level, even as the average price per token declined 47%. Open-weight models processed 71.5% of gateway tokens while accounting for approximately 23% of spending. DeepSeek's models alone handled 58% of tokens for 6.6% of spending. The report also identifies substantial differences between inexpensive high-volume models and premium models that attract disproportionate spending. Nearly one-third of teams changed their most-used model provider in September, suggesting increasingly fluid purchasing decisions. Chinese-origin models are also generating significant revenue for American infrastructure providers: nearly 70% of September gateway spending on those models went through US-based providers. These figures describe Vercel's observed traffic and estimated spending, not the entire inference market. The central finding is that declining unit prices and rapidly increasing aggregate consumption are occurring simultaneously.

2IBM finds open-weight agents can cut specialized design inference costs by 11.5 times

IBM Research published results highlighted October 15 from an autonomous agent system designed to optimize quantum error-correction decoder hardware. Under a matched budget of 100 design hypotheses, the all-proprietary configuration consumed approximately 215.9 million tokens and cost $299.45 in inference, achieving a 204-fold reduction in the decoder's estimated hardware latency. An all-open-weight configuration using GLM-5.3 and distributed inference consumed approximately 240.9 million tokens but cost only $25.95, achieving a 52-fold latency improvement. A hybrid configuration cost $105.25 and achieved a 126-fold improvement. The open-weight approach was therefore substantially cheaper despite consuming more tokens, but it did not match the proprietary configuration's best hardware result. IBM also reports a 98% prefix-cache hit rate in its open-weight deployment. The experiment demonstrates why token counts, inference bills, and completed-task quality must be evaluated together rather than treated as interchangeable efficiency measures.

3Tensormesh launches enterprise KV-cache management to avoid repeated inference computation

Tensormesh announced October 15 that its enterprise inference-caching platform is available for Kubernetes deployments. The system manages reusable key-value cache state across GPU high-bandwidth memory, CPU memory, and storage, allowing inference servers to retrieve previously computed context instead of repeatedly performing the same work. Its capabilities include prefix caching, reuse of non-prefix prompt blocks, peer-to-peer cache transfers, and policy controls governing retention and eviction. The company also describes security, observability, resilience, and tenant-isolation features intended for production deployments. Tensormesh argues that redundant context processing wastes GPU capacity and increases time to first token. The announcement does not establish a universal percentage reduction in inference bills, and actual savings depend on cache reuse, storage costs, workload characteristics, and deployment configuration. Nevertheless, it illustrates the growing importance of managing inference memory as a shared infrastructure resource rather than treating every model request as an independent computation.

4A new AI gateway puts token budgets and agent spending limits into the request path

AGAT Software announced its Pragatix AI Gateway on October 15, positioning the service as a centralized control layer for enterprise inference spending and security. According to the company, administrators can track costs by user, team, application, agent, project, model, and provider. The gateway supports cost-based and token-based limits, threshold alerts, and blocking requests when spending limits are exceeded. AGAT also describes cost-aware model routing, prompt caching, duplicate-request safeguards, and inspection of agent tool interactions. The service is intended to work across external API providers, private models, and internal AI systems. These are vendor-announced capabilities, not independently verified customer savings. The broader development is the convergence of AI FinOps and operational governance: budget enforcement is moving closer to individual inference requests and agent actions rather than relying exclusively on monthly invoices and retrospective dashboards.

LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer-Use Agents

Submitted October 15, LEASH investigates how to reduce the cost of long-running computer-use agents by allowing smaller models to perform routine steps and escalating to stronger models when uncertainty increases. The framework uses the smaller model's internal activations to estimate uncertainty without requiring a separate monitoring model or additional forward pass. It combines initial task assessment, step-level escalation, and continued oversight after a stronger model takes control. On OSWorld, the researchers report improvements of 6.55 to 11.94 percentage points over random escalation at matched escalation rates. Using Qwen3.5-35B-A3B, LEASH achieved 73.66% task success at $0.4117 per task, compared with 72.90% success for always-on Claude Opus 5, while reducing measured inference cost by 19.2%. These results apply to the evaluated computer-use tasks and model configurations.

Why it matters: The research demonstrates a practical approach to model routing that considers uncertainty during execution rather than assigning one model to an entire task. Its strongest economic feature is avoiding additional inference merely to decide whether more expensive inference is necessary.

Disagreement-Guided Weak-to-Strong Routing

Submitted October 15, this paper examines model routing when reliable ground-truth correctness labels are unavailable. Instead of requiring a trained classifier to predict whether a smaller model's answer is correct, the researchers derive a routing strategy based on disagreement between model-generated responses. Their analysis allows multiple responses from a smaller model and characterizes how comparisons with a stronger model can approximate an optimal escalation policy. Across evaluated single-turn and agentic tasks, disagreement-guided routing achieved favorable tradeoffs between response accuracy and routing cost. The authors also report that disagreement-based routing can outperform methods trained directly to predict correctness, even when labeled correctness data is available. The abstract does not establish a universal percentage reduction in inference spending.

Why it matters: Cost-aware routing often assumes that a system already knows when a cheap model will fail. This work addresses the cost of obtaining that knowledge. Better escalation decisions can preserve quality while limiting unnecessary frontier-model calls, although comparison and sampling overhead must remain part of the economic calculation.

DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction

Submitted October 15 and accepted at NeurIPS 2026, DropKV addresses the growing memory cost of retaining key-value cache state during long-context inference. The method evaluates the importance of individual cached tokens using a mathematical approximation of their effect on model output. By separating cache-eviction decisions into independent token scores, it avoids a difficult combinatorial optimization problem. The researchers implement the scoring method with fused Triton kernels that do not require constructing the complete attention matrix. They report a 19.8-fold speedup in the scoring kernel and up to 9.9% faster end-to-end prefill at matched batch sizes. Evaluations on RULER and LongBench indicate improved quality and latency tradeoffs against the tested eviction baselines at equivalent cache budgets. These are experimental performance measurements, not direct commercial API cost reductions.

Why it matters: Long-context inference is constrained by memory as well as computation. More selective cache retention can reduce resource requirements without assuming every previously processed token deserves permanent residence in expensive accelerator memory.

FAOPs: Normal Forms and an Algebra for AI Agent Workflows

Submitted October 15, this paper proposes applying database-style normalization and compiler principles to reusable agent operating procedures. The author identifies problems with conventional agent skills, including bundled operations, duplicated procedures, unrestricted tool access, and poorly defined execution boundaries. Functional Agent Operating Procedures reorganize workflows into atomic, typed, reusable steps with explicitly scoped tool permissions. The framework also introduces formal extensions for termination verification, optimization, and time and cost bounds. An architectural classifier helps determine whether a workflow requires AI at all. The paper presents a formal system and worked financial example rather than a large-scale production benchmark demonstrating percentage token savings. Its significance is the treatment of agent cost and tool access as properties that can be specified and checked before execution.

Why it matters: Agent efficiency is not exclusively a prompt-engineering problem. Making workflows reusable, limiting tool exposure, and moving deterministic behavior into executable procedures can reduce repeated reasoning and unnecessary tool-schema processing. The research offers a formal foundation for those controls, although practical savings remain to be measured.