Token Cost Radar

1What does your AI bill measure when almost every token is cached?

A September 24 analysis repriced 7,850 turns and 2.9 billion input-side tokens from six long-running agent sessions against Claude Opus 5 and Opus 5.5 rates. Only 0.015% of traffic was uncached input, while 93.4% was cache reads and 6.6% cache writes. Holding workload constant, the newer rate card reduced modeled cost 35.3%, from $3,534 to $2,286, largely because cache-read pricing fell 60%. Cache writes still represented 67.1% of the resulting Opus 5.5 bill.

2Public inference reaches 19.83 trillion tokens a day

Grid Intelligence Indices' September 24 print estimates 19.83 trillion routed tokens per day across its OpenRouter top-50 methodology, up 10.8% week over week. Daily notional spend reached $20.5 million while its spend-weighted basket unit cost rose 5.7% to $8.03 per million tokens. The split illustrates why aggregate AI spend can move differently from nominal provider price cuts: workload volume and model mix are moving too.

3Enterprise AI cost management expands beyond the token bill

Flexera's September 25 AI cost-management guidance tells enterprises to inventory model APIs, cloud GPUs, vector databases, AI software licenses, agents, users, and infrastructure before building an optimization case. It recommends model rightsizing, prompt caching, agent governance, GPU utilization, and contract consolidation, framing AI FinOps as a full cost-to-serve discipline rather than a per-token accounting exercise.

4Shared KV cache delivers 2.4 times more agentic inference on identical GPUs

Nebius and WEKA published September 24 benchmark results for shared KV-cache infrastructure on identical NVIDIA HGX B300 hardware. Their disaggregated cache setup served 2.4 times more agentic inference sessions than the comparison configuration by moving reusable context out of scarce GPU memory and into a shared NVMe-backed tier. The result reinforces a growing inference-economics theme: previously computed context is becoming an infrastructure asset rather than disposable intermediate state.

Total Cost of Agency: Exact Attribution of Memory Injection Cost in Multi-Agent LLM Workflows

This September paper decomposes agent cost into base prompt, inference, memory injection, miss penalty, and context accumulation. Across a 200-task enterprise benchmark using real model APIs, memory injection represented about 12% of full billed cost and rose to 27.6% of controllable variable cost at workflow depth six. Reducing retrieval-window capacity from 32 entries to 2 cut injected tokens by 28.7% with accuracy changes within seed-level variation.

Why it matters: Cost attribution changes optimization. Tokens added by agent memory look like ordinary input on the invoice, but identifying their origin exposes a specific lever that can be tuned without changing model tier.

AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows

AgentRouter assigns individual steps inside an agent trajectory to one of four model tiers rather than sending the entire workflow to a frontier model. Trained on 50,000 annotated trajectory steps, the authors report a 72% cost reduction versus frontier-only execution while retaining 97.3% of frontier-only quality, with less than 5 milliseconds of routing overhead per step on an A100.

Why it matters: This is modelmaxxing below the task level. Planning may justify frontier inference while formatting or extraction does not, so the economical routing unit can be an individual step rather than the whole agent.

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

This September 22 paper compresses long contexts into query-selected memory embeddings and combines them with a local context window in a two-tier KV-cache design. Across nine encoder-decoder combinations and four question-answering benchmarks, the authors report reductions of up to 20% in inference time and energy consumption and up to 50% in peak reserved GPU memory at 3,000 generation tokens.

Why it matters: Tokenminimizing can happen below visible text. Compressing relevant context into a smaller inference representation can reduce physical serving cost without simply deleting information from the task.

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

MetaKV chooses a KV-cache compression configuration separately for each prompt using user-specified latency and peak-memory budgets. Across ten configurations from three compression methods plus uncompressed FP16, it improved the fraction of prompts answered correctly while meeting both resource constraints by about 0.07 on average and as much as 0.135 versus the best static configuration.

Why it matters: Inference yield increasingly depends on adaptive resource allocation. The best cache policy changes with the prompt and the budget, just as the best model route changes with task difficulty.