Token Cost Radar

1Why can the cheaper model leave you with the bigger AI bill?

The Wall Street Journal reports October 5 that businesses are struggling to forecast token-based AI spending, with only 11% of nearly 400 companies in one study accurately predicting their AI costs. Research covering more than 6,800 tasks found that cheaper models ended up costing more than premium alternatives in 32% of cases because of inefficiency or failure. In one example cited by the Journal, Gemini 3.1 Pro completed a task for $1 while the cheaper Gemini 3 Flash accumulated $14 in token charges and still failed. Sticker price and completed-task cost are increasingly different numbers.

2AI token prices are down more than 52% from their May peak

The Financial Times reports October 5 that the Silicon Data Token Expenditure Index has fallen more than 52% from its May 2026 peak. The decline reflects cheaper token production, competition, and enterprises shifting lower-priority workloads toward less expensive models. The FT also describes an increasingly two-tier market, with frontier providers maintaining premium economics while open-weight inference competes more aggressively on price. Token deflation is real, but it does not imply that total AI spend is falling.

3Open inference traffic now includes 2.42 trillion cached tokens in 28 days

Surplus Intelligence's October 5 marketplace snapshot records 3.062 trillion fresh input tokens, 2.425 trillion cache tokens, and 56.36 billion output tokens across 61.1 million requests over the previous 28 UTC days. Its latest seven full days of eligible traffic show an 88.4% mean realized discount from direct-provider pricing. Cache traffic equal to roughly 79% of fresh input again demonstrates why raw token volume cannot be translated directly into spend without knowing how those tokens were billed.

How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks

Across eight frontier models on SWE-bench Verified, researchers found agentic coding tasks consuming roughly 1,000 times more tokens than simpler code-reasoning and chat tasks. Repeated runs on the same task varied by as much as 30 times in total token consumption, higher spending did not reliably improve accuracy, and frontier models systematically underestimated their own eventual token usage.

Why it matters: Today's business-budgeting problem has an empirical explanation. Agent token consumption is stochastic enough that model self-estimates and simple per-task averages are weak budget controls. External metering and hard execution limits become necessary.

AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows

AgentRouter assigns individual steps inside an agent trajectory to one of four model tiers rather than sending the complete workflow to a frontier model. Trained on 50,000 annotated trajectory steps, the authors report 72% lower cost than frontier-only execution while retaining 97.3% of frontier-only quality, with less than 5 milliseconds of routing overhead per step on an A100.

Why it matters: Modelmaxxing becomes more precise when routing happens inside the workflow. The cheapest effective model for planning may be entirely different from the cheapest effective model for extraction, formatting, or routine tool use.

An Empirical Cost Attribution of Context-Compression Gateways in Multi-Turn Coding Agents

This study decomposes coding-agent token savings into tool-schema filtering, content compression, and history summarization. In controlled runs, tool-schema filtering removed roughly 21,000 to 57,000 tokens from a typical turn and was the most consistently positive lever. Content compression saved only about 2% of the cache-priced prefix per turn initially, but its savings accumulated across subsequent turns as compressed material stopped being repeatedly resent.

Why it matters: The tool-surface tax is unusually predictable because unused schemas are charged again and again. Before inventing elaborate compression machinery, removing tools the current task does not need can produce a simpler recurring saving.

The KV Cache Is the New Memory Wall

This recent systems review finds that long-context inference eventually becomes constrained by KV-cache memory traffic rather than arithmetic throughput. For Llama-3-70B in BF16, the authors calculate that one 128,000-token sequence adds roughly 42 GB of KV cache. The paper compares quantization, eviction, paging, prefix sharing, and heterogeneous tiering under a common framework.

Why it matters: Token economics has a physical layer. Long context increases memory capacity and bandwidth pressure, which can reduce concurrency and raise cost even when the API's nominal per-token price does not change.