Token Cost Radar

1How do you handle an AI budget that is already blown?

The SaaS CFO argues that many companies set 2026 AI budgets before real usage patterns were clear and are now facing midyear reallocation decisions as AI spend grows faster than planning cycles.

Why it matters: This is the practical CFO version of token governance: budgets need to move from static annual guesses to active controls.

2The token price does not equal the real inference cost

CoreWeave explains that per-token pricing hides infrastructure costs such as autoscaling overhead, utilization gaps, latency, model loading, and capacity planning.

Why it matters: Token economics are broader than API sticker prices. Real cost lives in the serving system around the token meter.

3Claude Code creator says ROI focus is right, but experimentation still needs room

Business Insider reports that Anthropic's Boris Cherny supports focusing on AI ROI while warning that overly tight token controls can suppress useful experimentation and unexpected productivity gains.

Why it matters: The operating model is becoming bounded exploration, not unlimited burn or premature austerity.

4Cheap Chinese and open-source models gain enterprise attention

Axios reports that open-source models from China, including DeepSeek, are forcing executives to weigh cost and performance benefits against security, compliance, and geopolitical risk.

Why it matters: Cheaper tokens are no longer only a pricing story. They are becoming a governance, sourcing, and risk-management story.

5European firms diversify AI providers as token costs and access risks rise

Reuters reports that firms including Siemens, Renault, and Orange are diversifying AI providers while rising automated token consumption creates cost-control pressure.

Why it matters: AI cost governance is now intertwined with provider dependence, sovereignty, and infrastructure readiness.

Token Budgets: An Empirical Catalog of 63 LLM-Agent Budget-Overrun Incidents

This arXiv paper catalogs production LLM-agent budget failures and evaluates an affine-typed Rust mitigation for non-bypassable token budget ownership.

• Catalogs 63 confirmed production incidents from 21 orchestration frameworks.
• Identifies retry loops and delegation fanout as budget hazards.
• Uses affine ownership to prevent double-spend and post-delegation budget misuse.
• Reports zero cap violations in the evaluated live-API test.

Why it matters: Token overruns are being documented as a production reliability failure class, not merely a billing surprise.

Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference

This arXiv paper routes LLM requests to short-context or long-context serving pools based on estimated token budget.

• Targets waste from provisioning every vLLM instance for worst-case context length.
• Learns request token-budget estimates online from usage feedback.
• Reports 17 to 39 percent GPU instance reductions in simulated traces.
• Projects large annual savings at high request rates.

Why it matters: It turns token budget estimation into an infrastructure routing primitive.

The Price of Anarchy in Disaggregated Inference

This arXiv paper analyzes disaggregated inference as interacting games across prefill, decode, cache, and request routing decisions.

• Uses NVIDIA Dynamo as a concrete disaggregated serving case study.
• Models resource contention between prefill and decode pools.
• Finds saturation can sharply worsen latency and routing outcomes.
• Reports adaptive routing improved saturated-phase efficiency in evaluated topologies.

Why it matters: As serving stacks split apart, inference cost optimization becomes a control and routing problem.

Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving

This arXiv paper argues that agentic serving should schedule at the conversation level rather than predicting each turn in isolation.

• Raises the scheduling unit from individual turns to full conversations.
• Routes first-turn prefill separately from the long memory-bound tail.
• Reports 51.08 percent lower p95 time-to-first-effective-token versus a per-turn prediction baseline.
• Reports additional energy-efficiency gains with heterogeneous GPU tiers.

Why it matters: Agent cost and latency depend on multi-turn structure, not just single-call token counts.

TokenPilot: Cache-Efficient Context Management for LLM Agents

This arXiv paper addresses context accumulation in long-horizon LLM-agent sessions, where growing context drives inference cost.

• Focuses on cache-efficient context management for long-running agents.
• Targets inference cost growth from accumulated conversation and task state.
• Treats context management as a serving efficiency problem.
• Fits the broader shift from bigger context windows to better context discipline.

Why it matters: Agent token efficiency increasingly depends on managing accumulated context, not merely compressing individual prompts.