Token Cost Radar

1When does the AI bill become a financing problem?

Reuters reports that Alphabet is seeking to raise roughly $20 billion to $25 billion in a new bond sale after increasing its 2026 capital-spending forecast to $205 billion. The spending surge is tied heavily to AI infrastructure and comes as Alphabet reported negative quarterly free cash flow, showing that AI economics now extend well beyond per-token API prices.

2Uber says its tokenmaxxing era is ending

Business Insider reports that Uber CTO Praveen Neppalli Naga says the company is moving beyond maximal AI consumption after quadrupling its use of frontier AI tools this year. Uber says prompt caching, cheaper default models, better developer cost visibility, and experiments with open-weight models have lowered per-token costs.

3Microsoft makes token restraint an organizational control

TechRadar reports that Microsoft is introducing token budget targets for divisions, tracking internal AI consumption, and steering employees toward a cheaper default model. Executive vice president Jay Parikh told employees that tokenmaxxing is not the objective and that AI consumption should be tied to customer and business outcomes.

TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure

This new arXiv paper proposes a shared tensor-management layer spanning model weights, KV caches, checkpoints, and request routing. Integrated with vLLM and SGLang, one programmable policy improved median time to first token by as much as 93.2% under highly concurrent multi-turn agent workloads.

Why it matters: Inference efficiency increasingly depends on coordinating caches, routing, and persistent state across the serving stack rather than optimizing each component independently.

Runtime Observability for Heterogeneous Attention Memory

This new arXiv paper introduces a runtime risk ledger for compressed attention memory across KV caches, latent caches, sparse selectors, and recurrent state. The authors replayed more than 12.4 million memory-entry reads and applied the framework to a served DeepSeek-V4 stack.

Why it matters: Compression can improve inference economics only if the system knows when it is crossing a reliability boundary. Memory budgets need observable error budgets alongside them.

LaCache: Robust Semantic Caching for LLM Serving

This arXiv paper examines cache-collision attacks against semantic caching and proposes checking both query similarity and the first speculative output tokens before accepting a hit. The design aims to retain semantic caching's inference savings while preventing poisoned entries from silently contaminating later responses.

Why it matters: Semantic caching can eliminate repeated model calls, but a cheap wrong answer has terrible token yield. Cost controls need correctness boundaries, not just hit-rate targets.

Control Under Compression: Reliability Frontiers for Tool-Using Agents

This arXiv study evaluates 15,525 agent runs while progressively compressing persistent tool, policy, argument, and recovery instructions. Retaining 75% of control context kept success near the full-context baseline, but reliability fell sharply at more aggressive compression levels, particularly through tool-execution and action-parsing failures.

Why it matters: Tool and control context are legitimate tokenminimizing targets, but the savings curve has a cliff. The useful budget is the smallest one that still preserves executable behavior.

Practical Online KV Cache Compaction for LLM Agents

This arXiv paper studies online KV-cache compaction for long-running agents and finds that immediate compression often hurts task performance because future relevance is unknown. Token eviction preserved most accuracy while reducing KV-cache size by 80% in evaluated settings, with delayed compaction generally outperforming premature pruning.

Why it matters: Context efficiency has a timing dimension. Removing information before the agent knows what matters can trade an infrastructure saving for a task failure.