Token Cost Radar

1What happens when AI token usage keeps climbing but the money stops following?

Axios reported October 22 that effective AI token prices have fallen sharply, raising questions about whether expanding consumption will generate enough revenue to justify enormous infrastructure investments. Silicon Data's volume-weighted LLM Token Expenditure Index recently fell below $1 per million tokens, down 54% from its late-May peak of $2.07. Ramp's separate index, based on actual business spending with OpenAI and Anthropic, declined 48% from its late-February peak of $1.22 per million tokens. The indexes measure different populations and should not be treated as interchangeable. More consequentially, Ramp economist Ara Kharazian told Axios that business token spending remained essentially flat during August and September even as token usage increased. Silicon Data attributes part of the effective-price decline to organizations routing requests away from expensive frontier models toward cheaper alternatives. Model-provider price reductions and discounted cached inputs also contribute. Axios notes that Goldman Sachs expects major US hyperscalers to spend approximately $800 billion on capital expenditures during 2026 and $1.1 trillion in 2027. Those are forecasts, not committed expenditures. The emerging tension is straightforward: increasing token consumption does not guarantee proportionately increasing revenue for inference providers.

2Cognizant introduces AI Nexus as 72% of surveyed FinOps leaders report budget overruns

Cognizant announced AI Nexus on October 22, introducing a modular operating layer intended to connect AI spending, execution quality, model selection, and business outcomes. A new Cognizant survey of 250 senior FinOps leaders found that 72% of respondents' organizations exceeded their AI budgets during the preceding 12 months, while 16% overshot by more than 25%. The company also reports that 97% experienced at least one AI agent session exceeding expected cost or activity, with 36% describing the impact as material. The survey covered organizations with at least $500 million in annual revenue across the United States, United Kingdom, and India. AI Nexus combines five modules. Reveal is designed to attribute tokens, tool calls, and expenditure to projects and outcomes. Route selects models according to quality, cost, and control requirements, including frontier, open-weight, and self-hosted alternatives. Coach evaluates execution quality, while Engage and Orchestrate address workspaces and agent workflows. Cognizant describes these as expected capabilities and potential benefits, not independently demonstrated customer savings. The announcement illustrates how enterprise AI governance is expanding beyond measuring consumption toward connecting every dollar spent with a defensible operational result.

3Harness turns AI spending policies into weekly limits that can stop execution

Harness introduced AI Budgets on October 20, with additional industry coverage appearing October 22. The feature, part of its Cost Management Agent, allows organizations to establish weekly AI spending limits by team, developer, model, or project. At 90% of an allocated budget, the system issues a warning without restricting access. Once spending exceeds 100%, access pauses until the next weekly reset or an approved budget increase. Employees can request additional allocations through a self-service approval workflow. Harness argues that weekly enforcement is more effective than monthly reporting because uncontrolled agent execution, oversized model selection, and repeated retries can generate substantial expenses before an invoice arrives. The company cites research indicating that 73% of organizations have an AI cost policy but only 13% can determine whether that policy is working. Those figures are vendor-cited research findings rather than independently established market-wide rates. The feature addresses a practical limitation of conventional AI FinOps dashboards: identifying overspending after consumption has occurred is not the same as preventing it. However, hard limits can interrupt useful work, making exception handling and budget allocation important operational decisions.

4CoreWeave highlights research claiming up to 66% lower inference cost per token

CoreWeave highlighted a Signal65 infrastructure economics study on October 21. The underlying research was originally dated September 30 and compares NVIDIA Blackwell deployments across CoreWeave and three anonymized hyperscaler clouds. Using published infrastructure prices and MLPerf Inference v6.0 throughput results, Signal65 modeled a 5,000-GPU deployment over three years. The analysis reports that CoreWeave delivered up to 65% lower modeled infrastructure ownership costs on NVIDIA GB200 NVL72 systems. For DeepSeek-R1 inference, combining throughput and infrastructure pricing produced a reported cost-per-token advantage reaching approximately 66%. A separate comparison across 17 hosted models found median input-token prices 26% lower and output-token prices 15% lower on CoreWeave. The study was commissioned by CoreWeave, and its results depend on selected workloads, configurations, public pricing, and modeled utilization. They are not independently observed savings across arbitrary enterprise deployments. The important economic distinction is that GPU rental prices alone do not determine inference costs. Throughput, utilization, storage, networking, and supporting services determine how much useful output a deployment produces for each dollar spent.

uGDS: A User-Space Development Kit for High-Performance GPU Direct Storage

Submitted October 22, this paper introduces uGDS, an open-source GPU storage interface designed to reduce the software overhead involved in transferring data directly between NVMe SSDs and accelerator memory. Conventional GPU Direct Storage avoids unnecessary host-memory copies but still incurs kernel transitions and synchronization overhead. uGDS moves NVMe command management into user space, supports workload-aware scheduling, and provides compatible interfaces for NVIDIA CUDA and AMD HIP. In experiments using an NVIDIA A100 and Samsung 990 PRO SSD, the researchers report up to 12.12 times higher batch-read bandwidth and 30.69 times higher batch-write bandwidth for selected small-request configurations. More directly relevant to inference, integration with LMCache reduced median time to first token by up to 62.7% and increased serving goodput by 2.63 times. These are hardware- and workload-specific experimental results, not independently measured reductions in commercial API prices.

Why it matters: Long-context inference increasingly depends on moving cached model state between storage and accelerator memory. Faster storage operations can reduce the time GPUs spend waiting for reusable context. The paper demonstrates that storage software, not just accelerator hardware, can materially affect inference throughput and the effective cost of serving tokens.

Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving

Submitted October 22, Gandalf examines a less obvious expense associated with inference caching: the physical wear caused by repeatedly writing key-value cache state to solid-state storage. Long-running agents and multi-turn conversations can generate enough cache writes to shorten SSD service life substantially. Simply rejecting cache writes based on historical access frequency can reduce storage wear but also discard useful information, lowering cache-hit rates and increasing recomputation. Gandalf instead considers the attention architecture associated with cached state when deciding what should be written to SSD. It prioritizes cache entries from full-attention layers when their future reuse characteristics justify storage. Across the evaluated workload traces and hardware configurations, the researchers report extending SSD lifetime by an average of 7.3 times while maintaining nearly unchanged cache-hit rates. The paper does not quantify a corresponding reduction in complete data-center operating costs.

Why it matters: Caching reduces computation but creates storage, bandwidth, and hardware-maintenance expenses. Gandalf demonstrates that the cheapest cache is not necessarily the one retaining the most information. Cache admission policies must consider future reuse and the physical cost of preserving state, particularly as inference systems increasingly rely on SSD-backed memory expansion.

MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

Submitted October 21 and appearing in the October 23 arXiv listings, MoEZip investigates compressing the key-value caches of mixture-of-experts language models without disrupting their expert-selection behavior. Conventional cache eviction techniques generally evaluate attention importance, but removing cached information can also change which experts a sparse model activates during generation. The researchers introduce a routing-sensitivity metric derived from the Fisher information of expert-routing distributions and combine it with attention-based importance scoring. At a cache-retention ratio of just 10%, MoEZip improved average LongBench and RULER scores by 24 and 20 points respectively over the evaluated state-of-the-art baseline. The authors describe those results as 2.3 and 4.9 times higher benchmark scores. These are comparative quality results under aggressive cache compression, not measurements of commercial token-price savings. The paper reports acceptance at NeurIPS 2026.

Why it matters: Reducing cache memory can improve inference capacity, but compression that changes model behavior may create additional retries or unacceptable answers. MoEZip shows why efficient inference requires preserving the information that matters to the model's actual computation, not merely reducing memory consumption as aggressively as possible.

Repair Changes Which Layers to Remove: Exact Optima for Depth Pruning of Language Models

Submitted October 22, this paper investigates how to remove entire transformer layers while preserving language-model performance. Depth pruning can reduce computation and potentially improve inference efficiency, but selecting which layers to remove is difficult because different combinations interact. Some approaches attempt to restore quality by replacing removed layers with inexpensive linear transformations. The researchers exhaustively evaluated approximately 53,000 pruning configurations across four model families, comparing configurations with and without repair. They found that the optimal pruning pattern changes when repair is included, and that selecting layers without accounting for the eventual repair mechanism can miss the repaired optimum by up to 189% under the study's comparison metric. Their proposed greedy selection method reduced perplexity by 20% to 66% relative to ReplaceMe across five evaluated models containing up to 27 billion parameters. Under a task-specific objective, accuracy improvements reached 25 points. The study does not directly establish corresponding reductions in hosted inference prices.

Why it matters: Model compression is often treated as a straightforward way to reduce computational expenditure. This research demonstrates that optimization decisions must be evaluated as part of the complete execution system. Removing the wrong layers can damage quality, while accounting for subsequent repair can produce a better balance between model size and performance.