Token Cost Radar

1How did Anthropic cut compute costs from 251% to 48% of revenue?

A November 2 analysis by FourWeekMBA examines Anthropic's October 26 Form S-1 and finds a dramatic improvement in the company's compute economics. Anthropic reported $5.558 billion in compute and infrastructure expenses during the second quarter of 2026, representing approximately 48% of quarterly revenue. In the first quarter of 2025, the corresponding ratio was 251%. Revenue grew faster than infrastructure expenditure, but the absolute compute bill increased substantially. Anthropic spent $8.945 billion on compute and infrastructure during the first half of 2026. The filing also disclosed approximately $265.6 billion in future fixed or minimum compute commitments as of June 30, with additional agreements signed afterward. These figures combine training, inference, and infrastructure expenses rather than isolating the cost of generating customer tokens. The improvement demonstrates stronger revenue generation relative to computing expenditure, not a comparable reduction in the physical cost of inference. Anthropic also faces substantial contractual obligations that will require continued demand growth and efficient infrastructure utilization.

2SAP replaces expensive AI models with cheaper alternatives while targeting €2 billion in efficiency gains

Reuters reported October 28 that SAP is replacing expensive AI models with lower-cost alternatives when those alternatives meet its performance requirements. CFO Dominik Asam said SAP uses more than 70 models from providers including OpenAI, Anthropic, and Google. The company increasingly selects the least expensive model capable of completing each task to the required standard. SAP expects the approach to contribute toward more than €2 billion in AI-driven efficiency gains, although that figure should not be interpreted as savings attributable exclusively to model substitution. SAP expects total AI expenditure to continue increasing as adoption expands. Its commercial model makes inference efficiency especially important because customers pay for completed tasks rather than the underlying computing resources. Lower execution costs can therefore improve margins without requiring lower customer prices. Reuters also reports that customers in regulated industries increasingly restrict model selection according to geographic origin and data requirements, adding compliance considerations to cost routing.

3Bank of America and Capital One demonstrate why smaller models belong in enterprise AI

Banking Dive reported October 29 that Bank of America and Capital One are using smaller models to improve the economics of enterprise AI. Bank of America's technology chief Hari Gopalkrishnan described Orchestra, an internal routing layer that uses a small classifier to select financially appropriate models for different requests. The bank plans to double its AI budget in 2027 while avoiding unnecessary dependence on expensive frontier models. Capital One's chief enterprise AI officer Prem Natarajan described distilling open-weight models into smaller, specialized models for its Chat Concierge application. According to Natarajan, the approach reduced latency fivefold and increased dealership customer engagement by 50%. These are company-reported outcomes rather than independently verified benchmarks. Bank of America also emphasized agent guardrails that restrict execution time and autonomous task scope. The examples demonstrate that model routing, specialization, and execution controls can support greater AI adoption without assigning the most expensive available model to every operation.

4Enterprise survey finds 89% have scaled back or redesigned AI plans over token costs

ITPro reported October 28 that Cloudera's State of Enterprise AI 2026 survey found 89% of responding organizations had scaled back or redesigned AI plans because of token costs. Another 45% reported moving workloads in response to cost volatility. Although 55% increased AI investment, only 23% said their initiatives generated more value than they cost. Some 58% reported approximately equivalent value and expenditure, while 27% reported little or no measurable business impact. Cost visibility remains incomplete: 90% said they could measure end-to-end AI costs at least somewhat accurately, but only 27% expressed strong confidence in that measurement. These are survey findings, not an independent audit of corporate AI expenditure. The report illustrates a growing distinction between AI adoption and economically sustainable deployment. Organizations are increasingly revisiting model selection, infrastructure placement, and FinOps controls rather than assuming that declining advertised token prices will automatically produce lower bills.

HiState: Hierarchical Prefix Cache Management for Hybrid LLM Serving in Agentic Workloads

Submitted October 30, HiState addresses prefix-cache management for hybrid language models combining attention and recurrent layers. Agent workflows frequently revisit earlier context, but conventional cache policies may discard the checkpoints required to reuse that computation. HiState stores historical recurrent checkpoints on SSD while prioritizing key-value cache entries in faster memory. Adaptive checkpoint intervals and coordinated eviction policies balance storage overhead against repeated computation. Evaluated using public coding-agent traces, the system achieved up to 5.73 times faster 99th-percentile time to first token than vLLM-Marconi and up to 5.44 times faster than SGLang-HiCache.

Why it matters: Long-running agents repeatedly revisit accumulated context. Better cache management can reduce expensive recomputation without requiring agents to shorten every conversation. The reported improvements concern tail latency in the evaluated serving configurations, not guaranteed reductions in commercial API bills.

ACCT: Asymmetric Cross-LLM Context Transfer

Submitted October 30, ACCT investigates transferring cached context between language models of different sizes within the same model family. Conventional model routing can require the destination model to process the entire prompt again, erasing some of the savings from switching to a cheaper model. ACCT transforms a source model's key-value cache into sparse states usable by the destination model. Its components identify relevant context tokens, allocate cache capacity, and compile the transferred representation. Across the evaluated benchmarks, the researchers report comparable or better task performance while reducing KV-cache size by 75%. At approximately 19,000 context tokens, cache conversion was 6.5 to 21.2 times faster than reprocessing the context with the destination model.

Why it matters: Model routing is less economical when every switch requires another full-context prefill. Cross-model cache transfer could make dynamic routing more attractive for long agent sessions. The technique remains limited to compatible model families and the evaluated experimental configurations.

LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction

Submitted October 30, LEAP improves inference scheduling by predicting response lengths using information already generated during model prefill. Conventional length predictors can require substantial additional computation and perform poorly when workloads change. LEAP extracts a lightweight signal from the serving model's hidden states and updates its predictions asynchronously on the CPU. In the researchers' experiments, LEAP reduced average per-token latency by 15.9% to 40.8% relative to evaluated length-prediction baselines. It also achieved 1.9% higher throughput while using approximately 30,569 times fewer predictor parameters than the strongest compared baseline.

Why it matters: An optimization system can become expensive enough to undermine the efficiency it was designed to create. LEAP demonstrates that useful scheduling decisions may be extracted from existing model computation rather than requiring another substantial model. The results are workload-specific latency and throughput measurements.

Trainable Vector Quantization for Large Language Models

Submitted October 30, this paper introduces a trainable approach to vector quantization intended to reduce language-model memory requirements while preserving quality. Extreme low-bit quantization can substantially reduce model size but often damages accuracy. The researchers use trainable latent weights and knowledge distillation to improve compressed models after quantization. Their experiments demonstrate compatibility with existing vector-quantization methods and report near-lossless two-bit quantization of LLaMA2-70B, with approximately a one-percentage-point accuracy reduction on selected question-answering benchmarks. The study does not establish a universal inference-cost reduction across models or hardware platforms.

Why it matters: Memory requirements remain an important constraint on economical inference, particularly for self-hosted and smaller-scale deployments. Better low-bit compression can expand hardware options, but real savings depend on execution speed, supported kernels, model quality, and the cost of producing the compressed model.