Token Cost Radar

1Why are 73% of AI projects exceeding their budgets?

The Elec published the first installment of an AI FinOps series on October 20, examining why conventional cloud cost-management practices struggle with generative AI. Citing Harness's 2026 State of AI in FinOps survey of approximately 700 enterprise IT leaders, the publication reports that more than 67% of surveyed organizations spend at least $250,000 monthly on AI, while 73% of AI projects exceeded their budgets. Harness also estimated that approximately 26% of monthly AI budgets are wasted. The report identifies four major complications: token-based pricing, multimodel deployments, rapidly changing model capabilities, and increasingly complex agent workflows. Unlike predictable infrastructure workloads, agents can produce different output lengths, reasoning paths, and tool-call sequences for similar requests. Providers also impose different rate limits, caching policies, capacity commitments, and excess-usage rules. The survey figures are vendor-reported findings from the sampled organizations, not independently established rates for the entire enterprise market. The practical implication is that AI cost governance increasingly requires request-level observability, workload attribution, and budget enforcement rather than retrospective invoice analysis.

2AWS reports threefold faster first-token delivery by changing KV-cache transfers

AWS engineers published an October 19 technical report describing a push-based key-value cache transfer mechanism for disaggregated inference using vLLM. Disaggregated serving separates prompt processing from token generation, allowing each phase to run on independently optimized infrastructure. However, the resulting KV cache must move between the prefill and decode servers. Conventional pull-based transfers introduce coordination and setup operations after prefill completes, delaying generation. The new implementation registers destination memory earlier and moves transfer preparation into a background thread. In AWS benchmarks using Qwen3-32B on EC2 P5en instances connected through Elastic Fabric Adapter, push-based transfers improved time to first token by up to approximately threefold and time per output token by up to 30%. At eight queries per second with 16,000-token inputs, first-token latency fell from approximately 1,744 milliseconds to 540 milliseconds. Prefix caching was deliberately disabled to isolate transfer performance. These are workload-specific latency measurements, not demonstrated reductions in commercial token prices. The work shows that inference efficiency depends on coordination and memory movement as well as model computation.

3DeepSeek introduces per-call retrieval pricing as agents challenge token-based billing

Forkast's October 19 analysis examines DeepSeek's newly released Search and Fetch APIs, which charge $0.75 per 1,000 successful requests for each endpoint. Unlike model inference, which remains priced according to input and output tokens, these retrieval operations have a fixed charge independent of the amount of returned text. DeepSeek's documentation confirms that unsuccessful requests, empty results, and timeouts are not billed. Server-side Search and Fetch calls through supported coding-agent integrations are temporarily free, with billing scheduled to begin October 31 at 16:00 UTC. At the published rate, 10,000 successful search requests would cost $7.50, excluding subsequent model inference and any separate fetch operations. Forkast compares the rate with established retrieval providers charging approximately $5 to $12 per 1,000 requests, depending on service and configuration. DeepSeek imposes a five-request-per-second limit per endpoint and account. The development does not eliminate token-based inference billing, but it introduces a more predictable pricing unit for a common agent operation. Separating retrieval charges from reasoning charges could make high-frequency agent workflows easier to budget and evaluate.

4Nvidia-backed doubleAI claims a 67% reduction in customer computing costs

The Wall Street Journal reports October 20 that doubleAI, a Tel Aviv startup co-founded by Amnon Shashua, is introducing a technology category it calls code execution models. Its first model, Maxwell, is designed to improve the execution efficiency of software running on CPUs and GPUs rather than serving as another general-purpose conversational model. According to the company's claims reported by the Journal, the technology reduced one early customer's computing costs by 67%. DoubleAI has raised approximately $220 million from investors including Nvidia, Lightspeed Venture Partners, Bessemer Venture Partners, and Dell. The company intends to charge customers a percentage of the savings its technology generates. The reported savings are an early customer result, not an independently verified universal benchmark. The relevance to token economics is indirect but substantial: improvements in the software executing inference workloads can lower infrastructure expenditure without requiring changes to model weights, prompts, or token counts. Whether those savings reach API customers depends on deployment economics and provider pricing.

Neuro-Inspired Inhibitory Message Passing for Accurate and Token-Efficient Multi-Agent Reasoning

Submitted October 17 and appearing in the October 20 arXiv listings, this paper introduces a training-free communication mechanism inspired by inhibitory signaling in biological neural systems. Instead of allowing agents to repeatedly exchange entire messages, reasoning traces, and overlapping evidence, the method identifies evidence sentences that recipients have not already received. It forwards useful information once while suppressing redundant material and excluding the sender's reasoning and answers. Across three benchmarks, the researchers report improvements of 39 to 77 percentage points in peer-group accuracy while reducing tokens per run by 10% to 46%. The gains were evaluated across seven open-weight models, two commercial models, groups containing up to 64 agents, and several communication architectures. These are experimental results under the tested configurations, not guaranteed reductions in commercial inference bills.

Why it matters: Multi-agent collaboration can become expensive when every participant repeatedly processes information already known to the group. The research demonstrates that selectively transmitting new evidence can improve both reasoning quality and token efficiency. It provides a concrete alternative to simply shortening messages or imposing smaller context limits.

Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets

Submitted October 17 and newly listed October 20, Koi investigates how to configure inference infrastructure across heterogeneous clouds, accelerators, and workloads. The researchers identify approximately 140 interacting deployment variables, including hardware selection, parallelism, engine configuration, and replica counts. Koi combines language-model-assisted planning with deterministic performance prediction, candidate evaluation, and placement controls. It maintains a causal representation linking infrastructure decisions to runtime behavior, cache pressure, communication overhead, throughput, and latency. Across its benchmarks, Koi achieved 1.18 times the goodput of the strongest evaluated baseline while reducing cost per token by 20% in a real heterogeneous cluster. Ablation experiments showed that adding deterministic tools and structured planning improved results over using a language model alone. These findings apply to the evaluated infrastructure and workloads.

Why it matters: Model selection is only one part of inference economics. Hardware placement, utilization, cache behavior, and serving configuration also determine the effective cost of generating tokens. The research demonstrates how structured planning and deterministic measurement can improve those economics without requiring a new model architecture.

ATEST: Adaptive Resource-Aware Test-Time Scaling for LLM Inference

Submitted October 18 and appearing in the October 20 research listings, ATEST examines how reasoning systems should allocate additional inference when GPU operating conditions change. Conventional test-time scaling generates multiple candidate answers according to a fixed sampling budget, regardless of question difficulty or current infrastructure load. The researchers measured up to a 4.1-fold increase in energy consumption for the same four-sample query as GPU contention increased. ATEST-Online monitors answer stability and current GPU conditions, learns the energy cost of additional samples, and decides when to stop. Across four evaluated datasets using Phi-3-mini on an NVIDIA A100, the method maintained competitive accuracy while respecting energy budgets. On GSM8K under high load, it achieved 60% accuracy without deferrals, compared with 44% accuracy and 42.4% deferrals for the calibrated alternative. Including offline profiling, the online approach used 42 times less GPU time and 48 times less energy in the evaluated comparison.

Why it matters: The cost of another reasoning attempt is not constant. It depends on both the expected improvement in answer quality and the current cost of computation. Adaptive reasoning budgets can account for these conditions instead of treating every additional sample as equally valuable or equally expensive.

KVCompass: A Discovery Plane for Internet-Scale Prefix Cache Sharing in LLM Serving

Submitted October 16 and newly listed October 20, KVCompass investigates how inference providers can reuse previously computed key-value cache state across geographically distributed serving regions. Conventional prefix caching generally operates within individual clusters, leaving identical prompts to be processed again when requests arrive elsewhere. KVCompass introduces a discovery mechanism that identifies which workers hold reusable cached prefixes and routes requests according to cache availability, server load, and network distance. Its evaluation includes a simulated 64-worker deployment and a 14-worker testbed spanning three continents. The researchers report 1.3 to 14 times lower 99th-percentile time to first token than evaluated baselines, up to 3.7 times lower tail end-to-end latency, and up to 4.7 times higher throughput. These are experimental serving-performance results, not direct measurements of commercial API price reductions.

Why it matters: Repeated context processing can waste GPU capacity even when identical information has already been processed elsewhere. Sharing reusable cache state across regions offers another route to improving inference efficiency, although network-transfer costs, cache discovery overhead, and workload locality remain important considerations.

Cache2LoRA: Understanding and Improving Context Internalization

Submitted October 16 and appearing in the October 20 listings, Cache2LoRA investigates representing document knowledge through compact model adaptations rather than retaining the complete source text in active decoding context. The method retrieves relevant evidence from a reusable frozen document cache and constructs low-rank adaptation updates that preserve information needed for answering. Across 17 evaluated tasks, Cache2LoRA achieved a macro F1 score of 0.663, approaching the 0.671 obtained with full-context in-context learning. For 32,000-token documents and 256-token outputs at batch size 16, the researchers report decoding throughput 5.9 times that of the full-context baseline. The paper also examines how well compressed representations retain facts and follow instructions when additional information intervenes. These are research results from the evaluated models and workloads, not established reductions in commercial API charges.

Why it matters: The paper explores a distinction between information a system can access and information that must remain in its active decoding context. Reusable representations may reduce repeated processing of large documents, but their economic value depends on construction costs, reuse frequency, factual fidelity, and the ability to recover relevant details when needed.