Token Cost Radar

1How did Cloudflare cut its AI code-review bill 68% without abandoning capable models?

Cloudflare's October 19 engineering report documents how its internal AI code reviewer reduced average inference costs from $2.43 during its two most expensive weeks to $0.77 per review. Median completion time fell from four minutes and nine seconds to one minute and 32 seconds. The investigation began after average review costs climbed from $1.19 in April to approximately $2.46 by late August, despite no major architectural redesign. Cloudflare discovered that output tokens represented only 23% of spending, while cache writes accounted for 43% and cache reads another 34%. Although the overall cache-hit rate had reached 97%, the first request in a session reused a cached prefix only 14% to 26% of the time. Repository-specific information was preventing otherwise identical instructions and tool definitions from sharing cache entries. Fixing the prefix structure, removing inaccessible tools, eliminating redundant model turns, skipping unnecessary repeat reviews, and stopping canceled jobs accounted for approximately 40% of the reported savings. Model and reasoning-effort changes produced the remainder. Cloudflare also evaluated review quality, reporting that developers acted on 59% of serious code-quality findings after optimization, compared with 34% previously. These are internal measurements rather than independently verified defect-detection benchmarks. The important finding is that a rising AI bill can reflect inefficient execution and context handling rather than simply higher model prices.

2DeepSeek opens DeepInfer, exposing the inference machinery behind its commercial API

DeepSeek released DeepInfer on October 20, with substantial developer attention continuing October 21. The Apache 2.0 project exposes core inference technology used in DeepSeek's production services, initially supporting DeepSeek V4.1 Flash. Its capabilities include optimized attention and matrix-multiplication kernels, mixture-of-experts parallelism, asynchronous batch scheduling, speculative decoding through DSpark, confidence-scheduled verification, deterministic sampling, and structured output generation. Supported accelerators include NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. DeepSeek also released a companion project, deepseek-recipe, that reproduces important request-encoding and response-decoding behavior from its hosted API. The release is significant because it makes production-oriented inference optimizations available to developers pursuing self-hosted alternatives to commercial token billing. Important limitations remain: the published DeepInfer implementation supports offline inference and does not include the complete KV-cache management system, prefill/decode disaggregation infrastructure, or replica load balancing needed for a production serving service. Its published performance results are DeepSeek's own and have not been independently reproduced across comparable deployments. The release lowers the barrier to studying efficient inference, but does not make production deployment or its operating costs trivial.

3DDN and Nebul report nearly tenfold faster first-token delivery through reusable context

DDN announced October 21 that European AI cloud provider Nebul is using DDN Infinia to store and reuse key-value cache state across its NVIDIA-powered infrastructure. The deployment targets sovereign AI inference workloads where data residency, performance, and infrastructure utilization must be managed together. According to testing reported by DDN, its system delivered nearly ten times faster time to first token than Nebul's existing CPU-memory cache configuration, up to five times faster token responses than evaluated alternatives, and 2.5 times more input tokens per second from the same GPUs. The mechanism is straightforward: retaining and efficiently retrieving previously computed context can reduce repeated prompt processing and free accelerator capacity for additional requests. These figures are vendor-reported performance comparisons, not independently verified reductions in customer invoices. The benchmarks also compare particular storage and caching configurations rather than establishing universal gains across inference workloads. Nevertheless, the announcement illustrates how the economics of long-context agents increasingly depend on memory architecture and cache reuse rather than model execution alone.

4Nutanix and Supermicro target the cost per token of private AI

Nutanix and Supermicro announced October 21 that they plan to develop a jointly engineered infrastructure system for private AI inference. The proposed solution combines Supermicro servers containing eight NVIDIA Blackwell Ultra GPUs per node with Nutanix infrastructure software, NVIDIA Spectrum-X networking, integrated storage, and agent-management capabilities. The companies position the system as a way to reduce deployment complexity, improve GPU utilization, support open-weight models, and give enterprises more control over inference economics. Planned capabilities include model-as-a-service deployment, a secure agent gateway, integrated context caching, and centralized infrastructure management. The announcement is a product-development commitment, not evidence that the completed system already delivers a specific reduction in token costs. No independently verified cost-per-token benchmark accompanies the release. Its significance lies in the expanding market for private inference systems that compete on total operating economics, including hardware utilization, storage, networking, deployment overhead, and governance. For enterprises with sustained workloads, the comparison increasingly involves the total cost of owned or dedicated infrastructure versus consumption-based API pricing.

D-Skill: Distilling Teacher Knowledge through Agent Skills

Submitted October 20 and appearing in the October 22 research listings, D-Skill investigates transferring the behavioral advantages of expensive language-model agents to smaller models without changing model weights. The researchers analyze paired execution traces from teacher and student agents, identify meaningful behavioral differences, and consolidate the resulting lessons into natural-language skill documents. These skills are then supplied to the smaller model during inference. Across SpreadsheetBench-Verified, ALFWorld, and chess-puzzle evaluations, the authors report closing 75% to 95% of the performance gap between teacher and student models. Under the same model family, student inference costs were 5.7 to 8.3 times lower per task than those of the teacher. The method operates offline after collecting the paired trajectories and does not require further environment interaction during skill construction.

Why it matters: This is a concrete alternative to paying frontier-model prices for every execution. Expensive reasoning can be used to discover reusable procedures that improve cheaper agents. The economics depend on how often the skills are reused, how well they generalize, and whether the cost of creating and maintaining them is recovered through subsequent execution savings.

HGM-Dual: Lineage-Aware Guided Agent Harness Optimization for Long-Horizon Planning

Submitted October 21, HGM-Dual examines the cost of improving an agent's prompts, tools, and workflows through repeated experimentation. Conventional recursive improvement methods can waste substantial evaluation budgets exploring changes that do not address the agent's actual weaknesses. The proposed framework maintains structured records of previous modifications, including which behaviors improved, which regressed, and which changes proved ineffective. It then generates both general improvements and specialized candidates targeting identified planning failures. Across the evaluated planning tasks, HGM-Dual improved case accuracy by up to 14.45% relative to the strongest baseline and matched the final performance of existing Huxley-Godel Machine variants using two to four times less evaluation budget. These are benchmark-specific results, not measured reductions in production API bills.

Why it matters: Agent improvement itself consumes tokens and evaluation resources. A system that remembers which modifications failed can avoid paying repeatedly to rediscover the same lessons. The paper extends token-efficiency thinking beyond runtime execution into the development and optimization of agent workflows.

UlaCache: Unified Logical Addressing for Sparse LLM Serving

Submitted October 21, UlaCache introduces a memory-management system for sparse-attention inference. Sparse attention reduces computation by selecting relevant historical tokens, but the corresponding KV-cache entries must remain available for subsequent decoding. UlaCache separates logical cache references from physical placement across GPU memory and optional CPU storage. Its design retains frequently needed entries locally and transfers missing entries when required. A parallel eviction mechanism manages shared GPU cache capacity. Under matched CPU and GPU memory budgets with speculative decoding enabled, the researchers report peak decoding throughput improvements of 1.16 times for data-parallel attention and 1.70 times for tensor-parallel attention on NVIDIA H200 hardware. On B300 hardware, the corresponding improvements reached 1.73 and 2.51 times. These are experimental throughput results rather than direct measurements of commercial inference prices.

Why it matters: Reducing attention computation is only useful if cache management does not erase the savings. UlaCache demonstrates that efficient retrieval and placement of intermediate model state can materially improve token throughput. The principle applies broadly to inference systems where memory movement competes with computation for time and resources.

HAWK: Heterogeneity-Aware KV Cache Transfer for Efficient GPU-NPU Phase-Disaggregated LLM Inference

Submitted October 21, HAWK investigates using different accelerator types for the prefill and decoding stages of language-model inference. The researchers observe that GPUs are particularly effective at compute-intensive prefill, while NPUs can offer advantages during memory-bound decoding. Combining the devices requires transferring KV-cache state between architectures with different memory layouts and communication mechanisms. HAWK uses remote direct memory access and layout-aware destination buffers to transfer cache state without requiring additional data rearrangement. In experiments using commercially available GPUs and NPUs, the heterogeneous system achieved performance within 7.88% of a GPU-only configuration under equivalent service-level constraints while delivering up to 1.26 times greater power efficiency and 1.72 times greater cost efficiency. The results apply to the evaluated hardware and workloads.

Why it matters: The most economical accelerator for processing a prompt may not be the most economical accelerator for generating its answer. Separating inference phases across different hardware creates opportunities to reduce operating costs, provided communication overhead remains controlled. HAWK demonstrates a practical approach to managing that tradeoff.

RISK-KV: Quality-Aware Selective KV Cache Reconstruction for Distributed LLM Inference

Submitted October 21, RISK-KV investigates reducing the amount of historical KV-cache state that must be retained or transferred when inference requests migrate between servers or resume after interruption. Instead of preserving every cache entry, the method reconstructs selected missing layers using information from neighboring layers. A quality-aware selection process evaluates the effect of reconstruction on model predictions before deciding which layers can be omitted. Across three evaluated language models, representative configurations omitted 18.2% to 33.3% of historical KV-cache state while keeping perplexity changes below 0.5%. In a local recovery experiment using 512 tokens of history, RISK-KV achieved speedups ranging from 1.51 to 3.31 times compared with replaying the complete token history. These results concern the evaluated recovery conditions rather than general-purpose inference throughput.

Why it matters: Keeping every previously computed state consumes memory and transfer bandwidth, while recomputing everything wastes processing time. Selective reconstruction offers a middle ground that trades limited computation for reduced storage and communication. The paper illustrates how inference efficiency increasingly depends on deciding what information is worth preserving exactly.