Token Cost Radar

1Can a model that uses fewer tokens beat one with cheaper tokens?

Reflection AI released the weights for Beam on October 16 under the Apache 2.0 license. The model contains 501 billion total parameters, with 23 billion active during inference, and targets coding, reasoning, and long-running agent workloads. Reflection emphasizes reasoning efficiency, supported by reinforcement learning that penalizes unnecessary token generation while rewarding successful task completion. Its release includes BF16, FP8, and NVFP4 weights, a speculative-decoding draft model, and optimized serving integrations. Reflection also reports approximately 1.3 billion sandbox executions across training, evaluation, and related experiments. In separate reporting, VentureBeat describes Reflection's claim that Beam can achieve results comparable to GLM-5.2 while using substantially fewer generated tokens. The company has not established a universal commercial cost-per-task advantage, and independent comparisons remain necessary. The release illustrates an increasingly important distinction: a model can reduce total inference requirements by solving problems with fewer generated tokens, even when its individual tokens are not the cheapest available.

2AMD inference optimization delivers 87.4% more tokens per modeled dollar

The vLLM project published October 16 performance results for serving Moonshot AI's Kimi K3 on AMD Instinct MI355X accelerators. On the AgentX coding-agent workload, an eight-GPU AMD configuration delivered approximately 7,567 total tokens per second per GPU at a 50-token-per-second-per-user P90 interactivity target. The comparison NVIDIA HGX B300 configuration delivered approximately 6,083 tokens per second per GPU, giving AMD a 24.4% throughput advantage. Using SemiAnalysis's modeled ownership costs of $1.50 and $2.26 per GPU-hour respectively, the AMD configuration delivered 18.16 million tokens per modeled dollar against 9.69 million, an 87.4% advantage. The optimizations include eliminating duplicated mixture-of-experts computation, reducing memory transfers, improving attention kernels, and increasing cache efficiency. These results apply to the specified benchmark and serving configurations. The dollar comparison uses modeled total ownership costs rather than observed commercial rental prices. Nevertheless, it demonstrates that software optimization and hardware selection can substantially change inference economics without changing the underlying model.

3AWS CFO challenges the assumption that fewer tokens always mean better economics

Axios reported October 16 that Amazon Web Services CFO John Felton considers the company's AI infrastructure spending sustainable and says its investments are producing meaningful returns. Felton cited an approximately $500 billion order backlog, an AI revenue run rate exceeding $25 billion, and server investments that AWS says are achieving payback periods under three years. More significantly for token economics, Felton said an internal examination of AWS token consumption showed a roughly linear relationship between usage and productivity. He argues that AI should be evaluated as a means of increasing productive capacity and enabling new revenue-generating activities rather than exclusively reducing expenses. These are AWS management's observations, not independently published measurements establishing that additional tokens always produce additional value. The comments provide an important counterpoint to aggressive token reduction: consumption can be economically justified when the resulting productivity exceeds its cost.

4Replit's CEO says cost and performance influenced its move toward OpenAI

TechCrunch reported October 16 that Replit CEO Amjad Masad said the company's decision to favor OpenAI over Anthropic reflected a combination of cost, partnership considerations, and model performance. Masad cited internal evaluations indicating that OpenAI's Sol and Astra models performed better on Replit's application-building workloads. Replit announced an OpenAI partnership in August. However, TechCrunch also reported that a person familiar with the relationship disputed the suggestion that Replit had completely stopped working with Anthropic, and evidence indicated Anthropic models had remained in use during September. The precise status of the commercial relationship therefore remains contested. No comparative token-cost figures were disclosed. The episode nevertheless illustrates how AI platform buyers increasingly evaluate model providers against actual workload performance and commercial terms rather than maintaining permanent allegiance to a single model family.

When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

Submitted October 15 and appearing in the latest research listings, this paper examines a counterintuitive failure of remote key-value caching. In a disaggregated GLM-5.3 deployment using eight NVIDIA B300 accelerators, enabling a remote cache increased the 99th-percentile time to first token for 16,000-token requests from 3.96 seconds to 17.3 seconds. In an isolated production-trace replay, retrieving cached state took 0.88 seconds compared with 0.31 seconds to recompute the prompt. Synchronous cache loading eventually produced a p99 latency of 83.32 seconds while GPUs remained only 7% busy. The researcher identified bandwidth and fixed per-hit overhead as the principal problems. After sharding the cache, improving transfer bandwidth, and implementing asynchronous loading, p99 latency on the tested hit-heavy workload fell to 0.67 seconds. Prefill throughput improved by 14%, or 34% when combined with an upstream kernel correction.

Why it matters: A cache hit is not automatically an efficiency improvement. The relevant comparison is the total cost and latency of retrieval versus recomputation, including storage, network transfers, synchronization, and queueing. This research provides concrete evidence that poorly implemented caching can make inference substantially slower even when almost all prompt tokens are served from cached state.

HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding

Submitted October 15, HyCo investigates reducing cloud inference computation by performing part of speculative decoding on a mobile device. The method combines a server model's native multi-token prediction mechanism with a smaller draft model running on the device's neural processing unit. The server transmits selected intermediate state, allowing the device to generate candidate continuations that the server verifies. In experiments using a Galaxy S26 Ultra over LTE and 5G, the researchers report reducing server GPU time per output token by 38.6% on Spec-Bench and 29.2% on a 32,000-token summarization workload relative to server-side multi-token-prediction speculation. A stricter configuration preserving the original sampling behavior achieved smaller savings. The experiments also examine output-quality differences introduced by relaxed verification.

Why it matters: Inference cost optimization increasingly includes deciding where computation should occur. Moving suitable drafting work onto client hardware can reduce server GPU demand, but network transfers, device capabilities, verification overhead, and output quality must be included in the calculation. The reported GPU-time reductions do not automatically translate into equivalent reductions in commercial API bills.

FastVLA: Task-Conditioned and Temporally Adaptive Visual Token Compression for Efficient Vision-Language-Action Models

Submitted October 15, FastVLA introduces adaptive visual-token compression for models controlling robotic actions. Instead of processing every visual token at every control step, the framework evaluates which visual information matters for the current task and adjusts the compression rate over time. The researchers tested the training-free method across several vision-language-action models and robotic simulation benchmarks, followed by experiments on a physical Franka Research 3 robot. In simulation, FastVLA reduced language-model backbone computation by 35% to 59% and action latency by 9% to 32% while maintaining comparable task success. On the physical robot, end-to-end latency fell approximately 20% at comparable success rates. These results concern specialized multimodal inference rather than ordinary text-generation API pricing.

Why it matters: The principle generalizes beyond robotics: not every available input token is equally relevant to the current decision. Adaptive token budgets can preserve task-critical information while reducing computation devoted to less useful material. The economic benefit depends on whether the selection mechanism costs less than the processing it eliminates.

Improving Environment-Agnostic Coding Agents: The Role of Scaffolding

Submitted October 15, this paper examines whether stronger general-purpose coding agents can replace carefully designed procedural scaffolding. The researcher evaluated a Codex-based agent across ARC-AGI-3, AGI Maze, and Terminal-Bench using largely unchanged agent implementations. Small changes to instructions and control flow improved performance on some tasks, while moving reset decisions into a scripted controller recovered part of the advantage held by a specialized reference agent. Additional tutorial experience did not consistently improve results and sometimes reduced performance. Importantly, the paper reports that some performance improvements came with higher token-based costs. The experiments do not isolate whether the gains resulted from better information use or simply additional inference.

Why it matters: Better agent scores do not necessarily mean better token efficiency. Deterministic control procedures can sometimes replace repeated model reasoning, but additional scaffolding and continuation loops can also increase consumption. Evaluations should report both successful task completion and the token cost required to achieve it.