Token Cost Radar

1Can AWS shorten your prompts without sacrificing answer quality?

Amazon Web Services announced October 20 that Amazon Bedrock Advanced Prompt Optimization now supports prompt shortening. Previously, the optimizer pursued higher evaluation scores without constraining prompt length, sometimes producing longer and more expensive prompts. The new optional setting searches for a shorter prompt that scores at least as well as the original against the customer's evaluation criteria. If no candidate satisfies both conditions, Bedrock returns the original prompt. Results include original and optimized token counts and a compression ratio. The feature supports built-in evaluations, custom scoring criteria, LLM judges, and AWS Lambda evaluators. It operates on prompt templates rather than generated responses and does not guarantee that every prompt can be shortened. AWS recommends testing optimized prompts against held-out data because evaluation results may not generalize. The development makes token reduction an explicit, quality-constrained optimization objective rather than an informal prompt-engineering exercise.

2Flexera expands AI cost controls from tokens to infrastructure

Flexera announced October 20 that its AI Cost Management platform is expanding to cover AI cloud infrastructure and data platforms alongside model APIs, applications, agents, and workflows. Its existing Token Cost Control offering identifies users and agents responsible for AI consumption and supports cost allocation and chargeback. The expansion adds autonomous workload and data-cloud optimization, with Snowflake automation planned before the end of 2026 and Databricks automation expected in early 2027. Flexera also announced additional ecosystem integrations and an AI cost-management maturity assessment. The company cites research indicating that only 31% of organizations have adequate visibility into AI software and that nearly three-quarters struggle to demonstrate AI business outcomes to finance leadership. Those figures are drawn from cited research rather than a new independent audit. The significant development is the consolidation of token metering, infrastructure costs, and financial accountability into a broader AI FinOps framework.

3AI gateways begin enforcing budgets before the bill arrives

The Elec's October 21 AI FinOps coverage describes how South Korean infrastructure providers are introducing centralized token accounting and real-time spending controls. Lablup's Continuum Router records model identity, input and output tokens, and latency, while Continuum Hub aggregates usage by organization, project, team, and API key. According to Lablup, administrators can establish monthly token and spending limits, block individual keys that exceed their allocations, and assign separate budgets to models. Particularly notable is a policy that redirects requests to a designated cheaper model when the preferred model's budget is exhausted rather than automatically rejecting the work. The platform also supports departmental showback and chargeback. These are provider-described capabilities, not independently measured reductions in customer spending. The architectural shift is important: cost governance moves from retrospective dashboards into the request path, where spending decisions can be enforced before additional inference occurs.

4Anthropic's infrastructure spending approaches $9 billion in six months

Reuters reported October 20 that Anthropic spent nearly $9 billion on computing capacity and infrastructure during the first half of 2026, compared with approximately $2.3 billion during the same period in 2025. The figures come from a draft IPO prospectus reviewed by Reuters. The company also disclosed a first-half net loss of $80.7 billion, although approximately $72 billion represented a noncash accounting charge associated with convertible debt rather than operating expenditure. Anthropic has committed at least $518 billion over ten years to cloud services, computing, and infrastructure. Reuters separately reported that third-quarter revenue reached $16.8 billion, up 45% sequentially. These figures illustrate the scale of capital required to supply frontier inference, but they do not establish Anthropic's cost per generated token or the profitability of individual products. The economic question is whether growing inference revenue and improving hardware efficiency can support the industry's extraordinary infrastructure commitments.

5Mercury 3 challenges sequential inference with 1,068 tokens per second

Inception introduced Mercury 3 on October 19, positioning its diffusion-based language model as an alternative to conventional autoregressive generation. The company reports throughput of 1,068 tokens per second on NVIDIA GPUs, a one-million-token context window, and performance comparable to or better than selected speed-optimized frontier models. Published standard pricing is $0.20 per million input tokens and $0.70 per million output tokens, with a 50% introductory discount reducing those rates to $0.10 and $0.35. Cached input receives an additional 90% discount relative to the applicable input rate. Unlike conventional models that generate tokens sequentially, diffusion models can generate and refine multiple tokens in parallel. Inception reports substantial latency advantages in customer-support, tutoring, and browser-agent applications. Its speed and quality comparisons remain vendor-reported and require independent validation across representative workloads. The release nevertheless illustrates how architectural alternatives may change both inference latency and the economics of generated output.

The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token

Submitted October 20, this paper introduces an interpretable model for estimating GPU-board energy consumed per generated token. The researchers identify two principal factors: active bytes, representing model-weight and key-value cache memory demands, and realized decode time, representing the elapsed decoding time per output token. The formulation distinguishes model weights shared across batched requests from the private KV-cache histories associated with individual requests. Experiments cover three models using BF16 decoding on NVIDIA Hopper and Blackwell GPUs, including a two-GPU tensor-parallel configuration. With coefficients calibrated for each hardware and runtime configuration, the model's held-out condition-mean energy estimates differed from measurements by no more than 3.73%. The research concerns measured GPU energy rather than commercial API pricing or complete data-center operating costs.

Why it matters: The paper offers a concrete way to connect token production with physical resource consumption. Longer contexts increase private memory demand, while batching distributes shared model-weight costs across requests. Energy per token therefore depends on workload structure and serving configuration, not merely the model's parameter count. The framework could help evaluate infrastructure efficiency and the energy component of inference economics.

Adaptive Multi-Token Prediction: How Far Should We Go?

Submitted October 20, this paper investigates how speculative decoding systems should determine the number of future tokens to predict before verification. Longer speculative drafts can improve throughput when accepted, but rejected suffixes waste computation. The researchers introduce Adaptive Multi-Token Prediction, a lightweight controller that learns from verification feedback and selects the draft horizon expected to maximize accepted tokens per second. The method requires no additional model training and leaves the verifier unchanged. Across five draft-target model pairings evaluated on AIME 2025 and GSM8K, the system achieved decoding speeds up to 4.1 times standard decoding. It matched or exceeded fixed-horizon configurations in eight of ten tested settings, with additional throughput improvements reaching 10%. The results concern the evaluated models and tasks, not guaranteed savings across all inference workloads.

Why it matters: Speculative decoding illustrates why maximizing generated tokens is not equivalent to maximizing useful throughput. A longer draft can be more expensive when much of it is discarded. Adaptive prediction attempts to spend computation only where the probability of acceptance justifies it, improving the relationship between work performed and output committed.

Evolving Agent Interfaces

Submitted October 20, this paper examines whether agents can improve their own interaction mechanisms rather than relying entirely on interfaces designed for human users. The researchers introduce Agent S4, a system that studies agent execution traces and evolves alternative interfaces and usage procedures. The paper identifies inefficient patterns such as entering spreadsheet data cell by cell or printing thousands of source-code lines to locate a single function. On OSWorld V2, the evolved interface repository improved the binary task score of Qwen3.8-27B to 21.3%, approximately 3.8 times the score achieved by the same model with fixed interfaces. The researchers report that this exceeded the evaluated GPT-5.5 and Claude Opus 4.8 configurations while operating at approximately 4.3% and 1.5% of their respective costs. On ScienceBoard, success increased from 41.4% to 58.6%. These are benchmark-specific results, and the reported cost comparisons depend on the evaluated configurations.

Why it matters: The research challenges the assumption that better agent performance always requires a more capable or expensive language model. Improving the interface through which an agent performs work can increase success while reducing inefficient interactions. The economic opportunity is to eliminate unnecessary execution steps and information transfers rather than simply making each model call cheaper.

BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models

Submitted October 20 and accepted at NeurIPS 2026, BitMTP combines ultra-low-bit language-model execution with multi-token prediction. Low-bit representations reduce model storage and memory traffic, while multi-token prediction attempts to reduce the number of sequential decoding operations. The researchers found that simply combining the techniques does not reliably improve performance because additional computation and cache-management overhead can erase the expected savings. Their proposed system coordinates drafting, verification, accepted-token boundaries, and CUDA Graph execution. On NVIDIA H20 hardware, hybrid decoding achieved mean speedups of 2.2 times on BitNet b1.58-2B and 2.1 to 2.4 times on evaluated Bonsai models while maintaining comparable task quality. Additional experiments reported improvements on RTX 3090 and Jetson Orin hardware. The paper does not directly measure commercial inference prices.

Why it matters: Two individually efficient techniques do not necessarily produce an efficient combined system. BitMTP demonstrates that memory representation, decoding strategy, and execution scheduling must be optimized together. The findings are particularly relevant to organizations considering smaller, locally deployed, or specialized models as alternatives to expensive hosted inference.