Token Cost Radar

1What happens to token economics when the same hardware gets 38% faster?

Fresh coverage of MLPerf Inference v6.1 reports that NVIDIA's Vera Rubin NVL72 delivered up to 3.7 times the throughput of GB300 NVL72 on the Qwen3-VL benchmark and up to 2.5 times the throughput on DeepSeek-R1. The same round also showed AMD improving server-mode throughput on the gpt-oss 120B workload by 38% on unchanged MI355X hardware through ROCm software updates alone. The results underline a basic inference-economics point: cost per useful token can fall even when the model and accelerator bill of materials do not.

2DeepSeek scales agent infrastructure to more than 380,000 concurrent sandboxes

September 26 coverage of DeepSeek's DSec platform says the system coordinates more than 380,000 concurrent sandboxes for large-scale agent training and evaluation. The architecture combines containers, microVMs, and full virtual machines with shared storage, memory reclamation, and workload-aware scheduling. The scale illustrates why agent economics increasingly includes orchestration and infrastructure utilization alongside model tokens.

3OpenAI prepares a faster inference tier

TestingCatalog reports September 26 that OpenAI is preparing to broaden access to its Ultrafast API mode through the Playground. The report describes the mode as a higher-speed inference option and says broader availability may follow OpenAI DevDay. Pricing details were not yet reported, so the economic significance remains unresolved, but differentiated latency tiers add another dimension to model routing beyond simple cost per token.

DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression

DeepSeek's recent V4.1-Flash work targets an increasingly important property of long-horizon agents: workloads dominated by repeated input and growing context. The architecture focuses on reducing KV-cache requirements, attacking the memory cost attached to retaining long conversational and agentic state rather than merely reducing visible prompt length.

Why it matters: Inference yield depends heavily on how cheaply prior context can remain reusable. As agent workloads become input-heavy, KV-cache efficiency becomes a cost-control mechanism even when the API still presents the result as token pricing.

Cost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guarantees

SLARouter learns an online cost-aware model-routing policy from sparse production feedback while enforcing a user-satisfaction constraint. Across its benchmark evaluations, the authors report satisfying quality constraints while reducing operating cost by as much as 2.2 times compared with existing routing baselines, without per-benchmark tuning.

Why it matters: Modelmaxxing becomes much more useful when quality is a constraint rather than an aspiration. The cheapest model is only economical if users accept its answer.

Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving

This serving study observes that 80% to 95% of production requests are short even when inference instances are provisioned for worst-case context length. Routing requests by estimated total token budget into short-context and long-context pools reduced GPU-hours by 31% to 42% on Azure and LMSYS traces serving Llama-3-70B on A100 GPUs, with projected annual fleet savings of $2.86 million.

Why it matters: Token budgeting can control infrastructure allocation before a model generates anything. Expected context size affects KV-cache capacity and concurrency, so the token budget can determine where the request should run as well as which model should answer it.

Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference

A related serving study estimates total token demand before dispatch and separates short and long workloads into differently configured vLLM pools. On Azure and LMSYS traces, the authors report 17% to 39% fewer required GPU instances, corresponding to roughly $1.2 million to $2 million in annual savings at 1,000 requests per second.

Why it matters: The work separates token price from token production cost. Better workload placement can make the same nominal model output cheaper without changing the prompt, model, or provider rate card.