Token Cost Radar

1What if you stopped paying for tokens and paid for the electricity instead?

A September 22 San Francisco event on inference efficiency is presenting a serving API priced at a flat $10 per kilowatt-hour rather than per token. The organizers say their self-optimizing stack tunes batching, tensor parallelism, prefix caching, and expert routing and currently runs about 88% below comparable token rates. The vendor claims eight GPUs can deliver throughput normally provisioned across 16. The figures are vendor-reported, but energy-based billing is a notable attempt to replace token pricing with a unit closer to the underlying resource.

2DeepSeek inference gets a $0.20-per-hour experiment

A September 21 DeepSeek community post describes a reserved V4.1 Flash inference lane priced at $0.20 per hour with two concurrent requests, a 262K context window, and no per-token or cache charges. The operator says earlier stress tests processed billions of tokens while maintaining roughly 98% to 99% prefix-cache hit rates. This is a small third-party experiment rather than DeepSeek's own pricing model, but it is another signal that high-reuse workloads may be economically awkward to meter token by token.

3Grok 4.7 adds a long-context tax above 200K tokens

Current September 22 pricing verification for Grok 4.7 lists standard API rates of $2 per million input tokens, $0.50 per million cached input tokens, and $6 per million output tokens. Requests exceeding 200,000 prompt tokens are charged at double the standard token rates, with a maximum context window of 500,000 tokens. The tier makes context size an explicit cost-control variable rather than merely a technical limit.

4Frontier token prices are now 84% below the GPT-4-era baseline

BenchLM's pricing index, verified September 21, puts its frontier token-price index at 16 against a March 2023 baseline of 100, an 84% decline. The median flagship model in its 21-model active basket costs $6 per million blended tokens. The decline reinforces the inference paradox: the raw unit of intelligence keeps getting cheaper even as agents, long context, tools, and repeated inference make the quantity consumed much larger.

Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving

This serving study observes that 80% to 95% of production requests are short even though inference instances are commonly provisioned for worst-case context length. Routing requests by estimated total token budget into short-context and long-context pools reduced GPU-hours by 31% to 42% on Azure and LMSYS traces serving Llama-3-70B on A100s, with projected fleet-scale savings of $2.86 million annually.

Why it matters: This is token budgeting translated into infrastructure economics. Two requests can use the same model yet impose very different memory and concurrency costs, so expected token footprint can determine where a request should run.

Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference

A related study estimates each request's total token demand and dispatches it to differently configured vLLM pools without requiring a tokenizer. Across Azure and LMSYS traces, the authors report 17% to 39% fewer required GPU instances, corresponding to roughly $1.2 million to $2 million in annual savings at 1,000 requests per second.

Why it matters: The work exposes a useful distinction between token price and token production cost. Better placement can make the same model and the same nominal tokens cheaper before any prompt optimization occurs.

ContextPipe: Database-Inspired Context Assembly for Long-Horizon Agents

ContextPipe treats context construction as query planning under a hard context budget and byte-sensitive prompt cache. In its preliminary SWE-bench Pro evaluation, it reduced total token volume by 31%, LLM calls by 23%, and response time by 9% versus append-only context construction, although its KV-cache hit ratio declined.

Why it matters: Tokenminimizing can collide with cache economics. A smaller context is not automatically cheaper when changing the prefix forces the inference system to recompute work it could otherwise reuse.

Drift-Aware LLM Routing with Sparse Contexts and Shared Budgets

This recent routing study models model quality, monetary cost, latency, memory, compute, and workload mix as quantities that can drift over time. Its policy estimates reward and resource consumption from a rolling audit window, updates resource shadow prices online, and checks a shared budget before committing each request.

Why it matters: Modelmaxxing cannot remain a static cheapest-model table when prices and workloads move. The router itself increasingly has to behave like a small FinOps system.