long-form

The Hidden Cost of LLM Routing: Why Switching Models Costs More

Nhu Hoang replayed one 20-turn agent session under three routing policies on a vLLM server with prefix caching, then priced the token flows at Claude API rates. Switching models every turn saved 13%; the price sheet promised 40%.

Published 2026-10-20Source: Towards Data Science
Generated Tokenmaxxing editorial thumbnail for The Hidden Cost of LLM Routing: Why Switching Models Costs More

Why it matters

A model taking over mid-session has cached none of the conversation and re-reads all of it at full price. A Microsoft Azure Research study of 13.5 million GitHub Copilot sessions, cited by Hoang, puts cache reuse at 90% within a turn and 8% after a switch.

Tokenmaxxing read

Router pitches quote the price gap between models and leave out the cost of the handoff. Hoang's test: uncached tokens times the input rate, checked before every switch. Two planned phase-boundary switches saved 29%; nineteen per-turn switches saved 13%, not the sheet's 40%.

Source takeaway

Read the limits. The cache counts come from a 0.9B model replaying a script, and only input cost is priced. The live quality check is thin: Haiku 4.5 passed 29 of 30 tasks alone, and the first escalation to Opus 5 cost 2.1x what Opus alone spent on that task.

Topic links

Related projects

Tools that match this angle

#1Direct
Routing

LiteLLM

BerriAI/litellm

An OpenAI-compatible gateway and SDK for calling many model providers with budgets, logging, load balancing, guardrails, and cost tracking.

63.6K13.4KSource-available
gatewaycost-trackingrouting
#2Direct
Observability

Langfuse

langfuse/langfuse

Open-source LLM engineering platform for observability, traces, metrics, evals, prompt management, datasets, and playground workflows.

37.7K4.3KSource-available
tracesevalscosts
#4In spirit
Agents

LangGraph

langchain-ai/langgraph

A framework for building resilient stateful agents with explicit graphs, persistence, human-in-the-loop flows, and controllable execution.

45.2K7.8KMIT
agentsstateworkflows
Related feed

More source-linked context

Anthropic source artwork
newsA
news

Introducing Claude Fable 5.2 and Claude Mythos 5.2

Anthropic released Claude Fable 5.2 and Mythos 5.2 on Nov. 9, priced at $8 per million tokens in and $40 per million out, which it calls a lower price. An optional fast mode generates up to 2.5 times quicker at exactly double both rates.

tokenmaxxingpricingmodel-routing
Read note
Generated Tokenmaxxing editorial thumbnail for Enterprise AI budgets break at the handoff to production
long-formEC
long-form

Enterprise AI budgets break at the handoff to production

Express Computer interviews New Relic India's Ganesh Narasimhadevara on why AI bills keep climbing while the blended cost per million tokens has fallen over a year, from $18.40 in Q1 2025 down to $6.07 in Q1 2026.

tokenmaxxingai-spendcost-governance
Read note
Stripe source artwork
newsS
news

Stripe agrees to acquire OpenRouter to help businesses optimize token routing and usage

Stripe said on Aug. 19 that it has agreed to buy OpenRouter, the gateway that sits in front of 400+ models from over 80 providers. NVIDIA, Zoom and Lovable are named as users. The release gives no price.

tokenmaxxingmodel-routermodel-routing
Read note