LLM routers face churn as projects deprecate, refocus

Manifest announced the deprecation of its LLM router, joining a wave of projects that are either folding or pivoting. The move signals that the early hype around universal model dispatchers is giving way to more focused implementations.

The Manifest blog post titled Everyone is building LLM routers, we deprecated ours admits that the team will stop maintaining the service.¹ No new code will be merged, and existing users are urged to migrate to other solutions.¹ The announcement arrived alongside a lively Hacker News discussion that logged 127 points and dozens of comments, underscoring how many developers have already experimented with routing layers.

Open‑source routers gain traction

RouteLLM provides a lightweight OpenAI‑compatible server that swaps between a strong model and a weaker, cheaper one.² The repository shows a default pairing of gpt-4-1106-preview with anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1.² A cost threshold of 0.11593 routes roughly half the queries to GPT‑4, according to the project’s calibration guide.³²⁴ The router relies on LiteLLM for provider abstraction and still requires an OpenAI API key to generate embeddings for its ranking strategies.³⁵⁶⁷

The framework also supports local models and custom routing strategies via a simple CLI flag syntax.² Users can launch the server, point any OpenAI client at it, and let the router decide which backend to call.² The documentation emphasizes that the mf router delivers strong performance with low overhead, a claim backed by internal benchmarks but not independently verified.²

Any‑LLM takes a different angle. It offers a unified SDK that maps provider‑specific model strings onto a single interface. By swapping a provider name in the import statement, developers can switch between OpenAI, Anthropic, Azure, Mistral, Ollama, and others without rewriting request code. The project pulls official SDKs when available and falls back to a framework‑agnostic shim otherwise. It ships under the Apache 2.0 license and includes optional modules for API‑key management, usage analytics, and multi‑tenant support via the Mozilla‑AI otari stack.

Both projects aim to hide the quirks of each vendor’s API. Neither claims to eliminate the underlying cost differences; instead, they give engineers the plumbing needed to build their own routing logic.

Intent‑tuned and agent‑first routers

A smaller research effort introduced an intent‑tuned router that classifies queries before picking a model. The system trains a BAAI/bge‑base‑en‑v1.5 encoder on the GAIR‑NLP/Auto‑J scenario classification dataset, then merges the fine‑tuned weights at a 3:2 ratio. For each incoming prompt, the router generates intent categories, runs pairwise evaluations with GPT‑4‑turbo (April 2024), and computes Bradley‑Terry scores to rank candidate models. Scores are normalized to a 0‑1 range for downstream weighting.

The approach promises finer granularity than a simple cost‑threshold but adds latency from intent inference and pairwise scoring. The codebase does not include a production‑grade server; it remains a proof‑of‑concept for research labs that need per‑intent quality guarantees.

ClawRouter targets autonomous AI agents rather than human developers.³ Its README claims up to 87% inference‑cost reduction and a baseline 78% savings on typical workloads.³ The router evaluates each request across fifteen dimensions and selects the cheapest capable model in under 1 ms, all on the local machine. It supports 66 models from providers such as OpenAI, Anthropic, Google, xAI, and DeepSeek, and it integrates with the x402 protocol for USDC micropayments.³ The project is MIT‑licensed and ships with a one‑click install script that configures wallet authentication, model allowlists, and a local proxy on port 8402.

ClawRouter’s agent‑first design removes the need for API keys or credit‑card billing.³ Free tiers ship with eight open models, and developers can add paid providers by funding a wallet address with a few dollars of USDC.³ The router advertises zero‑knowledge routing: no request leaves the host before the local decision is made.³

Why the churn matters

The deprecation of Manifest’s router illustrates a broader market correction.¹ Early routers promised a one‑size‑fits‑all gateway, but cost structures, latency requirements, and security models differ sharply across deployments. Projects that survive tend to specialize: RouteLLM focuses on cheap‑versus‑expensive pairings, Any‑LLM on provider‑agnostic SDKs, intent‑tuned routers on query classification, and ClawRouter on autonomous agents.

Cost remains the primary driver. A GPT‑4 request can cost several cents, while Mixtral‑8x7B runs for a fraction of a cent on cloud GPUs. Routing half the traffic to the cheaper model, as RouteLLM demonstrates, can halve a bill without obvious quality loss for many workloads.³² However, the need for an OpenAI key to generate embeddings shows that even hybrid setups still depend on a dominant provider for certain primitives.³⁵⁶

Latency is another pressure point. ClawRouter’s claim of sub‑millisecond routing hinges on local decision logic and pre‑loaded model metadata.³ Cloud‑based routers that proxy through external services inevitably add network hops, which can be unacceptable for real‑time agents. The trade‑off is that local routers must keep model binaries up to date, a maintenance burden that smaller teams may avoid by relying on a hosted service.

Security and compliance also shape the landscape. Agent‑first routers that accept wallet signatures avoid storing API keys, reducing attack surface.³ Conversely, unified SDKs like Any‑LLM still require developers to embed provider credentials, which can be a liability in multi‑tenant environments.

Forward‑looking considerations

The next wave of routing tools will likely blend these concerns. Expect tighter integration with cost‑monitoring platforms, more granular quality‑of‑service controls, and open standards for intent classification. Keep an eye on the release cadence of RouteLLM’s mf router, any updates to ClawRouter’s x402 payment flow, and the emergence of benchmark suites that compare routing strategies at scale. The decisions teams make about routing will directly affect cloud spend, latency budgets, and the security posture of AI‑driven products.

What to watch: the first quarter of 2027 will see several cloud providers announce native routing primitives in their APIs. Monitoring those announcements will reveal whether third‑party routers can maintain a niche or will be subsumed by platform‑level features.