Meet Switchyard: A Rust Proxy and Library That Routes and Translates LLM Traffic Across OpenAI and Anthropic APIs

NVIDIA has released Switchyard, a Rust-based proxy and library designed to route and translate traffic between OpenAI and Anthropic APIs. The tool allows teams to run coding agents without rewriting their code to match specific backend requirements.

The solution addresses a common friction point. Developers often find that Claude Code relies on the Anthropic Messages API while Codex CLI speaks OpenAI. Meanwhile, the model a team actually wants to serve usually sits behind vLLM, NVIDIA NIM, or Ollama. Since rewriting every agent is not an option, the translation layer must live elsewhere.

Switchyard handles this by decoding inbound requests into provider-neutral Rust types, running a routing algorithm to select a backend, and re-encoding the request in that backend’s specific wire format. It then translates the response, including streaming events, back into the shape the client expects.

Deployment status

The binary installs from crates.io and the launcher from PyPI. Self-hosting is possible anywhere, but NVIDIA labels Switchyard as pre-alpha and experimental. The company warns it is not for production use and expects the API and algorithms to change significantly before version 1.0.

How it works

The server accepts three inbound formats: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Any of these three can address any route, and each configured LLM client selects one upstream format of its own. This decoupling means the agent’s API and the backend’s API no longer have to match.

Running the tool

There are three paths for deployment. The launcher path targets coding agents. Users install the published tool with uv tool install --python 3.12 "nemo-switchyard[cli]", then run switchyard launch claude, switchyard launch codex, or switchyard launch openclaw against a packaged deployment or a custom TOML file.

The server path installs the standalone proxy with cargo install --locked switchyard-server, validates a config with --dry-run, and serves on a host and port chosen by the user.

The library path uses switchyard-libsy. This embeds the routing algorithms in a Rust application without owning an HTTP stack. The algorithm decides which target to use and hands every model call back to the caller.

Routing logic

A route consists of one client-visible model ID plus the algorithm behind it. The server supports four methods:

• passthrough sends every request to one target.
• random splits traffic across targets using optional relative weights, with an optional seed that reproduces the selection sequence. This serves as the path for A/B testing and cost experiments.
• llm_classifier calls a classifier target for a capability verdict, then routes to a weak or strong target. base_threshold is required; min_confidence, capability_elevated_floor, and session_affinity tune it, and anything the judge cannot decide falls through to the strong target. Setting mode = "escalation" runs every turn on the weak tier first and lets a judge decide whether to rerun it on the strong tier.
• stage_router scores tool-result and agent-progress signals from recent turns to pick a capable or efficient target, avoiding an extra classifier call on most turns.

Strong, weak, capable, and efficient are roles inside a route, not fixed properties of a model. The same upstream model can serve different roles in different routes.

Observability

GET /metrics returns Prometheus text from the server’s process-wide OpenTelemetry provider. The families cover requests, errors, model-call latency, full-turn latency, prompt, completion, cached, cache-creation, and reasoning tokens, and upstream HTTP attempts by outcome and code. A tier label carries strong or weak for distinguishable classifier decisions, and classifier calls are excluded from those families.

The more interesting metric is switchyard_routing_overhead_ms, which reports the algorithm’s run time minus the call that served the request. Classifier calls are not subtracted, so an LLM-classifier route reports its classification time here while passthrough and random report the sub-millisecond cost of picking a target. Buckets start at 0.1 ms. Separately, --routing-log-file appends a JSON record per completed response, and GET /v1/routing/session-stats returns per-session call and token totals from that log.

Configuration

A TOML deployment has three layers. llm_clients define base URL, wire format, credential environment variable, and retry policy. targets bind one upstream model ID to a client. routes expose one client-visible model ID and its algorithm. Secrets never sit in the file, since api_key_env only names an environment variable. max_retries defaults to 2 and applies to transport failures, timeouts, HTTP 408/429, and 5xx responses.

What it means

For developers building agents, the tool removes the need to maintain multiple versions of code to talk to different models. It allows a single agent interface to talk to whichever backend is available, translating the differences in the OpenAI and Anthropic protocols automatically.