Designing a Dual-Format LLM Router: A Low-Level Design Guide

Every team that touches two model vendors eventually builds the same glue: one proxy that speaks both OpenAI-style and Anthropic-style HTTP so clients can point at a single base URL. It looks like a 200-line shim until you hit streaming. Here is how I’d design it from scratch, and why each decision lands where it does.

1. Frame the invariant first

The system has exactly one job, repeated per request:

client_format  = detect(request)
upstream       = select(request.model, config)
upstream_body  = encode(request, upstream.format)
response       = forward(upstream, upstream_body)
client_body    = decode(response, client_format)

Everything else — auth, model listing, health — is supporting cast. Write that four-line contract at the top of your design doc; every module you add must serve it or be cut.

2. Choose an internal representation, not N² adapters

The tempting design is pairwise adapters: openai_to_anthropic(), anthropic_to_openai(), and later openai_to_gemini(). Four formats means twelve converters, each with its own bugs.

Prefer a canonical IR:

IRRequest  { model, system[], messages[{role, blocks[]}], tools[], sampling, stream }
IRResponse { text, tool_calls[], stop, usage }

Then you write decode_from_openai, encode_to_openai, decode_from_anthropic, encode_to_anthropic. Adding a format costs two functions instead of 2(N−1). The trade-off is fidelity: an IR normalizes away vendor-specific fields. Handle that with an extra: map passthrough bucket that survives untouched — otherwise you will silently drop a customer’s reasoning_effort and spend a week debugging it.

If you only ever support two formats and never plan to add a third, skip the IR and keep two explicit converters. Be honest about which world you’re in.

3. Data model

Provider   { id, base_url, api_key, protocol: OPENAI|ANTHROPIC, timeout }
RouteConfig{ providers: Map<id, Provider>, default_id, client_keys[], mount_prefix }

Keep config immutable after startup. Build it once, freeze it, hand the same object to every handler. Mutable shared config is how you get cross-request surprises when tests build two apps in one process.

Infer protocol from the provider name (claude* → Anthropic) only as a convenience, and let an explicit field win. Name-based inference is fine as a default and terrible as a requirement.

4. Request pipeline

async handle(req):
    reject_if_unauthorized(req, cfg.client_keys)        # 401
    body = parse_json(req)                              # 400
    require(body.model)                                 # 400
    src  = detect_format(req.path, body, headers)
    prov = select_provider(cfg, body.model, headers)
    if prov is None: return 400
    if body.stream: return stream(body, src, prov)
    return unary(body, src, prov)

Detection should use the path as primary signal (/v1/messages → Anthropic), with body heuristics (system at top level, input without messages) as fallback for proxied or rewritten paths. Never rely on body heuristics alone: an empty messages array is legal in both dialects.

Provider selection is a priority list, documented and stable:

1. explicit per-request override header (escape hatch, always honored)
2. model-prefix routing (e.g. claude* → first Anthropic provider)
3. configured default
4. first provider in insertion order

Return None rather than raising; a missing provider is a client-visible 400, not a crash.

5. Streaming is the actual engineering

Non-streaming is one await client.post(...). Streaming is where the design earns its keep.

OpenAI emits flat deltas terminated by data: [DONE]. Anthropic emits a typed event grammar: message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop. To bridge them you need a per-stream state machine:

State { started, finished, block_type, block_index, prompt_tokens, output_tokens, id }

on event:
  if start_event:      emit_block_open(); block_type = TEXT
  if text_delta:       ensure_open(TEXT); emit_text()
  if tool_delta:       ensure_open(TOOL(idx)); emit_json_fragment()
  if finish_reason:    close_block(); emit_terminal(); finished = true

Three rules that prevent most production bugs:

• Same-format streams must not be parsed. If client and upstream agree, pipe raw bytes. Any parse-then-reserialize step will drop fields and change chunk boundaries.
• Buffer on blank lines, not on chunks. An SSE event can arrive split across TCP reads. Accumulate lines until \n\n, then parse the group.
• Flush after the loop. Many upstreams omit the trailing blank line; without a final flush you lose the last event and the client hangs.

Also close resources in a finally: the upstream response and the HTTP client, since the generator may be cancelled mid-stream when a client disconnects.

6. Error model

Pick one envelope and never leak vendor shapes. Normalize to {"error": {"type", "message"}}, and map:

Condition	Status
missing/invalid client key	401
malformed body	400
no provider for model	400
upstream connection failure	502
upstream 4xx/5xx	pass through status + normalized body
upstream 200 but unparsable	502

Returning error objects instead of raising is reasonable here — handlers already return responses, and it keeps the streaming path uniform.

7. Concurrency and ops

Use one HTTP client per app with a bounded pool, not per request; per-request clients destroy keep-alive under load. Set separate connect and read timeouts — connects should fail fast (5–10s), reads must tolerate long generations (minutes). Add no retries to streaming endpoints; a retried POST can double-bill tokens. Log one structured line per request: client format, provider, model, status, token counts, duration, whether it streamed.

8. Testing strategy

• Golden fixtures per direction: request-in → upstream-body, upstream-body → client-body.
• Contract tests that replay recorded SSE byte streams through the state machine and assert the emitted event sequence.
• A fake upstream (in-process ASGI app) so tests never touch the network.
• A matrix test asserting all four format pairs, streamed and not, plus every error branch above.

9. Traps checklist

Dropping tools on translation; forgetting max_tokens is mandatory for Anthropic; streaming token usage unless you request it; reusing upstream message IDs across translated streams; treating [DONE] as valid JSON; assuming one provider per protocol when the config allows several.

Build the IR, keep streaming raw when formats match, normalize errors, and test the SSE state machine with recorded bytes. That is the whole system.

Here’s the request flowchart:

How to read it:

• Three entry families. Chat and passthrough routes both authenticate and parse JSON; /v1/models and /health diverge early — /health never touches an upstream, and /v1/models sweeps every provider.
• detect_format runs on the path, so /v1/messages → claude and /v1/chat/completions → openai. The body heuristics only fire if the path substring doesn’t match.
• Green branch = no work. When the client’s format already matches the provider’s, the body is only shallow-copied (plus forced stream), and streams pass through byte-for-byte.
• Red = exit, and every one of them returns the same {"error": {"type", "message"}} envelope, including upstream failures.