Tools

A 1ms Open-Source LLM Gateway That Maintains Itself

A Rust-built gateway that unifies self-hosted, closed-source, and open-source models with BYOK overhead under 1ms. A Codex Agent opens daily PRs to keep the model registry in sync.

Font size

Yesterday, experientiallabs/experiential hit the front page of Hacker News under the banner "open OpenRouter." My first reaction: another open-source clone of OpenRouter, saving the 5% markup, nothing new to see.

Then I read the README. The interesting part isn't the latency numbers. It's that this project maintains itself: a Codex Agent crawls provider APIs every day, checks for new models, and opens PRs to update the model catalog. Developers just review. Nobody manually tracks metadata for 1,000+ models.

That turns out to be a much bigger story than shaving milliseconds.

Not just another OpenRouter alternative

OpenRouter solves "find a model, plug in a key, pay for it." Experiential wants to do the same thing, with an extra layer: an agent baked in to act as the repo's janitor.

That nuance is easy to miss. Look at "BYOK <1ms" and you'd think it's just another LiteLLM / Portkey / OpenRouter competing on the latency curve. But the maintainers kept emphasizing something else in the HN thread: "self-updating model registry."

When someone asked "isn't this just another LiteLLM?" the answer was roughly: LiteLLM's model list is hand-maintained. We use an agent that adds models automatically, runs daily, and opens a PR each time.

That's the real difference. Latency is just the entry ticket. The differentiation is "infrastructure that edits itself."

How the latency numbers happen

BYOK requests add under 1ms of overhead. Official-key requests come in under 2ms. These are the numbers the maintainers posted on HN, and they're in the README too.

Two design choices get them there:

  1. Native Rust, no Python wrapper. Python on ASGI/WSGI burns 5-20ms just on in-process serialization and async scheduling. Rust with tokio + hyper is basically route matching plus header passthrough.
  2. The request body is barely parsed. The gateway checks auth, validates the model name, picks the target endpoint, and forwards. No retry logic, no prompt rewriting, no caching. All of that is upstream's problem.

By comparison, gateways like LiteLLM and Portkey typically handle streaming format normalization, token counting, and retries with backoff, adding 5-50ms. Experiential pushes all of that downstream, which is why overhead stays under 1ms. The trade-off: if you want middleware behavior (say, a unified fallback policy), you build it yourself.

Official keys clock in under 2ms because the request still hits upstream auth and the TLS handshake. BYOK paths stay under 1ms because the key lives on your side and TLS sessions can be reused.

The Codex Agent's daily PR workflow is the real trick

This is the sharpest part of the project.

There's a workflow in the repo (I believe the path is .github/workflows/sync-models.yml) that runs roughly like this:

  1. The Codex Agent gets triggered on a schedule (daily at 00:00 UTC).
  2. It scans every provider's /v1/models endpoint or equivalent (OpenAI, Anthropic, Google, DeepSeek, Cohere, xAI, Mistral, plus a long tail of self-hosted backends).
  3. It diffs the result against the repo's models.json (or whatever the index file is called).
  4. New models, deprecations, price changes become a diff and a PR.
  5. Each PR ships a full changelog, source links, context window, pricing, and other metadata.

Maintainers just review. No more manual additions.

I tried to picture what happens after a week in production:

  • Day 1: syncs 1,000+ models, most are refreshes of existing entries.
  • Day 3: detects a new model from one provider, opens a PR to add it.
  • Day 5: a model gets deprecated, the bot flags it in a PR.
  • Day 7: GitHub Actions has run roughly seven times, PR count has climbed from zero to maybe 10-20.

This is the early shape of self-improving infrastructure. The infrastructure is no longer code you write once and forget. It grows alongside upstream changes.

I haven't seen anything quite like it. The ecosystem has auto-updating dependencies (Dependabot), but auto-updating model registries is new. The pain point is concrete: Anthropic ships four models a year, OpenAI matches that pace, Google goes faster, and the open-source world (Qwen, DeepSeek, Llama) drops new weights almost daily. Manual tracking can't keep up.

The real compatibility problems it tackles

Even if you don't adopt it as your primary gateway, the list of compatibility issues it handles is worth reading. Five categories:

1. Streaming format differences

OpenAI uses SSE with event type chat.completion.chunk and a delta of {role, content}. Anthropic uses a different SSE layout: message_start / content_block_delta / message_stop, with deeper delta nesting. Google Gemini is yet another scheme.

Experiential normalizes at the gateway. Send an OpenAI-format request, it gets translated to Anthropic format upstream. The Anthropic events come back translated into OpenAI deltas. Downstream code doesn't care who's on the other end.

2. Tool-calling JSON schema mismatches

OpenAI returns tool calls as {tool_calls: [{id, type, function: {name, arguments}}]}. Anthropic returns {content: [{type: "tool_use", id, name, input}]}. The arguments field is a string in OpenAI (you call json.loads yourself) and an object in Anthropic (use directly).

That schema mismatch forces every codebase to write two branches. Experiential flattens it at the gateway.

3. Model parameter mapping

max_tokens, temperature, top_p are mostly consistent, but stop sequences, frequency_penalty, and presence_penalty vary in support across providers. Anthropic's thinking parameter and OpenAI's reasoning_effort were both bolted on later.

4. Rate-limit headers are all over the place

HTTP 429 is standard, but Anthropic returns anthropic-ratelimit-requests-remaining while OpenAI returns x-ratelimit-remaining-requests. retry-after semantics differ too. Experiential normalizes these into a unified error structure.

5. Error handling

Each provider has its own error codes (context_length_exceeded vs. invalid_request_error vs. RESOURCE_EXHAUSTED). Experiential translates them into OpenAI-compatible error objects, so downstream catch logic only gets written once.

90% of teams writing their own wrappers hit at least some of these. Experiential handles them centrally, and the code savings add up.

Is it worth picking up?

A decision tree:

  • You're already on OpenRouter or LiteLLM with no real pain points: stay put and watch. Experiential's ecosystem is still young, and docs, monitoring, and dashboards are catching up.
  • You're self-hosting vLLM (or TGI, SGLang) for open models while also calling Claude / GPT / Gemini: the migration pays off. Experiential's unified auth, logging, and billing are cleaner than hand-rolled wrappers.
  • The exploding model catalog already hurts (you've noticed new models slipping through days late): try it now. This use case is first-of-kind. There's no replacement.

Source is at github.com/experientiallabs/experiential. Follow the README steps to get started. For production deployments, run dev mode first to gauge the Rust compile time; the project is large and a cold build can take 5-10 minutes.

Editor's take

Will self-maintaining infrastructure become a new pattern? My read: yes, but with boundaries.

Code that edits code has worked out in dependency management (Dependabot), security patches (Snyk auto-fix), and documentation generation (plenty of projects use LLMs to write changelogs). A model catalog is a harder version of those: cross-provider, high change frequency, many metadata dimensions. Experiential is the first to apply the pattern here, and that's worth noting.

But the problem it solves is specific: catalog sprawl. Not every piece of infrastructure is a fit for self-maintenance. State machines, distributed protocols, cryptographic primitives are too risky to hand to an agent. So the "agent maintains infra" pattern will spread, but only into the slice where changes are frequent and failure costs are low.

Open questions:

  1. Can the PR workflow stay reliable over time? What happens when Codex hallucinates a model that doesn't exist? Is there a safety net requiring maintainer approval before any agent PR gets merged? I haven't fully traced their current design.
  2. Will the big players catch up fast? OpenAI has floated the idea of a proxy (not a product). Anthropic hasn't shipped one. Cloudflare Workers AI is doing something adjacent. If Cloudflare copies this pattern, Experiential's head start shrinks.
  3. The real killer use case might not be the gateway. If "agent automatically maintains a model catalog" works, the next stops are obvious: prompt libraries, eval suites, RAG indexes. Experiential capped as a gateway has a ceiling. Experiential as a generic framework for "agent-maintained ML infra" would be a much more interesting story.

That's why I pulled it out for its own writeup. Not because of the latency numbers, but because it might be the seed of the next AI infrastructure pattern.

Source: https://github.com/experientiallabs/experiential