Switchboard

Problem

Apps that call LLMs each re-implement retries, provider failover, caching and spend tracking, usually badly. The hardest case, a provider dying after it has already streamed part of an answer, is rarely handled at all.

Approach

Building a gateway that speaks the OpenAI Chat Completions API, so any OpenAI SDK can use it by changing the base URL. Models are addressed as provider/model and each provider adapter streams through the gateway as OpenAI-format server-sent events. A mock provider fails on command through a request header (errors before streaming, dropped or stalled connections, malformed chunks, in-stream errors), so every failure mode can be tested on purpose.

Tech highlights

pnpm workspaces and Turborepo; a Hono gateway on Node with API-key auth and Zod validation; a shared core package with the SSE parser, OpenAI wire types and provider errors; Vitest tests, including one that drives the gateway with the official OpenAI SDK; Docker Compose with Postgres and pgvector, Redis and a Grafana OpenTelemetry stack; GitHub Actions running Biome, type-checks, tests and Docker builds.

Results

In progress, week 1 of a 6-week plan: the streaming gateway, the chaos mock provider, tests, Docker and CI are built, and the gateway is deployed on Vercel behind an API key. Retries and fallbacks with mid-stream resume, a circuit breaker, per-key budgets, a tenant-safe semantic cache and a Python evals service come next. Follow the build on GitHub.