Failover design in the SchneeAI Gateway

“What happens when OpenAI has an outage?” is one of the first questions a design partner asks. It is a fair question, and most “AI Gateway” landing pages answer it with a one-line promise — seamless cross-provider failover — and a diagram with arrows between provider logos.

This post is the longer version. We split provider resilience into three layers, tell you which layer ships today, which one is modeled in our domain code but not yet wired into the request path, and which one is on the roadmap. Then we explain why we held back the part that most vendors claim to ship on day one.

The short version: same-provider retry on 5xx and 429 is in production. Cross-provider failover is implemented as a domain model in internal/core/routing/ but is not yet invoked from the chat completion path. The reasons are not technical — the wiring is a few days of work. The reasons are about audit, billing, and cost predictability, and we want to ship failover in a form that does not surprise anyone on the bill.

1. Three layers of resilience

It helps to separate the three things teams mean when they say “failover”:

• Retry within a provider — the upstream returned a 5xx or a 429, or the network dropped. Retry the same call against the same provider, same model, with exponential backoff. This is what catches the majority of real-world transient failures.
• Cross-provider failover — the primary provider is degraded (elevated 5xx rate, regional outage, announced incident). Re-route the request to a different provider’s equivalent model. The caller gets an answer; the model may be different.
• Canary / model-migration failover — a new model version ships, behavior shifts, and you want a rapid, partial rollback to the previous version without a code deploy. This is closer to release engineering than to provider resilience, but it shows up in the same conversations.

These layers compose. The post focuses on the first two; the third is mentioned for completeness and is further out.

2. What ships today

Every provider call goes through LiteLLM, and the Go layer wraps the call with coreerrors.Retryable(). The retry rule is simple:

• Retry on HTTP status >= 500 or 429.
• Exponential backoff with jitter (1s base, 60s cap, ~25% jitter).
• Honor Retry-After when the provider sends it.
• Same provider, same model — the retry is a fresh POST to the same endpoint.
• Retry budget caps at 5 attempts by default.

If the retry budget is exhausted, the Gateway surfaces a structured error to the caller: 502 with code: provider_failed (the provider returned errors on every attempt) or 504 with code: provider_timeout (the call exceeded the timeout budget). Both are documented on the Error Handling page with the exact retry semantics.

This handles the common case: a transient 503 from a provider, a brief rate-limit blip, a network half-open connection. It does not handle the case where the provider is hard-down for an hour — for that, you need layer 2.

The principle here is the same one we apply to all failure modes: be explicit. A retry budget that quietly succeeds eventually is fine; a retry budget that quietly gives up and returns a 200 with a partial response is not. If we cannot serve the request, we tell you with a code you can branch on.

3. The domain model we’ve built (but not yet wired)

internal/core/routing/routing.go is a complete domain model for layer 2:

type RoutingPolicy struct {
    PrimaryProvider  string   // e.g. "openai"
    PrimaryModel     string   // e.g. "gpt-4o-mini"
    FallbackProvider string   // e.g. "anthropic"
    FallbackModel    string   // e.g. "claude-haiku-4-5"
    Strategy         Strategy // priority | cost | latency
}

type ProviderHealth struct {
    Provider     string
    Healthy      bool
    LatencyMs    int64
    CostMicroUSD int64
}

ResolveModel walks the policy: try the primary provider if healthy, fall back to the secondary if not, otherwise pick the best healthy provider by strategy (priority order, cost rank, or latency rank). Three strategies ship in the domain: StrategyPriority, StrategyCost, StrategyLatency.

What is not happening today: the chat completion handler at internal/delivery/http/ginserver/handler/openai_compat.go:193 calls h.catUC.ResolveModel(ctx, req.Model), which is catalog.ResolveModel — a slug lookup against the model catalog. It returns a single resolved model. The RoutingUsecase is instantiated in the DI container (internal/app/gateway/app.go:197) but is not passed into OpenAICompatHandler. The domain code is there; the request path does not call it.

Why build the domain first and wire it second? Because once the chat handler does call RoutingUsecase, every caller-visible behavior — resp.model, audit records, billing, PII scan results — has to be consistent with the routing decision. Building the domain first let us nail down the invariants before exposing them to callers. Wiring is the easy part; the caller contract is the hard part.

4. Why we held back

Cross-provider failover is the case where most gateway vendors overclaim and underdeliver. The honest reasons we have not shipped it yet:

Health checking. Knowing that a provider is unhealthy is itself a design problem. Active probing (synthetic requests on a schedule) adds load to the provider and may not reflect real customer traffic. Passive observation (EWMA of error rate and latency from real requests) is more accurate but lags — by the time the EWMA crosses a threshold, real customers have already seen failures. We need both, with the circuit breaker opening on the active signal and closing on the passive one.

Audit implications. When a request fails over from claude-haiku-4-5 to gpt-4o-mini, two providers’ data policies now apply to the same logical request. EU customers who picked Anthropic for its data-residency commitments may not want their prompt transiting OpenAI’s infrastructure, even for a single failover request. The failover has to be policy-aware: the routing policy has to know which providers are eligible for which tenants.

PII scanning. The pre-call PII scan runs against the primary provider’s policy. If the call fails over to a different provider, do we re-scan against the fallback’s policy? Today the scanner produces one set of findings; failover would need either cached results that travel with the request or a re-scan at the new boundary.

Cost predictability. A silent failover from Gemini Flash to Claude Opus is roughly a 50× cost surprise per token. Design partners are evaluating SchneeAI partly because they want predictable bills. An automatic failover that quietly upgrades the model tier on every transient 5xx would burn that trust in a single billing cycle.

Idempotency. A client sends Idempotency-Key: abc123. The primary provider fails mid-stream. We fail over to the fallback, which succeeds. Is that one billable interaction or two? Today the answer is simple — one interaction, one provider. Failover makes the answer interesting, and interesting billing is bad billing.

The conclusion we reached: ship retry first (it is safe — same provider, same model, same cost). Ship failover second, with explicit opt-in via routing policy, a cost ceiling that refuses fallbacks more expensive than N× the primary, and response extensions that make failover observable to the caller.

5. What’s needed to ship it

Concrete checklist, in dependency order:

1. Wire RoutingUsecase into the chat handler. Either swap catalogUC for routingUC, or chain them: catalog resolves the alias, routing decides the provider given the policy and health.
2. Provider health infrastructure. In-process circuit breakers per provider, fed by passive EWMA from real request outcomes, with optional active probing on a configurable schedule.
3. Policy-aware failover eligibility. The routing policy has to know which providers a tenant is allowed to fail over to (data residency, PII policy, contractual exclusions).
4. Cost ceiling. A routing policy field like max_fallback_cost_multiplier: 5 that refuses to fail over if the fallback’s per-token rate is more than 5× the primary’s. The refusal is a structured error, not a silent retry at the primary.
5. Response extensions. When failover fires, the response carries schneeai.fallback_reason, schneeai.fallback_from_model, schneeai.fallback_to_model so the caller can observe and audit. resp.model already shows the resolved model today; failover would extend the format.
6. Audit record enrichment. The audit log records which provider actually served the call, not just which model was requested.
7. Idempotency rule. Same Idempotency-Key is one billable interaction regardless of which provider served it. The reconciliation worker already handles the dual-reservation case from retries; failover reuses that machinery.

None of these are research problems. They are engineering work — a few weeks, scoped behind a feature flag, with a small set of design partners opted in for the canary.

6. What you’ll see when it lands

From the caller’s perspective, failover is opt-in and observable. Default behavior does not change: if you do not configure a routing policy with a fallback, your call goes to the primary provider and stays there, retrying on transient failures exactly as it does today.

When you do opt in, the response shape tells you what happened:

{
  "model": "auto→gpt-4o-mini (fallback: claude-haiku-4-5)",
  "schneeai": {
    "fallback_reason": "primary_provider_5xx",
    "fallback_from": "claude-haiku-4-5",
    "fallback_to": "gpt-4o-mini"
  },
  "request_id": "req_01HQ..."
}

The model field keeps its existing semantics — it tells you which model actually produced the output. The extension object makes the failover programmatically detectable so dashboards and alerting can track it.

The audit log records the actual provider and model, not just the requested one. The cost dashboard breaks out fallback distribution per service and per feature, so you can see whether your routing policy is firing failover more than expected.

7. Roadmap and how to prepare

There is no published phase spec for failover yet — the domain model exists, but the integration work is competing with other priorities (the "model": "auto" router itself, prompt-management UX, GA preparation). We expect to ship it in a future phase once the design-partner cohort has weighed in on the cost-ceiling and policy-eligibility defaults.

What you can do today to prepare:

• Pin models during evaluation. If you are benchmarking SchneeAI against direct provider calls, pin the same model on both sides so you are comparing apples to apples. The router is not yet doing anything interesting.
• Watch provider_failed and provider_timeout rates. These tell you how often a provider is hard-failing on you today. That number is the upper bound on what failover would actually catch.
• Tell us your constraints. If you have data-residency requirements that constrain which providers you can fail over to, or budget ceilings that constrain how expensive a fallback can be, that is exactly the input we need to default the policy fields correctly. Email [email protected].

Honest summary

For readers who want the short version:

• Shipped today: same-provider retry on HTTP 5xx / 429 with exponential backoff and Retry-After honor. Explicit failure modes (provider_timeout, provider_failed) when the retry budget is exhausted.
• Modeled but not wired: RoutingPolicy with primary/fallback providers and models, ProviderHealth tracking, three routing strategies (priority, cost, latency). The domain code is complete; the chat completion path does not call it yet.
• On the roadmap: cross-provider failover wiring with explicit opt-in, cost ceiling, policy-aware eligibility, response extensions, and audit enrichment.
• Not planned for v1: automatic provider-quality scoring and ML-based routing. The router today is policy-driven; we have no plans to make it opaque.

If a vendor tells you they “handle provider outages seamlessly”, ask them what the response shape looks like when failover fires, whether the audit log records the actual provider, and whether there is a cost ceiling. The answers are usually more interesting than the marketing copy.

For the broader walk-through of where retry and routing sit in the request path, see Inside the SchneeAI Gateway. For the error codes referenced here, see Error Handling. For how this fits into the build-vs-buy decision, see SchneeAI vs LiteLLM. And if you want to weigh in on the design — especially the cost-ceiling default — email us.