Inside the SchneeAI Gateway

Most “AI Gateway” pages on the internet describe the idea in one paragraph and the architecture in one box-and-arrow diagram. This post is the opposite: one paragraph for the idea, then every hop in the request path laid out as plain text, with the design choice behind it. If you are evaluating SchneeAI for a workload that needs audit-grade controls, this is what you are buying.

The security page has the summary and the diagram. This is the long version.

The eight hops

When your backend calls POST /v1/chat/completions, the request travels through the Gateway in this order:

1. Terminate TLS — request lands at the cluster ingress on TLS 1.3 (1.2 with forward secrecy for legacy clients).
2. Verify the Bearer JWT — JWKS fetched from Ory Hydra, signature checked, claims extracted.
3. Apply routing policy — model alias resolves, budget checked, PII policy loaded.
4. Scan for PII / secrets — 17 categories, before the request leaves the boundary.
5. Call the provider via LiteLLM — model alias mapped to provider, retry budget applied.
6. Write raw exchange into the Vault — S3 / R2 with server-side encryption (SSE-S3, AES-256); application-layer envelope encryption on the roadmap.
7. Write structured usage and audit records — Postgres, append-only audit table.
8. Return the response — same shape OpenAI returns, plus SchneeAI extensions when relevant.

Each hop is a failure surface and a design decision. Let’s walk them.

1. TLS termination

TLS is the first control and the one teams spend the least time thinking about. We enforce HSTS at the cluster edge, terminate TLS there, and re-encrypt for internal hops within the cluster (mTLS between Gateway and LiteLLM, between Gateway and Vault writer).

The choice that matters: we do not terminate TLS at a CDN and forward plaintext to the origin. This matters less for cost (CDN termination is cheap) and more for threat model — a CDN misconfiguration that exposes origin traffic is one of the more common ways “encrypted in transit” turns out to mean “encrypted between you and the CDN, then plaintext for the rest.” With end-to-end TLS + mTLS internal, a misconfigured edge still cannot read provider prompts.

2. JWT verification

The Gateway verifies every request against Ory Hydra’s JWKS. The JWT carries sub (platform user ID, UUIDv7), active_tenant_id, tenant_role, and a service_id claim that scopes the call to a specific registered service. If service_id is absent on a legacy token, we fall back to the first entry in aud.

Design choice worth noting: service_id is a claim on the token, not a header. We considered X-Service-ID for a long time because it’s easier to mint on the client side. The problem is that a header is trivially spoofable by anyone who can mint a valid user JWT — which means the service boundary becomes advisory instead of enforced. Putting it in the signed claim makes the service boundary a real boundary.

The JWKS is cached with a short TTL and rotated on a fixed schedule. Key rotation does not require redeploying the Gateway.

3. Routing policy

Once identity is established, the Gateway loads the routing policy: model alias → provider model, budget state for (service, tenant, feature), and the PII policy in force.

"model": "auto" is resolved here. It is not a wildcard. It is a routing decision made against your tenant’s configured policy, which can take into account:

• Feature tag (feature=reply_suggestion → route to value tier)
• Budget remaining (run out of Sonnet budget → fall back to Flash)
• Latency target (interactive features → prefer Gemini Flash / Haiku)
• PII policy (critical PII detected → block provider calls, return early)

The reason this lives in the Gateway instead of in your service code: policies drift. A team that hard-codes “use Sonnet for replies” in their backend will keep doing it long after Sonnet becomes the wrong choice. Centralizing the policy makes the change one config edit, audit-logged, with canary support.

4. PII / secret scan

Before the request leaves the SchneeAI boundary, we run a structured scan over both the system and user messages. Seventeen categories today: emails, credit cards (with Luhn), Japanese MyNumber (with check digit), US SSN, AWS / Google / OpenAI / Anthropic / Stripe / Slack API keys, JWTs, PEM private keys, phone numbers (JP / E.164), IPv4 / IPv6, and connection strings.

Each pattern has a severity tier. Critical findings (API keys, connection strings) can block the call entirely. Warning-level findings (emails, phones) can be masked or flagged. The action is policy-driven per tenant.

The non-obvious design choice: the scan runs before the provider call, not after. Once the request has left your boundary, you are at the provider’s data-residency and retention policy — you cannot un-send. Scanning pre-call means a leaked API key in a prompt never reaches the provider.

Read the design write-up: PII scanning in production.

5. Provider call via LiteLLM

The Gateway calls the provider through LiteLLM, which handles:

• Provider-specific auth (each provider has its own key, held by SchneeAI, scoped, rotated).
• Provider-specific request/response shape (OpenAI / Anthropic / Google / Mistral / Cohere / DeepSeek).
• Retries with exponential backoff and jitter, on 429 / 500 / 502 / 503 / 504.
• Streaming SSE pass-through.

LiteLLM is a library, not a black box. We pin the version, vendor its config, and treat it as a port adapter in our Clean Architecture — the Gateway’s domain layer never sees LiteLLM types. If we ever swap it for a different proxy (or a direct integration), the domain code does not change.

Why LiteLLM at all? Because building and maintaining provider adapters is a treadmill. New models ship weekly. Pricing changes monthly. Capability flags (tool calling, vision, reasoning content) appear and disappear. A team that maintains its own adapters spends a non-trivial fraction of engineering time on a layer that delivers no business value. Outsourcing that layer to a focused library frees the team to work on the layers that do.

6. Vault write

The raw request and response — system prompt, user message, model output, reasoning content if present — are written to the Vault. The Vault is S3 / R2-compatible object storage with server-side encryption enabled (SSE-S3, AES-256). The bucket never sees plaintext: S3 encrypts on write and decrypts on read, with keys managed by the storage provider.

Application-layer envelope encryption (AES-256-GCM with KMS-wrapped envelope keys, 90-day rotation) is on the roadmap. Once landed, each blob gets its own envelope key wrapped by a KMS master key — so key rotation won’t require re-encrypting existing blobs, and the storage provider’s keys become a second layer rather than the only one. The EncryptionKeyID column on vault_blobs is already in the schema to support this.

The non-obvious design choice: the Vault is separate from Postgres. We considered storing raw prompts in a Postgres column with column-level encryption. It’s simpler architecturally. The problem is retention — Postgres is optimized for online queries, not for holding years of large blobs that are almost never read. Keeping raw content in object storage means the operational database stays small and fast, while the audit-grade raw record lives in a store designed for cheap long-term retention. See Inside the Vault for the full walk-through.

7. Usage and audit records

Two writes happen after the provider responds:

• Usage record (Postgres): service, tenant, user, model, input tokens, output tokens, cost in micro-USD, latency, status code. Indexable, queryable via /v1/usage. This is what powers the cost dashboard and budget enforcement.
• Audit record (Postgres, append-only): actor, target, action, timestamp. Covers request, prompt change, policy update, operational action. This is what reviewers ask for during an incident.

Why split? Because the access patterns are different. Usage is queried constantly (dashboards, budget checks, customer-facing). Audit is queried rarely (incidents, compliance reviews) but must be impossible to silently modify. Keeping them in separate tables with separate access controls means a bug in the usage dashboard cannot accidentally corrupt the audit trail.

8. Response

The response is OpenAI-shaped: choices[0].message.content, usage.prompt_tokens, usage.completion_tokens. SchneeAI extensions live alongside (schneeai.budget_remaining, schneeai.vault_id, schneeai.trace_id) so existing OpenAI SDKs work unchanged and observability hooks have what they need.

If the call was streaming, the Gateway streams the provider’s SSE back to the client as it arrives. The Vault write and audit record happen after the stream closes.

When things go wrong

The eight-hop description is the happy path. The interesting design work is in the failure modes:

• JWKS temporarily unreachable → Gateway returns 503 with schneeai_error: jwks_unavailable. We do not fall back to a cached key beyond its TTL.
• Routing policy decides the call exceeds budget → 402 with schneeai_error: budget_exceeded. No provider call, no Vault write, audit record marks the refusal.
• PII scan detects a critical secret → 422 with schneeai_error: pii_detected, listing the categories. The finding itself is not echoed back (the secret should not appear in the error body).
• Provider times out → 504 with schneeai_error: provider_timeout after the retry budget is exhausted. Partial Vault write happens for whatever was streamed before the timeout.
• Vault write fails → Gateway returns 200 to the caller (they got their answer) but flags the interaction as vault_write_failed in the audit log. An on-call worker retries the write with backoff; if it keeps failing, the raw exchange is logged to encrypted disk as a last-resort buffer and paged on.

The principle: fail in the direction that protects the user’s data, not the direction that protects our error rate. A 200 with a vault_write_failed flag looks worse in our dashboards than silently returning 200 and dropping the record — but it’s what an incident responder needs.

What this buys you

Eight hops is a lot. The reason to run your AI traffic through them — rather than calling OpenAI directly from your backend — is that each hop is a control you would otherwise build yourself:

• TLS termination with end-to-end encryption: 1-2 eng-weeks
• JWT / JWKS verification with claim-based service scoping: 2-3 eng-weeks
• Centralized routing policy with per-tenant config: 3-4 eng-weeks
• PII scanning with 17 categories and policy actions: 4-8 eng-weeks
• Provider abstraction with retry and streaming: 1-2 eng-weeks (or ongoing, if you maintain it yourself)
• Encrypted raw retention with SSE-S3 (shipped) and application-layer envelope encryption (roadmap): 3-6 eng-weeks plus KMS setup plus legal review
• Structured usage and append-only audit: 2-4 eng-weeks
• OpenAI-compatible response shaping with extensions: 1-2 eng-weeks

Summed: roughly 17-31 engineer-weeks to build the equivalent, plus ongoing operations. During the design-partner phase SchneeAI passes provider cost through at 1× (no markup); the ai_model_pricing.markup_bps column is already wired so per-model margins can be introduced without a migration. For most teams, that is cheaper than the build.

If you want to see it for yourself, the Tutorial walks through the same eight hops from the caller’s perspective, with copy-runnable code in curl, Python, and TypeScript.