# Realtime session management

Provider-agnostic session layer for realtime, voice-first conversations. Backed today by a single provider — the OpenAI Realtime API — but designed so a second provider or a multi-stage pipeline (STT → LLM → TTS) can be added later without touching the session orchestrator, the WebSocket plumbing, or any client code.

## Why a WebSocket relay, not WebRTC-direct

The connection is **browser ↔ our SvelteKit server ↔ the model provider**, not browser-direct-to-provider. Concretely: the browser holds one WebSocket to our server; our server holds a second WebSocket to OpenAI; our server relays between them.

This was a deliberate choice, made explicitly rather than defaulted into. OpenAI's own recommended pattern for browser clients is the opposite: the browser holds a WebRTC connection **directly** to OpenAI, and the backend's only job is minting short-lived ephemeral tokens — lower latency, no audio bandwidth through your own server.

We chose the relay instead because:

- **One uniform transport.** Both "talk to OpenAI Realtime" today and "run a multi-stage pipeline" later look identical from the browser's side — a WebSocket to our server speaking our own protocol. A WebRTC-direct approach would need an entirely different client-side transport for a non-realtime pipeline (e.g. SSE), unified only by a shared interface at best. Here, the provider swap happens entirely server-side, so the browser-facing contract never has to change.
- **A server-side hook point.** Because every event passes through our server, we can add moderation, analytics, avatar-specific processing (e.g. viseme/lipsync generation from audio output), or business logic in the middle later — without ever touching the browser client.
- **No ephemeral-token dance.** Since the browser never talks to the provider directly, our server just uses the real `OPENAI_API_KEY` server-to-server. No `client_secrets` minting, no token refresh handling in the browser.

The accepted tradeoff: an extra network hop, and our server carries audio bandwidth in both directions instead of peer-to-peer. For a base project optimizing for extensibility and a clean client-facing contract over shaving off milliseconds, that's the right tradeoff — but it's worth knowing this was a real fork in the road, not an oversight, if you're ever deciding whether to add a WebRTC-direct transport alongside this one.

## Connection flow

```mermaid
sequenceDiagram
    participant B as Browser
    participant S as SvelteKit server
    participant O as OpenAI Realtime API

    B->>S: POST /api/realtime/session
    S-->>B: { token, expiresAt }
    B->>S: WS upgrade wss://.../api/realtime/ws?token=...
    S->>S: verify token (tokens.ts)
    S-->>B: 101 Switching Protocols
    B->>S: { type: "session.start", config? }
    S->>O: wss://api.openai.com/v1/realtime?model=...
    S->>O: { type: "session.update", session: {...} }
    O-->>S: session.created / session.updated
    S-->>B: { type: "session.ready", sessionId }
    B->>S: input_text.send / input_audio.chunk / ...
    S->>O: translated OpenAI wire events
    O-->>S: response.output_audio.delta / response.output_audio_transcript.delta / ...
    S-->>B: output_audio.chunk / transcript.delta / ...
```

Either side disconnecting or erroring tears down the other — see `RealtimeSession.close()` in [`session.ts`](../src/lib/server/core/session.ts).

## The extensibility seam

`src/lib/server/core/provider.ts` defines two interfaces:

```ts
interface RealtimeProvider {
	createSession(config: ResolvedSessionConfig): Promise<ProviderSession>;
}

interface ProviderSession {
	send(event: ClientEvent): void;
	onEvent(handler: (event: ServerEvent) => void): void;
	close(reason?: string): void;
}
```

The critical constraint: a `ProviderSession` speaks **only** the normalized `ClientEvent`/`ServerEvent` vocabulary from [`src/lib/core/protocol.ts`](../src/lib/core/protocol.ts) — never a provider's own wire format. `RealtimeSession` (the orchestrator in `session.ts`) only ever talks to that interface, never to OpenAI directly.

**To add a new provider or pipeline:** write a new file under `src/lib/server/core/providers/` implementing `RealtimeProvider`, translating between the normalized protocol and whatever that backend speaks (see `providers/openai.ts` for the reference implementation). Then wire it in wherever `OpenAiRealtimeProvider` is currently instantiated ([`server.js`](../server.js) for production, [`dev-plugin.ts`](../src/lib/server/core/dev-plugin.ts) for dev). Nothing else changes — not `session.ts`, not `ws-handler.ts`, not any client code.

## Resilience

`src/lib/client/core/resilient-client.ts`'s `ResilientRealtimeClient` wraps `RealtimeClient` (composition, not inheritance — `RealtimeClient` itself stays untouched, still pure transport) with automatic reconnection on an _unexpected_ transport drop. It exposes the same `send()`/`on()`/`close()` shape, plus a separate `onConnectionState(handler)` for transport-level state (`connecting | open | reconnecting | closed`) — kept out of the `ServerEvent`/protocol vocabulary deliberately, since that's the provider-relay contract, not the place for this transport's local retry bookkeeping.

**Reconnecting the browser↔relay socket does not resurrect a dead relay↔OpenAI session.** If the server proactively sends `session.closed` (`upstream_disconnected`, `client_requested`, etc.), that's a clean end — no auto-reconnect follows. Auto-reconnect only fires when the underlying socket closes with **no** preceding `session.closed` (a network blip, laptop sleep/wake, an idle timeout) — detected via `RealtimeClient.onClose()`, a small addition to `client.ts` alongside its existing `on()` (message events) for exactly this purpose, since without it there was no way to observe a drop that isn't itself a parsed `ServerEvent`.

Either way, a reconnect always starts a **brand-new** session — fresh token (`getToken` is called again; tokens are short-lived and a stale one can't be reused), fresh `session.start`. The model has no memory of the old session regardless, so there's no conversation state worth trying to resume. Callers see this plainly as a `reconnecting` → `open` transition followed by a fresh `session.ready`, and decide for themselves how to present it (clear the transcript, show a toast, etc.) — the wrapper doesn't hide it. See `/demo/realtime` for the reference wiring, including how it resets its own transcript on `reconnecting`.

Backoff is exponential (`baseDelayMs * 2^(attempt-1)`), capped at `maxAttempts` consecutive failures before giving up and reporting `closed` — both configurable per instance, no env var (this is browser-local behavior, not server config).

## Development without a live API

`src/lib/server/core/providers/mock.ts`'s `MockRealtimeProvider` implements the same `RealtimeProvider` interface as `OpenAiRealtimeProvider` (see "The extensibility seam" below), but replays a canned `MockScript` (`Array<{ delayMs, event: ServerEvent }>`, typed directly against `ServerEvent` so a fixture can't silently drift from the protocol) on a timer instead of talking to a real upstream. `send()` is a no-op — v1 is linear, one-shot replay, not interactive/branching scripting.

Select it with `REALTIME_PROVIDER=mock` (see [CONFIGURATION.md](CONFIGURATION.md)); `OPENAI_API_KEY` isn't required in that mode. `src/lib/server/core/providers/index.ts`'s `selectProvider(config)` is the one place both composition roots (`server.js`, `dev-plugin.ts`) call to pick between providers, instead of each hardcoding `new OpenAiRealtimeProvider(...)`.

This exists mainly so client/animation work (transcript rendering, `$lib/humanoid`'s avatar-state derivation — see [HUMANOID.md](HUMANOID.md)) can happen without an API key or live usage. `providers/fixtures/default.ts` ships one short canned exchange for exercising exactly that; see `/demo/humanoid`.

## The trust boundary

The browser's `session.start` message can only set `voice` (see `SessionStartConfig` in `protocol.ts`). `instructions`, `tools`, and `model` are always server-owned — sourced from the project's persona ([`src/lib/project/persona.ts`](../src/lib/project/persona.ts), `model` optionally overridden by `SESSION_MODEL`; see [docs/CONFIGURATION.md](CONFIGURATION.md)) via `resolveSessionDefaults()` (`defaults.ts`), and merged with the client-visible config in `resolveSessionConfig()` (`session.ts`). A server-side `defaultVoice` (`SESSION_DEFAULT_VOICE`) fills in when the browser's `session.start` doesn't specify a voice, but the browser's own choice still wins when it sends one — this doesn't loosen the trust boundary, since voice was already the one client-controllable field. This is deliberate: this relay is shared infrastructure that other people's browsers connect to, so a client project should never let an untrusted browser dictate what the model is told to do or which model/tools it can invoke. If a client project needs more client-controllable config, extend `SessionStartConfig` and `resolveSessionConfig()` explicitly and think about what's safe to expose — don't just pass the browser's config straight through.

## SvelteKit WebSocket plumbing

SvelteKit has no native WebSocket support as of this writing (a `crossws`-based PR for it was closed by the maintainers in Feb 2026 in favor of an unrelated SSE feature). Raw HTTP upgrade requests have to be intercepted below SvelteKit's own routing, at the underlying `http.Server` level:

- **Production**: [`server.js`](../server.js) wraps adapter-node's generated `build/handler.js` in a plain `http.createServer` and attaches upgrade handling manually.
- **Dev**: [`dev-plugin.ts`](../src/lib/server/core/dev-plugin.ts) is a Vite plugin that attaches the same handling to Vite's dev server via `configureServer`. It's careful to no-op for any path that isn't ours, since Vite's own HMR websocket shares the same underlying HTTP server — touching the socket for a non-matching path would silently break HMR.

Both call the same [`handleRealtimeUpgrade`](../src/lib/server/core/ws-handler.ts) function, so prod and dev behave identically.

**Why `src/lib/core/` and `src/lib/server/core/` avoid `$lib`/`$env` aliases:** `server.js` is a plain Node entrypoint running the _built_ app — it can't resolve SvelteKit's `$lib`/`$env` virtual modules, which only exist inside Vite's build graph. So this module tree uses plain relative imports and reads secrets via `process.env` instead of `$env/static/private`, which lets it be compiled standalone (`tsc -p tsconfig.server.json` → `build-server/`, run as part of `npm run build`) for `server.js` to import directly. SvelteKit _routes_ (`+server.ts`, `+page.svelte`) are unaffected by this and use the normal `$lib` alias as usual.

**Known gotcha:** because this module tree is only loaded via `vite.config.ts`'s plugin list (not through the app's normal module graph), it is **not** part of Vite's HMR. Editing anything under `src/lib/server/core/**` while `npm run dev` is running requires a full dev-server restart to take effect — a hot-reload will silently keep running the old code. If you change something here and testing doesn't reflect it, restart before assuming the change is broken.

## Protocol vocabulary

See [`protocol.ts`](../src/lib/core/protocol.ts) for the authoritative, current list — this is a summary, not a copy to keep in sync by hand.

- **Client → server**: `session.start`, `input_audio.chunk`, `input_audio.commit`, `input_text.send`, `response.cancel`, `session.end`, `debug.subscribe`
- **Server → client**: `session.ready`, `output_audio.chunk`, `output_audio.done`, `transcript.delta`, `transcript.done`, `response.done`, `error`, `session.closed`, `debug.subscribe.result`, `debug.log`

`debug.subscribe`/`debug.subscribe.result`/`debug.log` are a separate concern from the rest of this vocabulary — ephemeral developer debug-log forwarding, not session content. See [LOGGING.md](LOGGING.md).

## OpenAI Realtime API gotcha (verified live, Aug 2026)

OpenAI's official docs were incomplete when this was built (WebFetch on `developers.openai.com/api/reference/resources/realtime/*` truncated before covering response streaming events), and general web search suggested event names that turned out to be stale. The actual, currently-live event names for output content carry an `output_` prefix:

- `response.output_audio.delta` / `response.output_audio.done` — not `response.audio.delta`
- `response.output_audio_transcript.delta` / `response.output_audio_transcript.done` — not `response.audio_transcript.delta`

Response-lifecycle events (`response.created`, `response.output_item.added/done`, `response.done`) are unaffected. This is handled in [`fromOpenAiEvent`](../src/lib/server/core/providers/openai.ts). If OpenAI changes their event schema again, unrecognized events are silently dropped (by design — see `default: return null` in that function); if a response's content stops showing up in the browser, temporarily add a `console.debug` in `handleUpstreamMessage`'s unmapped-event branch, restart the dev server, and read the real event names from `preview_logs` rather than re-guessing from docs.

Still unverified: the exact event name for a completed **input** audio transcription (`conversation.item.input_audio_transcription.completed` is the current guess, unconfirmed since testing so far only exercised text input).

## What's deliberately not built yet

- Audio capture/playback in the browser client (`client.ts` is protocol-level only)
- Rate limiting on the token-mint endpoint
- Real user accounts (`tokens.ts` is anonymous-session-only)
- Multi-instance/shared session storage (in-memory, single-process only)
- Binary audio framing (currently JSON + base64, mirroring OpenAI's own wire format)

## Verifying changes

- `npm run check` / `npm run lint` / `npm run build` for type, lint, and build correctness (`build` also runs the standalone `tsc -p tsconfig.server.json` compile — watch for errors there specifically, since `svelte-check` alone won't catch problems in code that isn't reached through the SvelteKit app).
- `npm run test:unit` — unit tests for `tokens.ts`, `session.ts` (state machine, against a fake in-memory provider), `providers/openai.ts`'s translation functions, `providers/mock.ts`'s script replay, `resilient-client.ts`'s backoff/reconnect logic, and `$lib/humanoid`'s reducer/compositor.
- **Live smoke test**: set `OPENAI_API_KEY` and `REALTIME_TOKEN_SECRET` in `.env` (see `.env.example`), run `npm run dev`, open `/demo/realtime`, and actually send a message. Type-checking and unit tests can't catch a wire-protocol mismatch like the one above — only a real round trip against the real API can. Remember the HMR gotcha above if you're iterating on `src/lib/server/core/**`.
- **Mock-provider smoke test**: `REALTIME_PROVIDER=mock npm run dev`, open `/demo/humanoid`, connect, and watch `avatarState` move through the default fixture — zero OpenAI usage, useful for iterating on animation timing without live traffic.