webavatar-core/docs/REALTIME_SESSION.md

Realtime session management

Provider-agnostic session layer for realtime, voice-first conversations. Backed today by a single provider — the OpenAI Realtime API — but designed so a second provider or a multi-stage pipeline (STT → LLM → TTS) can be added later without touching the session orchestrator, the WebSocket plumbing, or any client code.

Why a WebSocket relay, not WebRTC-direct

The connection is browser ↔ our SvelteKit server ↔ the model provider, not browser-direct-to-provider. Concretely: the browser holds one WebSocket to our server; our server holds a second WebSocket to OpenAI; our server relays between them.

This was a deliberate choice, made explicitly rather than defaulted into. OpenAI's own recommended pattern for browser clients is the opposite: the browser holds a WebRTC connection directly to OpenAI, and the backend's only job is minting short-lived ephemeral tokens — lower latency, no audio bandwidth through your own server.

We chose the relay instead because:

• One uniform transport. Both "talk to OpenAI Realtime" today and "run a multi-stage pipeline" later look identical from the browser's side — a WebSocket to our server speaking our own protocol. A WebRTC-direct approach would need an entirely different client-side transport for a non-realtime pipeline (e.g. SSE), unified only by a shared interface at best. Here, the provider swap happens entirely server-side, so the browser-facing contract never has to change.
• A server-side hook point. Because every event passes through our server, we can add moderation, analytics, avatar-specific processing (e.g. viseme/lipsync generation from audio output), or business logic in the middle later — without ever touching the browser client.
• No ephemeral-token dance. Since the browser never talks to the provider directly, our server just uses the real OPENAI_API_KEY server-to-server. No client_secrets minting, no token refresh handling in the browser.

The accepted tradeoff: an extra network hop, and our server carries audio bandwidth in both directions instead of peer-to-peer. For a base project optimizing for extensibility and a clean client-facing contract over shaving off milliseconds, that's the right tradeoff — but it's worth knowing this was a real fork in the road, not an oversight, if you're ever deciding whether to add a WebRTC-direct transport alongside this one.

Connection flow

sequenceDiagram
    participant B as Browser
    participant S as SvelteKit server
    participant O as OpenAI Realtime API

    B->>S: POST /api/realtime/session
    S-->>B: { token, expiresAt }
    B->>S: WS upgrade wss://.../api/realtime/ws?token=...
    S->>S: verify token (tokens.ts)
    S-->>B: 101 Switching Protocols
    B->>S: { type: "session.start", config? }
    S->>O: wss://api.openai.com/v1/realtime?model=...
    S->>O: { type: "session.update", session: {...} }
    O-->>S: session.created / session.updated
    S-->>B: { type: "session.ready", sessionId }
    B->>S: input_text.send / input_audio.chunk / ...
    S->>O: translated OpenAI wire events
    O-->>S: response.output_audio.delta / response.output_audio_transcript.delta / ...
    S-->>B: output_audio.chunk / transcript.delta / ...

Either side disconnecting or erroring tears down the other — see RealtimeSession.close() in session.ts.

The extensibility seam

src/lib/server/core/provider.ts defines two interfaces:

interface RealtimeProvider {
	createSession(config: ResolvedSessionConfig): Promise<ProviderSession>;
}

interface ProviderSession {
	send(event: ClientEvent): void;
	onEvent(handler: (event: ServerEvent) => void): void;
	close(reason?: string): void;
}

The critical constraint: a ProviderSession speaks only the normalized ClientEvent/ServerEvent vocabulary from src/lib/core/protocol.ts — never a provider's own wire format. RealtimeSession (the orchestrator in session.ts) only ever talks to that interface, never to OpenAI directly.

To add a new provider or pipeline: write a new file under src/lib/server/core/providers/ implementing RealtimeProvider, translating between the normalized protocol and whatever that backend speaks (see providers/openai.ts for the reference implementation). Then wire it in wherever OpenAiRealtimeProvider is currently instantiated (server.js for production, dev-plugin.ts for dev). Nothing else changes — not session.ts, not ws-handler.ts, not any client code.

Resilience

src/lib/client/core/resilient-client.ts's ResilientRealtimeClient wraps RealtimeClient (composition, not inheritance — RealtimeClient itself stays untouched, still pure transport) with automatic reconnection on an unexpected transport drop. It exposes the same send()/on()/close() shape, plus a separate onConnectionState(handler) for transport-level state (connecting | open | reconnecting | closed) — kept out of the ServerEvent/protocol vocabulary deliberately, since that's the provider-relay contract, not the place for this transport's local retry bookkeeping.

Reconnecting the browser↔relay socket does not resurrect a dead relay↔OpenAI session. If the server proactively sends session.closed (upstream_disconnected, client_requested, etc.), that's a clean end — no auto-reconnect follows. Auto-reconnect only fires when the underlying socket closes with no preceding session.closed (a network blip, laptop sleep/wake, an idle timeout) — detected via RealtimeClient.onClose(), a small addition to client.ts alongside its existing on() (message events) for exactly this purpose, since without it there was no way to observe a drop that isn't itself a parsed ServerEvent.

Either way, a reconnect always starts a brand-new session — fresh token (getToken is called again; tokens are short-lived and a stale one can't be reused), fresh session.start. The model has no memory of the old session regardless, so there's no conversation state worth trying to resume. Callers see this plainly as a reconnecting → open transition followed by a fresh session.ready, and decide for themselves how to present it (clear the transcript, show a toast, etc.) — the wrapper doesn't hide it. See /demo/realtime for the reference wiring, including how it resets its own transcript on reconnecting.

Backoff is exponential (baseDelayMs * 2^(attempt-1)), capped at maxAttempts consecutive failures before giving up and reporting closed — both configurable per instance, no env var (this is browser-local behavior, not server config).

Development without a live API

src/lib/server/core/providers/mock.ts's MockRealtimeProvider implements the same RealtimeProvider interface as OpenAiRealtimeProvider (see "The extensibility seam" below), but replays a canned MockScript (Array<{ delayMs, event: ServerEvent }>, typed directly against ServerEvent so a fixture can't silently drift from the protocol) on a timer instead of talking to a real upstream. send() is a no-op — v1 is linear, one-shot replay, not interactive/branching scripting.

Select it with REALTIME_PROVIDER=mock (see CONFIGURATION.md); OPENAI_API_KEY isn't required in that mode. src/lib/server/core/providers/index.ts's selectProvider(config) is the one place both composition roots (server.js, dev-plugin.ts) call to pick between providers, instead of each hardcoding new OpenAiRealtimeProvider(...).

This exists mainly so client/animation work (transcript rendering, $lib/humanoid's avatar-state derivation — see HUMANOID.md) can happen without an API key or live usage. providers/fixtures/default.ts ships one short canned exchange for exercising exactly that; see /demo/humanoid.

The trust boundary

The browser's session.start message can only set voice (see SessionStartConfig in protocol.ts). instructions, tools, and model are always server-owned — sourced from the project's persona (src/lib/project/persona.ts, model optionally overridden by SESSION_MODEL; see docs/CONFIGURATION.md) via resolveSessionDefaults() (defaults.ts), and merged with the client-visible config in resolveSessionConfig() (session.ts). A server-side defaultVoice (SESSION_DEFAULT_VOICE) fills in when the browser's session.start doesn't specify a voice, but the browser's own choice still wins when it sends one — this doesn't loosen the trust boundary, since voice was already the one client-controllable field. This is deliberate: this relay is shared infrastructure that other people's browsers connect to, so a client project should never let an untrusted browser dictate what the model is told to do or which model/tools it can invoke. If a client project needs more client-controllable config, extend SessionStartConfig and resolveSessionConfig() explicitly and think about what's safe to expose — don't just pass the browser's config straight through.

SvelteKit WebSocket plumbing

SvelteKit has no native WebSocket support as of this writing (a crossws-based PR for it was closed by the maintainers in Feb 2026 in favor of an unrelated SSE feature). Raw HTTP upgrade requests have to be intercepted below SvelteKit's own routing, at the underlying http.Server level:

• Production: server.js wraps adapter-node's generated build/handler.js in a plain http.createServer and attaches upgrade handling manually.
• Dev: dev-plugin.ts is a Vite plugin that attaches the same handling to Vite's dev server via configureServer. It's careful to no-op for any path that isn't ours, since Vite's own HMR websocket shares the same underlying HTTP server — touching the socket for a non-matching path would silently break HMR.

Both call the same handleRealtimeUpgrade function, so prod and dev behave identically.

Why src/lib/core/ and src/lib/server/core/ avoid $lib/$env aliases: server.js is a plain Node entrypoint running the built app — it can't resolve SvelteKit's $lib/$env virtual modules, which only exist inside Vite's build graph. So this module tree uses plain relative imports and reads secrets via process.env instead of $env/static/private, which lets it be compiled standalone (tsc -p tsconfig.server.json → build-server/, run as part of npm run build) for server.js to import directly. SvelteKit routes (+server.ts, +page.svelte) are unaffected by this and use the normal $lib alias as usual.

Known gotcha: because this module tree is only loaded via vite.config.ts's plugin list (not through the app's normal module graph), it is not part of Vite's HMR. Editing anything under src/lib/server/core/** while npm run dev is running requires a full dev-server restart to take effect — a hot-reload will silently keep running the old code. If you change something here and testing doesn't reflect it, restart before assuming the change is broken.

Protocol vocabulary

See protocol.ts for the authoritative, current list — this is a summary, not a copy to keep in sync by hand.

• Client → server: session.start, input_audio.chunk, input_audio.commit, input_text.send, response.cancel, session.end
• Server → client: session.ready, output_audio.chunk, output_audio.done, transcript.delta, transcript.done, response.done, error, session.closed

OpenAI Realtime API gotcha (verified live, Aug 2026)

OpenAI's official docs were incomplete when this was built (WebFetch on developers.openai.com/api/reference/resources/realtime/* truncated before covering response streaming events), and general web search suggested event names that turned out to be stale. The actual, currently-live event names for output content carry an output_ prefix:

• response.output_audio.delta / response.output_audio.done — not response.audio.delta
• response.output_audio_transcript.delta / response.output_audio_transcript.done — not response.audio_transcript.delta

Response-lifecycle events (response.created, response.output_item.added/done, response.done) are unaffected. This is handled in fromOpenAiEvent. If OpenAI changes their event schema again, unrecognized events are silently dropped (by design — see default: return null in that function); if a response's content stops showing up in the browser, temporarily add a console.debug in handleUpstreamMessage's unmapped-event branch, restart the dev server, and read the real event names from preview_logs rather than re-guessing from docs.

Still unverified: the exact event name for a completed input audio transcription (conversation.item.input_audio_transcription.completed is the current guess, unconfirmed since testing so far only exercised text input).

What's deliberately not built yet

• Audio capture/playback in the browser client (client.ts is protocol-level only)
• Rate limiting on the token-mint endpoint
• Real user accounts (tokens.ts is anonymous-session-only)
• Multi-instance/shared session storage (in-memory, single-process only)
• Binary audio framing (currently JSON + base64, mirroring OpenAI's own wire format)

Verifying changes

• npm run check / npm run lint / npm run build for type, lint, and build correctness (build also runs the standalone tsc -p tsconfig.server.json compile — watch for errors there specifically, since svelte-check alone won't catch problems in code that isn't reached through the SvelteKit app).
• npm run test:unit — unit tests for tokens.ts, session.ts (state machine, against a fake in-memory provider), providers/openai.ts's translation functions, providers/mock.ts's script replay, resilient-client.ts's backoff/reconnect logic, and $lib/humanoid's reducer/compositor.
• Live smoke test: set OPENAI_API_KEY and REALTIME_TOKEN_SECRET in .env (see .env.example), run npm run dev, open /demo/realtime, and actually send a message. Type-checking and unit tests can't catch a wire-protocol mismatch like the one above — only a real round trip against the real API can. Remember the HMR gotcha above if you're iterating on src/lib/server/core/**.
• Mock-provider smoke test: REALTIME_PROVIDER=mock npm run dev, open /demo/humanoid, connect, and watch avatarState move through the default fixture — zero OpenAI usage, useful for iterating on animation timing without live traffic.