webavatar-core/docs/REALTIME_SESSION.md

Realtime session management

Provider-agnostic session layer for realtime, voice-first conversations. Backed today by a single provider — the OpenAI Realtime API — but designed so a second provider or a multi-stage pipeline (STT → LLM → TTS) can be added later without touching the session orchestrator, the WebSocket plumbing, or any client code.

Why a WebSocket relay, not WebRTC-direct

The connection is browser ↔ our SvelteKit server ↔ the model provider, not browser-direct-to-provider. Concretely: the browser holds one WebSocket to our server; our server holds a second WebSocket to OpenAI; our server relays between them.

This was a deliberate choice, made explicitly rather than defaulted into. OpenAI's own recommended pattern for browser clients is the opposite: the browser holds a WebRTC connection directly to OpenAI, and the backend's only job is minting short-lived ephemeral tokens — lower latency, no audio bandwidth through your own server.

We chose the relay instead because:

• One uniform transport. Both "talk to OpenAI Realtime" today and "run a multi-stage pipeline" later look identical from the browser's side — a WebSocket to our server speaking our own protocol. A WebRTC-direct approach would need an entirely different client-side transport for a non-realtime pipeline (e.g. SSE), unified only by a shared interface at best. Here, the provider swap happens entirely server-side, so the browser-facing contract never has to change.
• A server-side hook point. Because every event passes through our server, we can add moderation, analytics, avatar-specific processing (e.g. viseme/lipsync generation from audio output), or business logic in the middle later — without ever touching the browser client.
• No ephemeral-token dance. Since the browser never talks to the provider directly, our server just uses the real OPENAI_API_KEY server-to-server. No client_secrets minting, no token refresh handling in the browser.

The accepted tradeoff: an extra network hop, and our server carries audio bandwidth in both directions instead of peer-to-peer. For a base project optimizing for extensibility and a clean client-facing contract over shaving off milliseconds, that's the right tradeoff — but it's worth knowing this was a real fork in the road, not an oversight, if you're ever deciding whether to add a WebRTC-direct transport alongside this one.

Connection flow

sequenceDiagram
    participant B as Browser
    participant S as SvelteKit server
    participant O as OpenAI Realtime API

    B->>S: POST /api/realtime/session
    S-->>B: { token, expiresAt }
    B->>S: WS upgrade wss://.../api/realtime/ws?token=...
    S->>S: verify token (tokens.ts)
    S-->>B: 101 Switching Protocols
    B->>S: { type: "session.start", config? }
    S->>O: wss://api.openai.com/v1/realtime?model=...
    S->>O: { type: "session.update", session: {...} }
    O-->>S: session.created / session.updated
    S-->>B: { type: "session.ready", sessionId }
    B->>S: input_text.send / input_audio.chunk / ...
    S->>O: translated OpenAI wire events
    O-->>S: response.output_audio.delta / response.output_audio_transcript.delta / ...
    S-->>B: output_audio.chunk / transcript.delta / ...

Either side disconnecting or erroring tears down the other — see RealtimeSession.close() in session.ts.

The extensibility seam

packages/core/src/server/provider.ts defines two interfaces:

interface RealtimeProvider {
	createSession(config: ResolvedSessionConfig): Promise<ProviderSession>;
}

interface ProviderSession {
	send(event: ClientEvent): void;
	onEvent(handler: (event: ServerEvent) => void): void;
	close(reason?: string): void;
}

The critical constraint: a ProviderSession speaks only the normalized ClientEvent/ServerEvent vocabulary from protocol.ts — never a provider's own wire format. RealtimeSession (the orchestrator in session.ts) only ever talks to that interface, never to OpenAI directly.

To add a new provider or pipeline: write a new file under packages/core/src/server/providers/ implementing RealtimeProvider, translating between the normalized protocol and whatever that backend speaks (see providers/openai.ts for the reference implementation). Then wire it into selectProvider (providers/index.ts) — the one place both composition roots (apps/template/server.js for production, the ./vite plugin for dev) resolve a provider through, via the shared createRealtimeRuntime factory (see ARCHITECTURE.md's "Composition roots"). Nothing else changes — not session.ts, not ws-handler.ts, not any client code.

Resilience

packages/core/src/client/resilient-client.ts's ResilientRealtimeClient wraps RealtimeClient (composition, not inheritance — RealtimeClient itself stays untouched, still pure transport) with automatic reconnection on an unexpected transport drop. It exposes the same send()/on()/close() shape, plus a separate onConnectionState(handler) for transport-level state (connecting | open | reconnecting | closed) — kept out of the ServerEvent/protocol vocabulary deliberately, since that's the provider-relay contract, not the place for this transport's local retry bookkeeping.

Reconnecting the browser↔relay socket does not resurrect a dead relay↔OpenAI session. If the server proactively sends session.closed (upstream_disconnected, client_requested, etc.), that's a clean end — no auto-reconnect follows. Auto-reconnect only fires when the underlying socket closes with no preceding session.closed (a network blip, laptop sleep/wake, an idle timeout) — detected via RealtimeClient.onClose(), a small addition to client.ts alongside its existing on() (message events) for exactly this purpose, since without it there was no way to observe a drop that isn't itself a parsed ServerEvent.

Either way, a reconnect always starts a brand-new session — fresh token (getToken is called again; tokens are short-lived and a stale one can't be reused), fresh session.start. The model has no memory of the old session regardless, so there's no conversation state worth trying to resume. Callers see this plainly as a reconnecting → open transition followed by a fresh session.ready, and decide for themselves how to present it (clear the transcript, show a toast, etc.) — the wrapper doesn't hide it. See /demo/realtime for the reference wiring, including how it resets its own transcript on reconnecting.

Backoff is exponential (baseDelayMs * 2^(attempt-1)), capped at maxAttempts consecutive failures before giving up and reporting closed — both configurable per instance, no env var (this is browser-local behavior, not server config).

Development without a live API

packages/core/src/server/providers/mock.ts's MockRealtimeProvider implements the same RealtimeProvider interface as OpenAiRealtimeProvider (see "The extensibility seam" below), but replays a canned MockScript (Array<{ delayMs, event: ServerEvent }>, typed directly against ServerEvent so a fixture can't silently drift from the protocol) on a timer instead of talking to a real upstream. send() is a no-op — v1 is linear, one-shot replay, not interactive/branching scripting.

Select it with REALTIME_PROVIDER=mock (see CONFIGURATION.md); OPENAI_API_KEY isn't required in that mode. providers/index.ts's selectProvider(config) is the one place createRealtimeRuntime calls to pick between providers, instead of either composition root hardcoding new OpenAiRealtimeProvider(...).

This exists mainly so client/animation work (transcript rendering, @webavatar/webavatar-humanoid's avatar-state derivation — see HUMANOID.md) can happen without an API key or live usage. providers/fixtures/default.ts ships one short canned exchange for exercising exactly that; see /demo/humanoid.

The trust boundary

The browser's session.start message can only set voice (see SessionStartConfig in protocol.ts). instructions, tools, and model are always server-owned — sourced from the project's persona (apps/template/src/lib/project/persona.ts, model optionally overridden by SESSION_MODEL; see docs/CONFIGURATION.md) via resolveSessionDefaults() (defaults.ts), and merged with the client-visible config in resolveSessionConfig() (session.ts). A server-side defaultVoice (SESSION_DEFAULT_VOICE) fills in when the browser's session.start doesn't specify a voice, but the browser's own choice still wins when it sends one — this doesn't loosen the trust boundary, since voice was already the one client-controllable field. This is deliberate: this relay is shared infrastructure that other people's browsers connect to, so a client project should never let an untrusted browser dictate what the model is told to do or which model/tools it can invoke. If a client project needs more client-controllable config, extend SessionStartConfig and resolveSessionConfig() explicitly and think about what's safe to expose — don't just pass the browser's config straight through.

SvelteKit WebSocket plumbing

SvelteKit has no native WebSocket support as of this writing (a crossws-based PR for it was closed by the maintainers in Feb 2026 in favor of an unrelated SSE feature). Raw HTTP upgrade requests have to be intercepted below SvelteKit's own routing, at the underlying http.Server level:

• Production: apps/template/server.js wraps adapter-node's generated build/handler.js in a plain http.createServer, calls createRealtimeRuntime({ ..., isDev: false }), and attaches runtime.handleUpgrade to the 'upgrade' event.
• Dev: @webavatar/webavatar-core/vite's realtimeDevWsPlugin (packages/core/src/vite/index.ts), used in apps/template/vite.config.ts, is a Vite plugin that calls the same createRealtimeRuntime({ ..., isDev: true }) and attaches runtime.handleUpgrade to Vite's dev server via configureServer. It's careful to no-op for any path that isn't ours, since Vite's own HMR websocket shares the same underlying HTTP server — touching the socket for a non-matching path would silently break HMR.

Both go through the same handleRealtimeUpgrade function inside createRealtimeRuntime's handleUpgrade closure, so prod and dev behave identically — see ARCHITECTURE.md's "Composition roots" for why this collapsed into one factory.

Why core avoids $lib/$env aliases: @webavatar/webavatar-core's server code uses plain relative imports and reads secrets via process.env instead of SvelteKit's $env/static/private, so it stays a portable package compiled by its own tsc build rather than depending on Vite's build graph — server.js (a plain Node entrypoint running the built app) imports it directly, with no SvelteKit virtual modules involved. SvelteKit routes (+server.ts, +page.svelte) are unaffected by this and use the normal $lib alias as usual, going through apps/template/src/lib/server/webavatar-core.ts.

Dev-mode reload behavior (verified live during the monorepo migration, superseding the old single-gotcha warning):

Path	After editing package source, with the package's tsc --watch running
Browser code importing @webavatar/webavatar-core/client or @webavatar/webavatar-humanoid (e.g. resilient-client.ts, avatar-state.ts)	Hot reload. Once tsc --watch recompiles dist/, Vite detects the change and hot-updates the browser module graph — confirmed via a live edit to client/logger.ts propagating to /demo/realtime without a page reload.
SSR route code going through apps/template/src/lib/server/webavatar-core.ts (e.g. the session-token route)	Hot reload, for the same reason — confirmed via a live edit to tokens.ts's mintSessionToken showing up on the very next POST /api/realtime/session, with no server restart.
The realtime WebSocket relay wired by @webavatar/webavatar-core/vite's realtimeDevWsPlugin (used in apps/template/vite.config.ts)	Restart required. vite.config.ts's plugin list is evaluated once when the dev server starts; a new WS connection after editing the plugin's own composition still ran the stale, already-registered upgrade handler — confirmed live, the edit only took effect after restarting npm run dev.

The tsc --watch precondition above matters: apps/template's own npm run dev in isolation does not recompile the packages, so package-source edits appear to do nothing at all unless you're running the root npm run dev (which orchestrates watch -w packages/core, watch -w packages/humanoid, and the app's dev server together) or a package's npm run watch separately.

Protocol vocabulary

See protocol.ts for the authoritative, current list — this is a summary, not a copy to keep in sync by hand.

• Client → server: session.start, input_audio.chunk, input_audio.commit, input_text.send, response.cancel, session.end, debug.subscribe
• Server → client: session.ready, output_audio.chunk, output_audio.done, transcript.delta, transcript.done, response.done, error, session.closed, debug.subscribe.result, debug.log

Parsed browser JSON is runtime-validated against the client-event contract before it reaches analytics or a provider adapter; the TypeScript union alone is not a wire-level guarantee. Missing or wrongly typed required fields are rejected with invalid_client_event, while an unexpected throw in the async message path closes only that connection with internal_error. This containment is process-critical: one Node process serves every realtime session.

debug.subscribe/debug.subscribe.result/debug.log are a separate concern from the rest of this vocabulary — ephemeral developer debug-log forwarding, not session content. See LOGGING.md.

OpenAI Realtime API gotcha (verified live, Aug 2026)

OpenAI's official docs were incomplete when this was built (WebFetch on developers.openai.com/api/reference/resources/realtime/* truncated before covering response streaming events), and general web search suggested event names that turned out to be stale. The actual, currently-live event names for output content carry an output_ prefix:

• response.output_audio.delta / response.output_audio.done — not response.audio.delta
• response.output_audio_transcript.delta / response.output_audio_transcript.done — not response.audio_transcript.delta

Response-lifecycle events (response.created, response.output_item.added/done, response.done) are unaffected. This is handled in fromOpenAiEvent. If OpenAI changes their event schema again, unrecognized events are silently dropped (by design — see default: return null in that function); if a response's content stops showing up in the browser, temporarily add a console.debug in handleUpstreamMessage's unmapped-event branch, restart the dev server, and read the real event names from preview_logs rather than re-guessing from docs.

Still unverified: the exact event name for a completed input audio transcription (conversation.item.input_audio_transcription.completed is the current guess, unconfirmed since testing so far only exercised text input).

What's deliberately not built yet

• Audio capture/playback in the browser client (client.ts is protocol-level only)
• Rate limiting on the token-mint endpoint
• Real user accounts (tokens.ts is anonymous-session-only)
• Multi-instance/shared session storage (in-memory, single-process only)
• Binary audio framing (currently JSON + base64, mirroring OpenAI's own wire format)

Verifying changes

• npm run check / npm run lint / npm run build for type, lint, and build correctness (build also runs the standalone tsc -p tsconfig.server.json compile — watch for errors there specifically, since svelte-check alone won't catch problems in code that isn't reached through the SvelteKit app).
• npm run test:unit — unit tests for tokens.ts, session.ts (state machine, against a fake in-memory provider), providers/openai.ts's translation functions, providers/mock.ts's script replay, resilient-client.ts's backoff/reconnect logic, and @webavatar/webavatar-humanoid's reducer/compositor.
• Live smoke test: set OPENAI_API_KEY and REALTIME_TOKEN_SECRET in .env (see .env.example), run npm run dev from the repo root, open /demo/realtime, and actually send a message. Type-checking and unit tests can't catch a wire-protocol mismatch like the one above — only a real round trip against the real API can. Remember the "Dev-mode reload behavior" table above if you're iterating on packages/core/src/server/** — you need the root npm run dev, not apps/template's own, for changes there to take effect at all.
• Mock-provider smoke test: REALTIME_PROVIDER=mock npm run dev, open /demo/humanoid, connect, and watch avatarState move through the default fixture — zero OpenAI usage, useful for iterating on animation timing without live traffic.