Document project architecture and the realtime session subsystem

Adds docs/ARCHITECTURE.md (module layering, foundational infra decisions,
subsystem index) and docs/REALTIME_SESSION.md (the WS relay design, the
provider extensibility seam, the trust boundary, SvelteKit's WS plumbing
workaround, and the OpenAI event-naming gotcha found via live testing).

Restructures AGENTS.md around orientation, conventions, and process, and
establishes documentation as part of the same change as the work itself
rather than a follow-up pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

AGENTS.md

@@ -1,6 +1,37 @@
-You are able to use the Svelte MCP server, where you have access to comprehensive Svelte 5 and SvelteKit documentation. Here's how to use the available tools effectively:
+# webavatar-core
 
-## Available Svelte MCP Tools:
+Base SvelteKit project meant to be branched off for future webavatar client projects. It provides reusable core infrastructure — session management, auth tokens, realtime model connections — so client projects don't reimplement the basics from scratch. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full picture; `docs/` holds per-subsystem deep-dives beyond that.
+
+## Project structure
+
+- `src/lib/core/` — shared client+server contracts (types, protocols). No SvelteKit-specific imports (`$env`, `$app`, etc.) — must stay portable.
+- `src/lib/server/core/` — server-only implementation. SvelteKit enforces this boundary at build time; browser code cannot import from here, even transitively.
+- `src/lib/client/core/` — thin browser-side wrappers around the shared contracts. Transport only, no business logic.
+- `src/routes/` — thin route glue; delegates into the above rather than holding logic itself.
+- `docs/` — architecture and subsystem documentation (see "Documentation" below).
+
+## Development
+
+- `npm run dev` / `npm run build` / `npm run preview` — standard SvelteKit commands. `build` also compiles `src/lib/core/**` + `src/lib/server/core/**` standalone (`tsc -p tsconfig.server.json`) for production's custom `server.js` to import — see [docs/REALTIME_SESSION.md](docs/REALTIME_SESSION.md) for why that's needed.
+- `npm run check` — type-check (`svelte-check`). `npm run lint` / `npm run format` — prettier + eslint. `npm run test:unit` — vitest.
+- Copy `.env.example` to `.env` and fill in `OPENAI_API_KEY` / `REALTIME_TOKEN_SECRET` before running anything that touches the realtime session layer.
+- **Gotcha**: `src/lib/server/core/**` is only loaded via `vite.config.ts`'s plugin list, not through the app's normal module graph — so it's **not** part of Vite's HMR. Restart `npm run dev` after editing anything there; a hot-reload will silently keep running the old code, which reads as a mysterious bug if you don't know this.
+
+## Documentation
+
+This project is documented for both humans and agents, and keeping it that way is part of the development process, not a follow-up task:
+
+- **Architecture and subsystem docs live in `docs/`.** [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) is the high-level map, with a table linking to per-subsystem deep-dives (e.g. [`docs/REALTIME_SESSION.md`](docs/REALTIME_SESSION.md)). When you build a new subsystem, or make a decision that isn't obvious from reading the code (a tradeoff, a "we chose X over Y because Z", a constraint that shapes how client projects can deploy or extend this), add or update the relevant doc under `docs/` and link it from `docs/ARCHITECTURE.md`'s table — as part of the same change, not a separate pass later.
+- **This file stays focused on orientation, conventions, and process — not deep architecture.** If you're about to write a paragraph explaining _why_ something is built a certain way, that belongs in a `docs/` file; link to it from here instead of inlining it.
+- **Write down what was hard to find out, not what's easy to re-derive.** Decision rationale and gotchas discovered the hard way (like the OpenAI event-naming mismatch documented in `docs/REALTIME_SESSION.md`, only found by testing live) are worth the words. Code structure, file layout, or anything obvious from reading the source usually isn't — don't document what the code already says clearly.
+
+## Verifying changes
+
+Type-checking and unit tests don't prove a feature actually works, especially anything touching an external API's wire protocol — assumptions baked into the implementation end up baked into tests of that implementation too, so they can pass while the feature is genuinely broken. For UI changes, use the feature in a real browser before calling it done. For the realtime session layer specifically, that means an actual round trip against OpenAI via `/demo/realtime`, not just `npm run check` passing — see "Verifying changes" in [docs/REALTIME_SESSION.md](docs/REALTIME_SESSION.md).
+
+## Svelte MCP server
+
+You have access to the Svelte MCP server, with comprehensive Svelte 5 and SvelteKit documentation. Use the available tools as follows:
 
 ### 1. list-sections
 

docs/ARCHITECTURE.md

@@ -0,0 +1,26 @@
+# Architecture
+
+webavatar-core is a base SvelteKit project meant to be branched off for future webavatar client projects. Its job is to provide the core infrastructure those projects would otherwise reimplement from scratch — session management, auth tokens, and the realtime model connection — as a clean, extensible foundation rather than a finished product.
+
+## Module layering
+
+Code that's part of this core infrastructure lives under `src/lib/` in three places, split by where it's allowed to run:
+
+- **`src/lib/core/`** — shared, isomorphic contracts (types, protocols, constants). Importable by both browser and server code. Must stay free of SvelteKit-specific magic (no `$env/*`, no server-only imports) so it can also be compiled standalone outside Vite when needed (see [REALTIME_SESSION.md](REALTIME_SESSION.md) for why that matters).
+- **`src/lib/server/core/`** — server-only implementations: session orchestration, provider adapters, token minting/verification, raw HTTP upgrade handling. SvelteKit enforces this boundary at build time — browser code cannot import from here, even transitively.
+- **`src/lib/client/core/`** — thin browser-side wrappers around the shared contracts. No business logic; just transport.
+
+Routes (`src/routes/`) stay thin and delegate into these — a route file is glue, not where logic lives.
+
+## Foundational infrastructure decisions
+
+- **This app runs as a persistent Node process, not serverless.** The adapter is `@sveltejs/adapter-node` — configured inline in [`vite.config.ts`](../vite.config.ts)'s `sveltekit({ adapter: ... })` option, since this project has no separate `svelte.config.js` — started via a custom [`server.js`](../server.js) rather than adapter-node's default generated entrypoint. This is a hard requirement, not a preference: the realtime session relay holds a WebSocket connection to a browser and a second WebSocket connection to the model provider simultaneously, for the lifetime of a conversation. No serverless/edge adapter can hold connections like that. Any client project branching from this base inherits that constraint — plan hosting accordingly.
+- **Auth tokens are generic infrastructure, not tied to any one feature.** `src/lib/server/core/tokens.ts` mints/verifies short-lived HMAC-signed tokens as a reusable primitive, currently used to gate the realtime WebSocket handshake but intentionally not coupled to it.
+
+## Subsystems
+
+| Subsystem                                                                                     | Doc                                        |
+| --------------------------------------------------------------------------------------------- | ------------------------------------------ |
+| Realtime session management (OpenAI Realtime API today, extensible to other models/pipelines) | [REALTIME_SESSION.md](REALTIME_SESSION.md) |
+
+Add a row here (and a corresponding doc) for each new subsystem as it's built — see the "Documentation" section in [AGENTS.md](../AGENTS.md) for when and how.

docs/REALTIME_SESSION.md

@@ -0,0 +1,114 @@
+# Realtime session management
+
+Provider-agnostic session layer for realtime, voice-first conversations. Backed today by a single provider — the OpenAI Realtime API — but designed so a second provider or a multi-stage pipeline (STT → LLM → TTS) can be added later without touching the session orchestrator, the WebSocket plumbing, or any client code.
+
+## Why a WebSocket relay, not WebRTC-direct
+
+The connection is **browser ↔ our SvelteKit server ↔ the model provider**, not browser-direct-to-provider. Concretely: the browser holds one WebSocket to our server; our server holds a second WebSocket to OpenAI; our server relays between them.
+
+This was a deliberate choice, made explicitly rather than defaulted into. OpenAI's own recommended pattern for browser clients is the opposite: the browser holds a WebRTC connection **directly** to OpenAI, and the backend's only job is minting short-lived ephemeral tokens — lower latency, no audio bandwidth through your own server.
+
+We chose the relay instead because:
+
+- **One uniform transport.** Both "talk to OpenAI Realtime" today and "run a multi-stage pipeline" later look identical from the browser's side — a WebSocket to our server speaking our own protocol. A WebRTC-direct approach would need an entirely different client-side transport for a non-realtime pipeline (e.g. SSE), unified only by a shared interface at best. Here, the provider swap happens entirely server-side, so the browser-facing contract never has to change.
+- **A server-side hook point.** Because every event passes through our server, we can add moderation, analytics, avatar-specific processing (e.g. viseme/lipsync generation from audio output), or business logic in the middle later — without ever touching the browser client.
+- **No ephemeral-token dance.** Since the browser never talks to the provider directly, our server just uses the real `OPENAI_API_KEY` server-to-server. No `client_secrets` minting, no token refresh handling in the browser.
+
+The accepted tradeoff: an extra network hop, and our server carries audio bandwidth in both directions instead of peer-to-peer. For a base project optimizing for extensibility and a clean client-facing contract over shaving off milliseconds, that's the right tradeoff — but it's worth knowing this was a real fork in the road, not an oversight, if you're ever deciding whether to add a WebRTC-direct transport alongside this one.
+
+## Connection flow
+
+```mermaid
+sequenceDiagram
+    participant B as Browser
+    participant S as SvelteKit server
+    participant O as OpenAI Realtime API
+
+    B->>S: POST /api/realtime/session
+    S-->>B: { token, expiresAt }
+    B->>S: WS upgrade wss://.../api/realtime/ws?token=...
+    S->>S: verify token (tokens.ts)
+    S-->>B: 101 Switching Protocols
+    B->>S: { type: "session.start", config? }
+    S->>O: wss://api.openai.com/v1/realtime?model=...
+    S->>O: { type: "session.update", session: {...} }
+    O-->>S: session.created / session.updated
+    S-->>B: { type: "session.ready", sessionId }
+    B->>S: input_text.send / input_audio.chunk / ...
+    S->>O: translated OpenAI wire events
+    O-->>S: response.output_audio.delta / response.output_audio_transcript.delta / ...
+    S-->>B: output_audio.chunk / transcript.delta / ...
+```
+
+Either side disconnecting or erroring tears down the other — see `RealtimeSession.close()` in [`session.ts`](../src/lib/server/core/session.ts).
+
+## The extensibility seam
+
+`src/lib/server/core/provider.ts` defines two interfaces:
+
+```ts
+interface RealtimeProvider {
+	createSession(config: ResolvedSessionConfig): Promise<ProviderSession>;
+}
+
+interface ProviderSession {
+	send(event: ClientEvent): void;
+	onEvent(handler: (event: ServerEvent) => void): void;
+	close(reason?: string): void;
+}
+```
+
+The critical constraint: a `ProviderSession` speaks **only** the normalized `ClientEvent`/`ServerEvent` vocabulary from [`src/lib/core/protocol.ts`](../src/lib/core/protocol.ts) — never a provider's own wire format. `RealtimeSession` (the orchestrator in `session.ts`) only ever talks to that interface, never to OpenAI directly.
+
+**To add a new provider or pipeline:** write a new file under `src/lib/server/core/providers/` implementing `RealtimeProvider`, translating between the normalized protocol and whatever that backend speaks (see `providers/openai.ts` for the reference implementation). Then wire it in wherever `OpenAiRealtimeProvider` is currently instantiated ([`server.js`](../server.js) for production, [`dev-plugin.ts`](../src/lib/server/core/dev-plugin.ts) for dev). Nothing else changes — not `session.ts`, not `ws-handler.ts`, not any client code.
+
+## The trust boundary
+
+The browser's `session.start` message can only set `voice` (see `SessionStartConfig` in `protocol.ts`). `instructions`, `tools`, and `model` are always server-owned — set in [`defaults.ts`](../src/lib/server/core/defaults.ts) and merged in `resolveSessionConfig()` (`session.ts`). This is deliberate: this relay is shared infrastructure that other people's browsers connect to, so a client project should never let an untrusted browser dictate what the model is told to do or which model/tools it can invoke. If a client project needs more client-controllable config, extend `SessionStartConfig` and `resolveSessionConfig()` explicitly and think about what's safe to expose — don't just pass the browser's config straight through.
+
+## SvelteKit WebSocket plumbing
+
+SvelteKit has no native WebSocket support as of this writing (a `crossws`-based PR for it was closed by the maintainers in Feb 2026 in favor of an unrelated SSE feature). Raw HTTP upgrade requests have to be intercepted below SvelteKit's own routing, at the underlying `http.Server` level:
+
+- **Production**: [`server.js`](../server.js) wraps adapter-node's generated `build/handler.js` in a plain `http.createServer` and attaches upgrade handling manually.
+- **Dev**: [`dev-plugin.ts`](../src/lib/server/core/dev-plugin.ts) is a Vite plugin that attaches the same handling to Vite's dev server via `configureServer`. It's careful to no-op for any path that isn't ours, since Vite's own HMR websocket shares the same underlying HTTP server — touching the socket for a non-matching path would silently break HMR.
+
+Both call the same [`handleRealtimeUpgrade`](../src/lib/server/core/ws-handler.ts) function, so prod and dev behave identically.
+
+**Why `src/lib/core/` and `src/lib/server/core/` avoid `$lib`/`$env` aliases:** `server.js` is a plain Node entrypoint running the _built_ app — it can't resolve SvelteKit's `$lib`/`$env` virtual modules, which only exist inside Vite's build graph. So this module tree uses plain relative imports and reads secrets via `process.env` instead of `$env/static/private`, which lets it be compiled standalone (`tsc -p tsconfig.server.json` → `build-server/`, run as part of `npm run build`) for `server.js` to import directly. SvelteKit _routes_ (`+server.ts`, `+page.svelte`) are unaffected by this and use the normal `$lib` alias as usual.
+
+**Known gotcha:** because this module tree is only loaded via `vite.config.ts`'s plugin list (not through the app's normal module graph), it is **not** part of Vite's HMR. Editing anything under `src/lib/server/core/**` while `npm run dev` is running requires a full dev-server restart to take effect — a hot-reload will silently keep running the old code. If you change something here and testing doesn't reflect it, restart before assuming the change is broken.
+
+## Protocol vocabulary
+
+See [`protocol.ts`](../src/lib/core/protocol.ts) for the authoritative, current list — this is a summary, not a copy to keep in sync by hand.
+
+- **Client → server**: `session.start`, `input_audio.chunk`, `input_audio.commit`, `input_text.send`, `response.cancel`, `session.end`
+- **Server → client**: `session.ready`, `output_audio.chunk`, `output_audio.done`, `transcript.delta`, `transcript.done`, `response.done`, `error`, `session.closed`
+
+## OpenAI Realtime API gotcha (verified live, Aug 2026)
+
+OpenAI's official docs were incomplete when this was built (WebFetch on `developers.openai.com/api/reference/resources/realtime/*` truncated before covering response streaming events), and general web search suggested event names that turned out to be stale. The actual, currently-live event names for output content carry an `output_` prefix:
+
+- `response.output_audio.delta` / `response.output_audio.done` — not `response.audio.delta`
+- `response.output_audio_transcript.delta` / `response.output_audio_transcript.done` — not `response.audio_transcript.delta`
+
+Response-lifecycle events (`response.created`, `response.output_item.added/done`, `response.done`) are unaffected. This is handled in [`fromOpenAiEvent`](../src/lib/server/core/providers/openai.ts). If OpenAI changes their event schema again, unrecognized events are silently dropped (by design — see `default: return null` in that function); if a response's content stops showing up in the browser, temporarily add a `console.debug` in `handleUpstreamMessage`'s unmapped-event branch, restart the dev server, and read the real event names from `preview_logs` rather than re-guessing from docs.
+
+Still unverified: the exact event name for a completed **input** audio transcription (`conversation.item.input_audio_transcription.completed` is the current guess, unconfirmed since testing so far only exercised text input).
+
+## What's deliberately not built yet
+
+- A second provider or pipeline implementation (the point of the design above is that it's addable later without a rewrite)
+- Audio capture/playback in the browser client (`client.ts` is protocol-level only)
+- Reconnect/retry handling
+- Rate limiting on the token-mint endpoint
+- Real user accounts (`tokens.ts` is anonymous-session-only)
+- Multi-instance/shared session storage (in-memory, single-process only)
+- Binary audio framing (currently JSON + base64, mirroring OpenAI's own wire format)
+
+## Verifying changes
+
+- `npm run check` / `npm run lint` / `npm run build` for type, lint, and build correctness (`build` also runs the standalone `tsc -p tsconfig.server.json` compile — watch for errors there specifically, since `svelte-check` alone won't catch problems in code that isn't reached through the SvelteKit app).
+- `npm run test:unit` — unit tests for `tokens.ts`, `session.ts` (state machine, against a fake in-memory provider), and `providers/openai.ts`'s translation functions.
+- **Live smoke test**: set `OPENAI_API_KEY` and `REALTIME_TOKEN_SECRET` in `.env` (see `.env.example`), run `npm run dev`, open `/demo/realtime`, and actually send a message. Type-checking and unit tests can't catch a wire-protocol mismatch like the one above — only a real round trip against the real API can. Remember the HMR gotcha above if you're iterating on `src/lib/server/core/**`.