Browse Source
Wraps RealtimeClient with reconnect-on-drop (ResilientRealtimeClient), adds a MockRealtimeProvider so client/animation work doesn't need a live OpenAI key, and introduces "optional package" as a third module-layering kind alongside core/project — humanoid is the first instance, providing a renderer-agnostic avatar-state reducer and ARKit blend-shape animation compositor that future eye look-at/blink/lip-sync libraries can build on without forcing that surface onto non-humanoid (e.g. 2D) projects. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>main
26 changed files with 1329 additions and 48 deletions
@ -0,0 +1,59 @@ |
|||
# Humanoid package |
|||
|
|||
`humanoid` is the first instance of the **optional package** convention (see "Module layering" in [ARCHITECTURE.md](ARCHITECTURE.md)) — a renderer-agnostic contract layer for 3D humanoid avatars rigged with ARKit blend shapes, scoped out of `core` because a 2D or non-humanoid project has no use for it. Nothing in `core` or `project` imports from it; a project opts in by importing directly from `$lib/humanoid` / `$lib/client/humanoid`, and if nothing imports it, it's simply not part of that project's bundle. |
|||
|
|||
This package ships the **contract** future animation libraries (eye look-at, blink, lip-sync, body/gesture) plug into — not the libraries themselves, and not a renderer. It has no idea what's rendering the avatar (Three.js, Babylon, a game engine WebGL bridge, anything else) — that stays entirely project-owned, per the base project's founding constraint that appearance/UI is never core's job. |
|||
|
|||
## `$lib/humanoid/state.ts` — avatar state |
|||
|
|||
```ts |
|||
type AvatarState = 'idle' | 'listening' | 'thinking' | 'talking'; |
|||
``` |
|||
|
|||
Deliberately just four states, not the `IdleActive`/`IdlePassive` split a humanoid rig's own animation design (e.g. a future eye look-at library) may eventually want — that distinction needs a user-presence signal (webcam/face-tracking) this package has no opinion on. A project with that signal can layer its own finer idle split on top; `idle` here just means "no signal that the user or the model is currently doing anything." |
|||
|
|||
`nextAvatarState(current, signal)` is a pure reducer — no WS/browser dependency, directly unit-tested in `state.spec.ts`. The signal set is small and closed (`listening_started`/`listening_stopped`, `response_requested`, `talking_started`/`talking_stopped`, `reset`). |
|||
|
|||
Note `listening_stopped` moves to `thinking`, not `idle` — ending listening almost always means a turn just completed and a response is expected next, so the avatar shouldn't visibly relax for the split second before the model's response starts arriving. |
|||
|
|||
## `$lib/client/humanoid/avatar-state.ts` — deriving it live |
|||
|
|||
`deriveAvatarState(source)` takes a `RealtimeClient` or `ResilientRealtimeClient` (accepted structurally, via an `AvatarStateSource` interface — this package doesn't care which transport variant is in use) and folds its `ServerEvent`s through `nextAvatarState`: |
|||
|
|||
| `ServerEvent` | Signal | |
|||
| ------------------------------------- | ----------------- | |
|||
| `session.ready` / `session.closed` | `reset` | |
|||
| `output_audio.chunk` | `talking_started` | |
|||
| `transcript.delta` (role: assistant) | `talking_started` | |
|||
| `output_audio.done` / `response.done` | `talking_stopped` | |
|||
| `transcript.done` (role: assistant) | `talking_stopped` | |
|||
|
|||
Two signals can't come from the event stream alone, so the returned store exposes them imperatively: |
|||
|
|||
- **`setListening(active)`** — call from whatever mic-capture code a project uses. That pipeline doesn't exist in `core` yet (see "Known limitations" below), and the client is the only thing that currently knows when the user starts/stops talking: the protocol doesn't surface OpenAI's `input_audio_buffer.speech_*` VAD events — see the "not part of our normalized vocabulary" comment in `server/core/providers/openai.ts`'s `fromOpenAiEvent`. |
|||
- **`notifyResponseRequested()`** — call right when sending a typed `input_text.send` (the voice path already gets this for free from `setListening(false)`). |
|||
|
|||
The returned `AvatarStateStore.subscribe()` follows the Svelte store contract (calls `run` immediately with the current value, then on every change), so it's usable directly as `$store` in a `.svelte` file without a hard Svelte dependency in this package itself. See `/demo/humanoid` (`src/routes/demo/humanoid/+page.svelte`) for the reference wiring, including the manual "listening" toggle standing in for real mic-capture UI. |
|||
|
|||
### Known limitations |
|||
|
|||
- **`thinking` is a heuristic**, not a precise reflection of server-side activity — the time between a request and the assistant's first output. The browser never sees server-executed tool calls at all (see [TOOLS.md](TOOLS.md)), so "the model is running a tool" isn't observable from here. |
|||
- **No presence-based idle split** (see above) — a project with webcam/face-tracking data can extend on top. |
|||
|
|||
## `$lib/humanoid/animation.ts` — animation-channel composition |
|||
|
|||
```ts |
|||
type ARKitBlendShape = 'jawOpen' | 'eyeBlinkLeft' | /* ...all 52 ARFaceAnchor.BlendShapeLocation names */; |
|||
type AnimationFrame = Partial<Record<ARKitBlendShape, number>>; // sparse, 0–1 weights |
|||
``` |
|||
|
|||
`compositeAnimationFrames(layers)` folds an ordered list of `AnimationLayer`s (`{ frame, mode: 'add' | 'override', weight? }`) into one `AnimationFrame`, clamping each channel to `[0, 1]`. This generalizes the eye look-at design doc's own additive-layers pattern — Macro, Micro, and Nano gaze offsets summed into one effective offset — to arbitrary channels, plus `'override'` for layers whose values can't be meaningfully summed (a blink closing an eyelid a gaze layer also touches, for instance). |
|||
|
|||
This is pure, isomorphic, and has no idea how the resulting `AnimationFrame` gets applied to a mesh — that binding (which bone/morph target each `ARKitBlendShape` name maps to in a given rig) is entirely the project's job. |
|||
|
|||
## What's not built here (on purpose) |
|||
|
|||
- The eye look-at, blink, lip-sync, and body/gesture libraries themselves — future work, built as consumers of this contract, not part of it. |
|||
- Mic-capture/audio-playback in the browser (still open — see `docs/REALTIME_SESSION.md`'s "What's deliberately not built yet"). |
|||
- Server-side viseme/phoneme generation — an open design fork (server-side, since the relay already touches raw audio, vs. client-side from the playback buffer) to decide when the lip-sync library actually gets built. |
|||
- Any protocol change (`$lib/core/protocol.ts` is untouched by this package) — `AvatarState` derivation works entirely from events the protocol already carries. |
|||
@ -0,0 +1,176 @@ |
|||
import { describe, expect, it, vi } from 'vitest'; |
|||
import type { ClientEvent, ServerEvent } from '../../core/protocol.js'; |
|||
import { ResilientRealtimeClient, type RealtimeTransport } from './resilient-client.js'; |
|||
|
|||
/** Minimal stand-in for RealtimeClient — only what ResilientRealtimeClient touches. */ |
|||
class FakeTransport implements RealtimeTransport { |
|||
sent: ClientEvent[] = []; |
|||
closed = false; |
|||
connectCalls: string[] = []; |
|||
connectResult: 'resolve' | 'reject' = 'resolve'; |
|||
private eventHandler: ((event: ServerEvent) => void) | null = null; |
|||
private closeHandler: (() => void) | null = null; |
|||
|
|||
async connect(token: string): Promise<void> { |
|||
this.connectCalls.push(token); |
|||
if (this.connectResult === 'reject') throw new Error('connect failed'); |
|||
} |
|||
|
|||
send(event: ClientEvent): void { |
|||
this.sent.push(event); |
|||
} |
|||
|
|||
on(handler: (event: ServerEvent) => void): () => void { |
|||
this.eventHandler = handler; |
|||
return () => { |
|||
this.eventHandler = null; |
|||
}; |
|||
} |
|||
|
|||
onClose(handler: () => void): () => void { |
|||
this.closeHandler = handler; |
|||
return () => { |
|||
this.closeHandler = null; |
|||
}; |
|||
} |
|||
|
|||
close(): void { |
|||
this.closed = true; |
|||
} |
|||
|
|||
/** Simulates a ServerEvent arriving from the (fake) relay. */ |
|||
emit(event: ServerEvent): void { |
|||
this.eventHandler?.(event); |
|||
} |
|||
|
|||
/** Simulates the underlying socket closing, for any reason. */ |
|||
drop(): void { |
|||
this.closeHandler?.(); |
|||
} |
|||
} |
|||
|
|||
function makeHarness( |
|||
opts: { |
|||
maxAttempts?: number; |
|||
baseDelayMs?: number; |
|||
configureTransport?: (t: FakeTransport) => void; |
|||
} = {} |
|||
) { |
|||
const transports: FakeTransport[] = []; |
|||
let tokenCount = 0; |
|||
const client = new ResilientRealtimeClient({ |
|||
getToken: async () => `token-${++tokenCount}`, |
|||
maxAttempts: opts.maxAttempts, |
|||
baseDelayMs: opts.baseDelayMs ?? 100, |
|||
createTransport: () => { |
|||
const t = new FakeTransport(); |
|||
opts.configureTransport?.(t); |
|||
transports.push(t); |
|||
return t; |
|||
} |
|||
}); |
|||
return { client, transports }; |
|||
} |
|||
|
|||
describe('ResilientRealtimeClient', () => { |
|||
it('connects, sends session.start, and proxies send()/on() to the live transport', async () => { |
|||
const { client, transports } = makeHarness(); |
|||
const states: string[] = []; |
|||
client.onConnectionState((s) => states.push(s)); |
|||
|
|||
await client.connect(); |
|||
|
|||
expect(transports).toHaveLength(1); |
|||
expect(transports[0].sent).toContainEqual({ type: 'session.start' }); |
|||
expect(states).toEqual(['connecting', 'open']); |
|||
|
|||
const received: ServerEvent[] = []; |
|||
client.on((e) => received.push(e)); |
|||
transports[0].emit({ type: 'session.ready', sessionId: 's1' }); |
|||
expect(received).toEqual([{ type: 'session.ready', sessionId: 's1' }]); |
|||
|
|||
client.send({ type: 'input_text.send', text: 'hi' }); |
|||
expect(transports[0].sent).toContainEqual({ type: 'input_text.send', text: 'hi' }); |
|||
}); |
|||
|
|||
it('does not reconnect after a clean session.closed', async () => { |
|||
const { client, transports } = makeHarness(); |
|||
const states: string[] = []; |
|||
client.onConnectionState((s) => states.push(s)); |
|||
await client.connect(); |
|||
|
|||
transports[0].emit({ type: 'session.closed', reason: 'client_requested' }); |
|||
// The underlying socket closing shortly after, as it does in practice —
|
|||
// must not be mistaken for an unexpected drop.
|
|||
transports[0].drop(); |
|||
|
|||
expect(states).toEqual(['connecting', 'open', 'closed']); |
|||
expect(transports).toHaveLength(1); |
|||
}); |
|||
|
|||
it('reconnects with backoff after an unexpected drop, fetching a fresh token and re-sending session.start', async () => { |
|||
vi.useFakeTimers(); |
|||
try { |
|||
const { client, transports } = makeHarness({ baseDelayMs: 100 }); |
|||
const states: string[] = []; |
|||
client.onConnectionState((s) => states.push(s)); |
|||
await client.connect(); |
|||
|
|||
transports[0].drop(); // no prior session.closed — unexpected
|
|||
|
|||
expect(states.at(-1)).toBe('reconnecting'); |
|||
expect(transports).toHaveLength(1); // waiting on backoff, hasn't retried yet
|
|||
|
|||
await vi.advanceTimersByTimeAsync(100); |
|||
|
|||
expect(transports).toHaveLength(2); |
|||
expect(transports[1].connectCalls).toEqual(['token-2']); // fresh token, not reused
|
|||
expect(transports[1].sent).toContainEqual({ type: 'session.start' }); |
|||
expect(states).toEqual(['connecting', 'open', 'reconnecting', 'open']); |
|||
} finally { |
|||
vi.useRealTimers(); |
|||
} |
|||
}); |
|||
|
|||
it('gives up after maxAttempts consecutive failures', async () => { |
|||
vi.useFakeTimers(); |
|||
try { |
|||
const { client, transports } = makeHarness({ |
|||
maxAttempts: 2, |
|||
baseDelayMs: 10, |
|||
configureTransport: (t) => { |
|||
t.connectResult = 'reject'; |
|||
} |
|||
}); |
|||
const states: string[] = []; |
|||
client.onConnectionState((s) => states.push(s)); |
|||
|
|||
await client.connect(); // initial attempt fails
|
|||
await vi.advanceTimersByTimeAsync(10); // retry 1 fails (delay = 10 * 2^0)
|
|||
await vi.advanceTimersByTimeAsync(20); // retry 2 fails (delay = 10 * 2^1) -> exceeds maxAttempts
|
|||
|
|||
expect(states.at(-1)).toBe('closed'); |
|||
expect(transports).toHaveLength(3); // initial attempt + 2 retries
|
|||
} finally { |
|||
vi.useRealTimers(); |
|||
} |
|||
}); |
|||
|
|||
it('close() tears down the transport and suppresses any pending reconnect', async () => { |
|||
vi.useFakeTimers(); |
|||
try { |
|||
const { client, transports } = makeHarness({ baseDelayMs: 100 }); |
|||
await client.connect(); |
|||
transports[0].drop(); |
|||
expect(transports[0].closed).toBe(true); |
|||
|
|||
client.close(); |
|||
await vi.advanceTimersByTimeAsync(1000); |
|||
|
|||
expect(transports).toHaveLength(1); // no reconnect happened after close()
|
|||
expect(() => client.send({ type: 'session.end' })).toThrow(); |
|||
} finally { |
|||
vi.useRealTimers(); |
|||
} |
|||
}); |
|||
}); |
|||
@ -0,0 +1,198 @@ |
|||
import type { ClientEvent, ServerEvent } from '../../core/protocol.js'; |
|||
import { RealtimeClient, type RealtimeClientOptions } from './client.js'; |
|||
|
|||
export type ConnectionState = 'connecting' | 'open' | 'reconnecting' | 'closed'; |
|||
|
|||
type ServerEventHandler = (event: ServerEvent) => void; |
|||
type ConnectionStateHandler = (state: ConnectionState) => void; |
|||
|
|||
/** |
|||
* The subset of `RealtimeClient`'s surface `ResilientRealtimeClient` needs — |
|||
* exists so tests (and, in principle, an alternative transport) can supply a |
|||
* fake without touching a real WebSocket. `RealtimeClient` satisfies this |
|||
* structurally; no explicit `implements`. |
|||
*/ |
|||
export interface RealtimeTransport { |
|||
connect(token: string, options?: RealtimeClientOptions): Promise<void>; |
|||
send(event: ClientEvent): void; |
|||
on(handler: ServerEventHandler): () => void; |
|||
onClose(handler: () => void): () => void; |
|||
close(): void; |
|||
} |
|||
|
|||
export interface ResilientRealtimeClientOptions extends RealtimeClientOptions { |
|||
/** |
|||
* Re-fetches a fresh session token (e.g. `POST /api/realtime/session`). |
|||
* Required because tokens are short-lived — a stale one can't be reused |
|||
* across a reconnect — and because reconnecting has to happen without the |
|||
* caller's involvement for it to be worth wrapping in the first place. |
|||
*/ |
|||
getToken: () => Promise<string>; |
|||
/** Max consecutive reconnect attempts before giving up and reporting 'closed'. */ |
|||
maxAttempts?: number; |
|||
/** Base delay (ms) for exponential backoff between attempts. */ |
|||
baseDelayMs?: number; |
|||
/** For tests. Defaults to `() => new RealtimeClient()`. */ |
|||
createTransport?: () => RealtimeTransport; |
|||
} |
|||
|
|||
const DEFAULT_MAX_ATTEMPTS = 5; |
|||
const DEFAULT_BASE_DELAY_MS = 500; |
|||
|
|||
/** |
|||
* Wraps `RealtimeClient` with automatic reconnection on an *unexpected* |
|||
* transport drop — see docs/REALTIME_SESSION.md's "Resilience" section for |
|||
* the full reasoning. `RealtimeClient` itself is untouched (composition, not |
|||
* inheritance) and stays pure transport; `send()`/`on()`/`close()` here have |
|||
* the same shape, transparently proxied to whichever underlying transport is |
|||
* currently live across reconnects. Connection-state changes are a separate |
|||
* observable (`onConnectionState`) from the `ServerEvent`/protocol |
|||
* vocabulary — `protocol.ts` is the provider-relay contract, not the place |
|||
* for this transport's local retry bookkeeping. |
|||
* |
|||
* Critical design point: reconnecting the browser↔relay socket does not |
|||
* resurrect a dead relay↔OpenAI session. If the server proactively sends |
|||
* `session.closed`, that's a clean end — no auto-reconnect. Auto-reconnect |
|||
* only fires on an unexpected drop (the underlying socket closing with no |
|||
* preceding `session.closed`), and it always starts a **brand-new** session |
|||
* (fresh token, fresh `session.start`) — the model has no memory of the old |
|||
* one either way, so there's no conversation state worth trying to resume. |
|||
* Callers see this plainly as a `reconnecting` → `open` transition and a |
|||
* fresh `session.ready`, and decide for themselves how to present it (e.g. |
|||
* clear the transcript, show a toast) — this wrapper doesn't hide it. |
|||
*/ |
|||
export class ResilientRealtimeClient { |
|||
private readonly createTransport: () => RealtimeTransport; |
|||
private readonly maxAttempts: number; |
|||
private readonly baseDelayMs: number; |
|||
|
|||
private transport: RealtimeTransport | null = null; |
|||
private state: ConnectionState = 'closed'; |
|||
private attempt = 0; |
|||
private closedByCaller = false; |
|||
private reconnectTimer: ReturnType<typeof setTimeout> | null = null; |
|||
private stopEvents: (() => void) | null = null; |
|||
private stopClose: (() => void) | null = null; |
|||
|
|||
private readonly eventHandlers = new Set<ServerEventHandler>(); |
|||
private readonly stateHandlers = new Set<ConnectionStateHandler>(); |
|||
|
|||
constructor(private readonly options: ResilientRealtimeClientOptions) { |
|||
this.createTransport = options.createTransport ?? (() => new RealtimeClient()); |
|||
this.maxAttempts = options.maxAttempts ?? DEFAULT_MAX_ATTEMPTS; |
|||
this.baseDelayMs = options.baseDelayMs ?? DEFAULT_BASE_DELAY_MS; |
|||
} |
|||
|
|||
async connect(): Promise<void> { |
|||
this.closedByCaller = false; |
|||
this.attempt = 0; |
|||
await this.establish(); |
|||
} |
|||
|
|||
send(event: ClientEvent): void { |
|||
if (!this.transport) throw new Error('ResilientRealtimeClient is not connected.'); |
|||
this.transport.send(event); |
|||
} |
|||
|
|||
/** Returns an unsubscribe function. */ |
|||
on(handler: ServerEventHandler): () => void { |
|||
this.eventHandlers.add(handler); |
|||
return () => this.eventHandlers.delete(handler); |
|||
} |
|||
|
|||
/** Returns an unsubscribe function. */ |
|||
onConnectionState(handler: ConnectionStateHandler): () => void { |
|||
this.stateHandlers.add(handler); |
|||
return () => this.stateHandlers.delete(handler); |
|||
} |
|||
|
|||
close(): void { |
|||
this.closedByCaller = true; |
|||
if (this.reconnectTimer) { |
|||
clearTimeout(this.reconnectTimer); |
|||
this.reconnectTimer = null; |
|||
} |
|||
this.teardownTransport(); |
|||
this.setState('closed'); |
|||
} |
|||
|
|||
private async establish(): Promise<void> { |
|||
this.setState(this.attempt === 0 ? 'connecting' : 'reconnecting'); |
|||
|
|||
let token: string; |
|||
try { |
|||
token = await this.options.getToken(); |
|||
} catch { |
|||
this.handleUnexpectedDrop(); |
|||
return; |
|||
} |
|||
if (this.closedByCaller) return; // close() was called while awaiting the token
|
|||
|
|||
const nextTransport = this.createTransport(); |
|||
this.transport = nextTransport; |
|||
this.stopEvents = nextTransport.on((event) => { |
|||
this.forwardEvent(event); |
|||
if (event.type === 'session.closed' && this.transport === nextTransport) { |
|||
this.teardownTransport(); |
|||
this.setState('closed'); |
|||
} |
|||
}); |
|||
this.stopClose = nextTransport.onClose(() => { |
|||
if (this.transport !== nextTransport || this.state !== 'open') return; |
|||
this.handleUnexpectedDrop(); |
|||
}); |
|||
|
|||
try { |
|||
await nextTransport.connect(token, this.options); |
|||
} catch { |
|||
if (this.transport === nextTransport) this.handleUnexpectedDrop(); |
|||
return; |
|||
} |
|||
|
|||
// Torn down while connecting — a session.closed raced in, or close()
|
|||
// was called mid-handshake. Don't resurrect it as 'open'.
|
|||
if (this.transport !== nextTransport) return; |
|||
|
|||
this.attempt = 0; |
|||
this.setState('open'); |
|||
nextTransport.send({ type: 'session.start' }); |
|||
} |
|||
|
|||
private handleUnexpectedDrop(): void { |
|||
this.teardownTransport(); |
|||
if (this.closedByCaller) { |
|||
this.setState('closed'); |
|||
return; |
|||
} |
|||
this.attempt++; |
|||
if (this.attempt > this.maxAttempts) { |
|||
this.setState('closed'); |
|||
return; |
|||
} |
|||
this.setState('reconnecting'); |
|||
const delay = this.baseDelayMs * 2 ** (this.attempt - 1); |
|||
this.reconnectTimer = setTimeout(() => { |
|||
this.reconnectTimer = null; |
|||
void this.establish(); |
|||
}, delay); |
|||
} |
|||
|
|||
private teardownTransport(): void { |
|||
this.stopEvents?.(); |
|||
this.stopEvents = null; |
|||
this.stopClose?.(); |
|||
this.stopClose = null; |
|||
this.transport?.close(); |
|||
this.transport = null; |
|||
} |
|||
|
|||
private forwardEvent(event: ServerEvent): void { |
|||
for (const handler of this.eventHandlers) handler(event); |
|||
} |
|||
|
|||
private setState(state: ConnectionState): void { |
|||
if (this.state === state) return; |
|||
this.state = state; |
|||
for (const handler of this.stateHandlers) handler(state); |
|||
} |
|||
} |
|||
@ -0,0 +1,103 @@ |
|||
import { describe, expect, it, vi } from 'vitest'; |
|||
import type { ServerEvent } from '../../core/protocol.js'; |
|||
import { deriveAvatarState, type AvatarStateSource } from './avatar-state.js'; |
|||
|
|||
/** Minimal stand-in for RealtimeClient/ResilientRealtimeClient — only .on() is used. */ |
|||
class FakeSource implements AvatarStateSource { |
|||
private handler: ((event: ServerEvent) => void) | null = null; |
|||
|
|||
on(handler: (event: ServerEvent) => void): () => void { |
|||
this.handler = handler; |
|||
return () => { |
|||
this.handler = null; |
|||
}; |
|||
} |
|||
|
|||
emit(event: ServerEvent): void { |
|||
this.handler?.(event); |
|||
} |
|||
} |
|||
|
|||
describe('deriveAvatarState', () => { |
|||
it('subscribe() emits the current state immediately', () => { |
|||
const store = deriveAvatarState(new FakeSource()); |
|||
const run = vi.fn(); |
|||
store.subscribe(run); |
|||
expect(run).toHaveBeenCalledTimes(1); |
|||
expect(run).toHaveBeenCalledWith('idle'); |
|||
}); |
|||
|
|||
it('walks listening -> thinking -> talking -> idle across a full voice turn', () => { |
|||
const source = new FakeSource(); |
|||
const store = deriveAvatarState(source); |
|||
const seen: string[] = []; |
|||
store.subscribe((s) => seen.push(s)); |
|||
|
|||
store.setListening(true); |
|||
store.setListening(false); |
|||
source.emit({ type: 'output_audio.chunk', audio: '' }); |
|||
source.emit({ type: 'output_audio.done' }); |
|||
|
|||
expect(seen).toEqual(['idle', 'listening', 'thinking', 'talking', 'idle']); |
|||
}); |
|||
|
|||
it('notifyResponseRequested() drives thinking for the typed-text path', () => { |
|||
const source = new FakeSource(); |
|||
const store = deriveAvatarState(source); |
|||
const seen: string[] = []; |
|||
store.subscribe((s) => seen.push(s)); |
|||
|
|||
store.notifyResponseRequested(); |
|||
source.emit({ type: 'transcript.delta', role: 'assistant', text: 'hi' }); |
|||
source.emit({ type: 'transcript.done', role: 'assistant' }); |
|||
|
|||
expect(seen).toEqual(['idle', 'thinking', 'talking', 'idle']); |
|||
}); |
|||
|
|||
it('ignores user transcript events (no signal either way)', () => { |
|||
const source = new FakeSource(); |
|||
const store = deriveAvatarState(source); |
|||
const seen: string[] = []; |
|||
store.subscribe((s) => seen.push(s)); |
|||
|
|||
source.emit({ type: 'transcript.delta', role: 'user', text: 'hello' }); |
|||
|
|||
expect(seen).toEqual(['idle']); |
|||
}); |
|||
|
|||
it('session.ready and session.closed reset to idle from any state', () => { |
|||
const source = new FakeSource(); |
|||
const store = deriveAvatarState(source); |
|||
const seen: string[] = []; |
|||
store.subscribe((s) => seen.push(s)); |
|||
|
|||
source.emit({ type: 'output_audio.chunk', audio: '' }); // -> talking
|
|||
source.emit({ type: 'session.closed', reason: 'client_requested' }); |
|||
|
|||
expect(seen).toEqual(['idle', 'talking', 'idle']); |
|||
}); |
|||
|
|||
it('response.done is a fallback that stops talking even without output_audio.done', () => { |
|||
const source = new FakeSource(); |
|||
const store = deriveAvatarState(source); |
|||
const seen: string[] = []; |
|||
store.subscribe((s) => seen.push(s)); |
|||
|
|||
source.emit({ type: 'output_audio.chunk', audio: '' }); |
|||
source.emit({ type: 'response.done' }); |
|||
|
|||
expect(seen).toEqual(['idle', 'talking', 'idle']); |
|||
}); |
|||
|
|||
it('destroy() stops reacting to further events', () => { |
|||
const source = new FakeSource(); |
|||
const store = deriveAvatarState(source); |
|||
const seen: string[] = []; |
|||
store.subscribe((s) => seen.push(s)); |
|||
|
|||
store.destroy(); |
|||
source.emit({ type: 'output_audio.chunk', audio: '' }); |
|||
|
|||
expect(seen).toEqual(['idle']); |
|||
}); |
|||
}); |
|||
@ -0,0 +1,98 @@ |
|||
import type { ServerEvent } from '../../core/protocol.js'; |
|||
import { nextAvatarState, type AvatarSignal, type AvatarState } from '../../humanoid/state.js'; |
|||
|
|||
/** |
|||
* The subset of `RealtimeClient`'s (or `ResilientRealtimeClient`'s) surface |
|||
* `deriveAvatarState` needs — accepted structurally so this package doesn't |
|||
* care which transport variant is in use. |
|||
*/ |
|||
export interface AvatarStateSource { |
|||
on(handler: (event: ServerEvent) => void): () => void; |
|||
} |
|||
|
|||
export interface AvatarStateStore { |
|||
/** Svelte-store-compatible: calls `run` immediately with the current value, then on every change. */ |
|||
subscribe(run: (state: AvatarState) => void): () => void; |
|||
/** |
|||
* Called by whatever mic-capture code a project uses (that pipeline |
|||
* doesn't exist in core yet — see docs/HUMANOID.md's non-goals). The |
|||
* client, not the server, is the only thing that currently knows when |
|||
* the user starts/stops talking: the protocol doesn't surface OpenAI's |
|||
* `input_audio_buffer.speech_*` VAD events — see the "not part of our |
|||
* normalized vocabulary" comment in |
|||
* `server/core/providers/openai.ts`'s `fromOpenAiEvent`. |
|||
*/ |
|||
setListening(active: boolean): void; |
|||
/** |
|||
* Call right when sending a request the server-event stream alone won't |
|||
* announce — chiefly a typed `input_text.send` (the voice path already |
|||
* gets this for free from `setListening(false)`). |
|||
*/ |
|||
notifyResponseRequested(): void; |
|||
/** Stops listening to the source and releases all subscribers. */ |
|||
destroy(): void; |
|||
} |
|||
|
|||
/** |
|||
* Derives a reactive `AvatarState` from a live `RealtimeClient` or |
|||
* `ResilientRealtimeClient`. Translates `ServerEvent`s (plus the imperative |
|||
* calls above, for signals the event stream alone can't carry) into |
|||
* `AvatarSignal`s and folds them through the pure `nextAvatarState` reducer |
|||
* from `$lib/humanoid/state.ts`. |
|||
* |
|||
* `thinking` is an explicit heuristic — the time between a request and the |
|||
* assistant's first output — not a precise reflection of server-side |
|||
* activity: the browser never sees server-executed tool calls at all (see |
|||
* docs/TOOLS.md), so "the model is running a tool" isn't observable here. |
|||
*/ |
|||
export function deriveAvatarState(source: AvatarStateSource): AvatarStateStore { |
|||
let state: AvatarState = 'idle'; |
|||
const subscribers = new Set<(state: AvatarState) => void>(); |
|||
|
|||
function apply(signal: AvatarSignal): void { |
|||
const next = nextAvatarState(state, signal); |
|||
if (next === state) return; |
|||
state = next; |
|||
for (const run of subscribers) run(state); |
|||
} |
|||
|
|||
const stopEvents = source.on((event) => { |
|||
switch (event.type) { |
|||
case 'session.ready': |
|||
case 'session.closed': |
|||
apply({ type: 'reset' }); |
|||
break; |
|||
case 'output_audio.chunk': |
|||
apply({ type: 'talking_started' }); |
|||
break; |
|||
case 'transcript.delta': |
|||
if (event.role === 'assistant') apply({ type: 'talking_started' }); |
|||
break; |
|||
case 'output_audio.done': |
|||
case 'response.done': |
|||
apply({ type: 'talking_stopped' }); |
|||
break; |
|||
case 'transcript.done': |
|||
if (event.role === 'assistant') apply({ type: 'talking_stopped' }); |
|||
break; |
|||
} |
|||
}); |
|||
|
|||
return { |
|||
subscribe(run) { |
|||
subscribers.add(run); |
|||
run(state); |
|||
return () => subscribers.delete(run); |
|||
}, |
|||
setListening(active) { |
|||
apply({ type: active ? 'listening_started' : 'listening_stopped' }); |
|||
}, |
|||
notifyResponseRequested() { |
|||
apply({ type: 'response_requested' }); |
|||
}, |
|||
destroy() { |
|||
stopEvents(); |
|||
subscribers.clear(); |
|||
} |
|||
}; |
|||
} |
|||
@ -0,0 +1,52 @@ |
|||
import { describe, expect, it } from 'vitest'; |
|||
import { compositeAnimationFrames } from './animation.js'; |
|||
|
|||
describe('compositeAnimationFrames', () => { |
|||
it('sums add-mode layers onto the same channel', () => { |
|||
// 0.25 + 0.25, not 0.2 + 0.1, so the sum is exactly representable in
|
|||
// binary floating point and this isn't a disguised precision test.
|
|||
expect( |
|||
compositeAnimationFrames([ |
|||
{ frame: { eyeLookInLeft: 0.25 }, mode: 'add' }, |
|||
{ frame: { eyeLookInLeft: 0.25 }, mode: 'add' } |
|||
]) |
|||
).toEqual({ eyeLookInLeft: 0.5 }); |
|||
}); |
|||
|
|||
it('override replaces rather than sums, later layers winning', () => { |
|||
expect( |
|||
compositeAnimationFrames([ |
|||
{ frame: { eyeBlinkLeft: 0.1 }, mode: 'add' }, |
|||
{ frame: { eyeBlinkLeft: 1 }, mode: 'override' } |
|||
]) |
|||
).toEqual({ eyeBlinkLeft: 1 }); |
|||
}); |
|||
|
|||
it('applies a layer weight multiplier before composing', () => { |
|||
expect( |
|||
compositeAnimationFrames([{ frame: { jawOpen: 0.5 }, mode: 'add', weight: 0.5 }]) |
|||
).toEqual({ jawOpen: 0.25 }); |
|||
}); |
|||
|
|||
it('clamps summed channels to [0, 1]', () => { |
|||
expect( |
|||
compositeAnimationFrames([ |
|||
{ frame: { mouthSmileLeft: 0.8 }, mode: 'add' }, |
|||
{ frame: { mouthSmileLeft: 0.8 }, mode: 'add' } |
|||
]) |
|||
).toEqual({ mouthSmileLeft: 1 }); |
|||
}); |
|||
|
|||
it('leaves untouched channels from other layers alone', () => { |
|||
expect( |
|||
compositeAnimationFrames([ |
|||
{ frame: { eyeBlinkLeft: 1 }, mode: 'override' }, |
|||
{ frame: { jawOpen: 0.4 }, mode: 'add' } |
|||
]) |
|||
).toEqual({ eyeBlinkLeft: 1, jawOpen: 0.4 }); |
|||
}); |
|||
|
|||
it('returns an empty frame for no layers', () => { |
|||
expect(compositeAnimationFrames([])).toEqual({}); |
|||
}); |
|||
}); |
|||
@ -0,0 +1,114 @@ |
|||
/** |
|||
* Isomorphic, renderer-agnostic animation-channel contract — the other half |
|||
* of the optional `humanoid` package alongside `state.ts` (see |
|||
* docs/HUMANOID.md). Defines a shared channel vocabulary (the standard |
|||
* ARKit blend shapes most 3D humanoid rigs in this project family share) and |
|||
* a compositor so independently-built animation sources (a future eye |
|||
* look-at library, blink, lip-sync, body/gesture) can layer onto the same |
|||
* frame without colliding — mirrors the additive-layers pattern from the |
|||
* eye look-at design doc (Macro/Micro/Nano summed into one gaze offset), |
|||
* generalized to arbitrary channels and to layers that override rather than |
|||
* sum. No rendering, no bone/mesh knowledge — a project's own rig-binding |
|||
* code is what actually applies a composited `AnimationFrame` to a mesh. |
|||
*/ |
|||
|
|||
/** The 52 standard ARKit face blend shapes (`ARFaceAnchor.BlendShapeLocation`). */ |
|||
export type ARKitBlendShape = |
|||
| 'browDownLeft' |
|||
| 'browDownRight' |
|||
| 'browInnerUp' |
|||
| 'browOuterUpLeft' |
|||
| 'browOuterUpRight' |
|||
| 'cheekPuff' |
|||
| 'cheekSquintLeft' |
|||
| 'cheekSquintRight' |
|||
| 'eyeBlinkLeft' |
|||
| 'eyeBlinkRight' |
|||
| 'eyeLookDownLeft' |
|||
| 'eyeLookDownRight' |
|||
| 'eyeLookInLeft' |
|||
| 'eyeLookInRight' |
|||
| 'eyeLookOutLeft' |
|||
| 'eyeLookOutRight' |
|||
| 'eyeLookUpLeft' |
|||
| 'eyeLookUpRight' |
|||
| 'eyeSquintLeft' |
|||
| 'eyeSquintRight' |
|||
| 'eyeWideLeft' |
|||
| 'eyeWideRight' |
|||
| 'jawForward' |
|||
| 'jawLeft' |
|||
| 'jawOpen' |
|||
| 'jawRight' |
|||
| 'mouthClose' |
|||
| 'mouthDimpleLeft' |
|||
| 'mouthDimpleRight' |
|||
| 'mouthFrownLeft' |
|||
| 'mouthFrownRight' |
|||
| 'mouthFunnel' |
|||
| 'mouthLeft' |
|||
| 'mouthLowerDownLeft' |
|||
| 'mouthLowerDownRight' |
|||
| 'mouthPressLeft' |
|||
| 'mouthPressRight' |
|||
| 'mouthPucker' |
|||
| 'mouthRight' |
|||
| 'mouthRollLower' |
|||
| 'mouthRollUpper' |
|||
| 'mouthShrugLower' |
|||
| 'mouthShrugUpper' |
|||
| 'mouthSmileLeft' |
|||
| 'mouthSmileRight' |
|||
| 'mouthStretchLeft' |
|||
| 'mouthStretchRight' |
|||
| 'mouthUpperUpLeft' |
|||
| 'mouthUpperUpRight' |
|||
| 'noseSneerLeft' |
|||
| 'noseSneerRight' |
|||
| 'tongueOut'; |
|||
|
|||
/** Sparse channel → weight (0–1) map for a single frame. */ |
|||
export type AnimationFrame = Partial<Record<ARKitBlendShape, number>>; |
|||
|
|||
export interface AnimationLayer { |
|||
frame: AnimationFrame; |
|||
/** |
|||
* `'add'` sums onto whatever the channel already holds (the eye look-at |
|||
* doc's Macro/Micro/Nano pattern — several small offsets combining into |
|||
* one). `'override'` replaces the channel outright — for layers whose |
|||
* values can't be meaningfully summed (e.g. a blink closing an eyelid |
|||
* that a gaze layer also touches). |
|||
*/ |
|||
mode: 'add' | 'override'; |
|||
/** Multiplies every weight in `frame` before composing. Defaults to 1. */ |
|||
weight?: number; |
|||
} |
|||
|
|||
/** |
|||
* Folds `layers` in order into one `AnimationFrame`, clamping each channel |
|||
* to [0, 1] at the end. Later layers in the array apply after earlier ones — |
|||
* order matters for `'override'` layers, not for `'add'` ones. |
|||
*/ |
|||
export function compositeAnimationFrames(layers: AnimationLayer[]): AnimationFrame { |
|||
const result: AnimationFrame = {}; |
|||
|
|||
for (const layer of layers) { |
|||
const weight = layer.weight ?? 1; |
|||
for (const [channel, value] of Object.entries(layer.frame) as Array< |
|||
[ARKitBlendShape, number] |
|||
>) { |
|||
const scaled = value * weight; |
|||
if (layer.mode === 'override') { |
|||
result[channel] = scaled; |
|||
} else { |
|||
result[channel] = (result[channel] ?? 0) + scaled; |
|||
} |
|||
} |
|||
} |
|||
|
|||
for (const channel of Object.keys(result) as ARKitBlendShape[]) { |
|||
result[channel] = Math.min(1, Math.max(0, result[channel]!)); |
|||
} |
|||
|
|||
return result; |
|||
} |
|||
@ -0,0 +1,34 @@ |
|||
import { describe, expect, it } from 'vitest'; |
|||
import { nextAvatarState } from './state.js'; |
|||
|
|||
describe('nextAvatarState', () => { |
|||
it('listening_started always moves to listening', () => { |
|||
expect(nextAvatarState('idle', { type: 'listening_started' })).toBe('listening'); |
|||
expect(nextAvatarState('talking', { type: 'listening_started' })).toBe('listening'); |
|||
}); |
|||
|
|||
it('listening_stopped moves to thinking (a response is expected next), only from listening', () => { |
|||
expect(nextAvatarState('listening', { type: 'listening_stopped' })).toBe('thinking'); |
|||
expect(nextAvatarState('idle', { type: 'listening_stopped' })).toBe('idle'); |
|||
}); |
|||
|
|||
it('response_requested moves to thinking unless already talking', () => { |
|||
expect(nextAvatarState('idle', { type: 'response_requested' })).toBe('thinking'); |
|||
expect(nextAvatarState('listening', { type: 'response_requested' })).toBe('thinking'); |
|||
expect(nextAvatarState('talking', { type: 'response_requested' })).toBe('talking'); |
|||
}); |
|||
|
|||
it('talking_started always moves to talking', () => { |
|||
expect(nextAvatarState('thinking', { type: 'talking_started' })).toBe('talking'); |
|||
}); |
|||
|
|||
it('talking_stopped only returns to idle from talking', () => { |
|||
expect(nextAvatarState('talking', { type: 'talking_stopped' })).toBe('idle'); |
|||
expect(nextAvatarState('idle', { type: 'talking_stopped' })).toBe('idle'); |
|||
}); |
|||
|
|||
it('reset always returns to idle', () => { |
|||
expect(nextAvatarState('talking', { type: 'reset' })).toBe('idle'); |
|||
expect(nextAvatarState('listening', { type: 'reset' })).toBe('idle'); |
|||
}); |
|||
}); |
|||
@ -0,0 +1,54 @@ |
|||
/** |
|||
* Isomorphic, renderer-agnostic avatar state — part of the optional |
|||
* `humanoid` package (see docs/HUMANOID.md and the "Module layering" section |
|||
* of docs/ARCHITECTURE.md for what that means). Pure types and a pure |
|||
* reducer, no WS/browser dependency, so future animation libraries (eye |
|||
* look-at, blink, lip-sync, body/gesture) can all key off the same signal |
|||
* without each re-deriving turn-taking themselves. |
|||
* |
|||
* Deliberately just four states, not the `IdleActive`/`IdlePassive` split a |
|||
* humanoid rig's own animation design may eventually want — that |
|||
* distinction needs a user-presence signal (webcam/face-tracking) this |
|||
* package has no opinion on. A project with that signal can layer its own |
|||
* finer-grained idle split on top; `idle` here just means "no signal that |
|||
* the user or the model is currently doing anything." |
|||
*/ |
|||
export type AvatarState = 'idle' | 'listening' | 'thinking' | 'talking'; |
|||
|
|||
export type AvatarSignal = |
|||
| { type: 'listening_started' } |
|||
| { type: 'listening_stopped' } |
|||
| { type: 'response_requested' } |
|||
| { type: 'talking_started' } |
|||
| { type: 'talking_stopped' } |
|||
| { type: 'reset' }; |
|||
|
|||
/** |
|||
* Pure state transition — no side effects, directly unit-testable. The |
|||
* browser-facing translation from real `ServerEvent`s (and the imperative |
|||
* `setListening` a project's mic-capture code calls) into `AvatarSignal`s |
|||
* lives in `$lib/client/humanoid/avatar-state.ts`, which folds them through |
|||
* this function; this module doesn't know a `RealtimeClient` exists. |
|||
*/ |
|||
export function nextAvatarState(current: AvatarState, signal: AvatarSignal): AvatarState { |
|||
switch (signal.type) { |
|||
case 'listening_started': |
|||
return 'listening'; |
|||
// Ending listening almost always means a turn just completed and a
|
|||
// response is expected next — go straight to 'thinking' rather than
|
|||
// 'idle', so the avatar doesn't visibly relax for the split second
|
|||
// before the model's response starts arriving.
|
|||
case 'listening_stopped': |
|||
return current === 'listening' ? 'thinking' : current; |
|||
case 'response_requested': |
|||
return current === 'talking' ? current : 'thinking'; |
|||
case 'talking_started': |
|||
return 'talking'; |
|||
case 'talking_stopped': |
|||
return current === 'talking' ? 'idle' : current; |
|||
case 'reset': |
|||
return 'idle'; |
|||
default: |
|||
return current; |
|||
} |
|||
} |
|||
@ -0,0 +1,19 @@ |
|||
import type { MockScript } from '../mock.js'; |
|||
|
|||
/** |
|||
* Short canned exchange for exercising transcript rendering and (via |
|||
* output_audio.chunk/done timing) humanoid avatar-state derivation without a |
|||
* real API key. `audio` is an empty string — nothing decodes/plays it yet (no |
|||
* client-side audio capture/playback pipeline exists in core today), only its |
|||
* start/stop timing matters here. |
|||
*/ |
|||
export const DEFAULT_MOCK_SCRIPT: MockScript = [ |
|||
{ delayMs: 300, event: { type: 'output_audio.chunk', audio: '' } }, |
|||
{ |
|||
delayMs: 200, |
|||
event: { type: 'transcript.delta', role: 'assistant', text: "Hey, I'm a mock avatar reply." } |
|||
}, |
|||
{ delayMs: 1200, event: { type: 'transcript.done', role: 'assistant' } }, |
|||
{ delayMs: 0, event: { type: 'output_audio.done' } }, |
|||
{ delayMs: 0, event: { type: 'response.done' } } |
|||
]; |
|||
@ -0,0 +1,17 @@ |
|||
import type { AppConfig } from '../config.js'; |
|||
import type { RealtimeProvider } from '../provider.js'; |
|||
import { DEFAULT_MOCK_SCRIPT } from './fixtures/default.js'; |
|||
import { MockRealtimeProvider } from './mock.js'; |
|||
import { OpenAiRealtimeProvider } from './openai.js'; |
|||
|
|||
/** |
|||
* Picks the `RealtimeProvider` implementation for `config.realtimeProvider` |
|||
* — the one place both composition roots (`server.js`, `dev-plugin.ts`) call, |
|||
* instead of each hardcoding `new OpenAiRealtimeProvider(...)` directly. |
|||
*/ |
|||
export function selectProvider(config: AppConfig): RealtimeProvider { |
|||
if (config.realtimeProvider === 'mock') { |
|||
return new MockRealtimeProvider(DEFAULT_MOCK_SCRIPT); |
|||
} |
|||
return new OpenAiRealtimeProvider({ apiKey: config.openaiApiKey }); |
|||
} |
|||
@ -0,0 +1,56 @@ |
|||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; |
|||
import type { ServerEvent } from '../../../core/protocol.js'; |
|||
import { MockRealtimeProvider, type MockScript } from './mock.js'; |
|||
|
|||
const SCRIPT: MockScript = [ |
|||
{ delayMs: 100, event: { type: 'transcript.delta', role: 'assistant', text: 'hi' } }, |
|||
{ delayMs: 50, event: { type: 'transcript.done', role: 'assistant' } } |
|||
]; |
|||
|
|||
beforeEach(() => { |
|||
vi.useFakeTimers(); |
|||
}); |
|||
|
|||
afterEach(() => { |
|||
vi.useRealTimers(); |
|||
}); |
|||
|
|||
describe('MockRealtimeProvider', () => { |
|||
it('replays the script in order, at the cumulative scheduled delay', async () => { |
|||
const provider = new MockRealtimeProvider(SCRIPT); |
|||
const session = await provider.createSession(); |
|||
const received: ServerEvent[] = []; |
|||
session.onEvent((event) => received.push(event)); |
|||
|
|||
await vi.advanceTimersByTimeAsync(99); |
|||
expect(received).toEqual([]); |
|||
|
|||
await vi.advanceTimersByTimeAsync(1); // total 100ms — first step fires
|
|||
expect(received).toEqual([{ type: 'transcript.delta', role: 'assistant', text: 'hi' }]); |
|||
|
|||
await vi.advanceTimersByTimeAsync(50); // total 150ms — second step fires
|
|||
expect(received).toEqual([ |
|||
{ type: 'transcript.delta', role: 'assistant', text: 'hi' }, |
|||
{ type: 'transcript.done', role: 'assistant' } |
|||
]); |
|||
}); |
|||
|
|||
it('close() cancels any not-yet-fired steps', async () => { |
|||
const provider = new MockRealtimeProvider(SCRIPT); |
|||
const session = await provider.createSession(); |
|||
const received: ServerEvent[] = []; |
|||
session.onEvent((event) => received.push(event)); |
|||
|
|||
session.close(); |
|||
await vi.advanceTimersByTimeAsync(1000); |
|||
|
|||
expect(received).toEqual([]); |
|||
}); |
|||
|
|||
it('send() is a no-op — the script does not react to browser input', async () => { |
|||
const provider = new MockRealtimeProvider(SCRIPT); |
|||
const session = await provider.createSession(); |
|||
|
|||
expect(() => session.send({ type: 'input_text.send', text: 'hello' })).not.toThrow(); |
|||
}); |
|||
}); |
|||
@ -0,0 +1,54 @@ |
|||
import type { ServerEvent } from '../../../core/protocol.js'; |
|||
import type { ProviderSession, RealtimeProvider } from '../provider.js'; |
|||
|
|||
/** One scripted event, replayed `delayMs` after the previous step (or after `createSession` for the first step). */ |
|||
export interface MockScriptStep { |
|||
delayMs: number; |
|||
event: ServerEvent; |
|||
} |
|||
|
|||
export type MockScript = MockScriptStep[]; |
|||
|
|||
/** |
|||
* `RealtimeProvider` that replays a canned `MockScript` on a timer instead of |
|||
* talking to a real upstream — lets client work (transcript rendering, avatar |
|||
* state, animation timing) happen without an API key or live usage. See |
|||
* docs/REALTIME_SESSION.md's "Development without a live API" section. |
|||
* |
|||
* Deliberately simple: linear, one-shot replay. `send()` is a no-op — the |
|||
* script doesn't react to what the browser sends. Interactive/branching |
|||
* fixtures are a plausible future enhancement, not needed yet. |
|||
*/ |
|||
export class MockRealtimeProvider implements RealtimeProvider { |
|||
constructor(private readonly script: MockScript) {} |
|||
|
|||
// Ignores `config` — the script is canned, not generated from session config.
|
|||
async createSession(): Promise<ProviderSession> { |
|||
return new MockProviderSession(this.script); |
|||
} |
|||
} |
|||
|
|||
class MockProviderSession implements ProviderSession { |
|||
private handler: ((event: ServerEvent) => void) | null = null; |
|||
private readonly timers: NodeJS.Timeout[] = []; |
|||
|
|||
constructor(script: MockScript) { |
|||
let elapsed = 0; |
|||
for (const step of script) { |
|||
elapsed += step.delayMs; |
|||
this.timers.push(setTimeout(() => this.handler?.(step.event), elapsed)); |
|||
} |
|||
} |
|||
|
|||
// No-op — see the class doc comment on MockRealtimeProvider.
|
|||
send(): void {} |
|||
|
|||
onEvent(handler: (event: ServerEvent) => void): void { |
|||
this.handler = handler; |
|||
} |
|||
|
|||
close(): void { |
|||
for (const timer of this.timers) clearTimeout(timer); |
|||
this.timers.length = 0; |
|||
} |
|||
} |
|||
@ -0,0 +1,107 @@ |
|||
<script lang="ts"> |
|||
import { |
|||
ResilientRealtimeClient, |
|||
type ConnectionState |
|||
} from '$lib/client/core/resilient-client.js'; |
|||
import { deriveAvatarState, type AvatarStateStore } from '$lib/client/humanoid/avatar-state.js'; |
|||
import type { ServerEvent } from '$lib/core/protocol.js'; |
|||
import type { AvatarState } from '$lib/humanoid/state.js'; |
|||
|
|||
type Status = 'idle' | 'connecting' | 'reconnecting' | 'ready' | 'error' | 'closed'; |
|||
|
|||
let status = $state<Status>('idle'); |
|||
let statusMessage = $state(''); |
|||
let avatarState = $state<AvatarState>('idle'); |
|||
let listening = $state(false); |
|||
|
|||
let client: ResilientRealtimeClient | null = null; |
|||
let stateStore: AvatarStateStore | null = null; |
|||
let stopStateSub: (() => void) | null = null; |
|||
|
|||
async function fetchToken(): Promise<string> { |
|||
const res = await fetch('/api/realtime/session', { method: 'POST' }); |
|||
if (!res.ok) throw new Error(`Failed to mint session token (${res.status})`); |
|||
const { token } = await res.json(); |
|||
return token; |
|||
} |
|||
|
|||
async function connect() { |
|||
status = 'connecting'; |
|||
statusMessage = ''; |
|||
try { |
|||
const nextClient = new ResilientRealtimeClient({ getToken: fetchToken }); |
|||
nextClient.onConnectionState(handleConnectionState); |
|||
nextClient.on(handleServerEvent); |
|||
await nextClient.connect(); |
|||
client = nextClient; |
|||
|
|||
// See $lib/humanoid — this is the "humanoid" optional package's |
|||
// contract layer, not tied to any renderer/UI. deriveAvatarState |
|||
// takes the client structurally (RealtimeClient or |
|||
// ResilientRealtimeClient both work). |
|||
const store = deriveAvatarState(nextClient); |
|||
stateStore = store; |
|||
stopStateSub = store.subscribe((s) => (avatarState = s)); |
|||
} catch (err) { |
|||
status = 'error'; |
|||
statusMessage = err instanceof Error ? err.message : 'Failed to connect.'; |
|||
} |
|||
} |
|||
|
|||
function handleConnectionState(connState: ConnectionState) { |
|||
if (connState === 'connecting') { |
|||
status = 'connecting'; |
|||
} else if (connState === 'reconnecting') { |
|||
status = 'reconnecting'; |
|||
statusMessage = 'Connection lost — reconnecting…'; |
|||
} else if (connState === 'closed' && status !== 'error') { |
|||
status = 'closed'; |
|||
} |
|||
} |
|||
|
|||
function handleServerEvent(event: ServerEvent) { |
|||
if (event.type === 'session.ready') { |
|||
status = 'ready'; |
|||
statusMessage = ''; |
|||
} else if (event.type === 'error') { |
|||
status = 'error'; |
|||
statusMessage = event.message; |
|||
} else if (event.type === 'session.closed') { |
|||
statusMessage = event.reason; |
|||
} |
|||
} |
|||
|
|||
function toggleListening() { |
|||
listening = !listening; |
|||
stateStore?.setListening(listening); |
|||
} |
|||
|
|||
function disconnect() { |
|||
client?.send({ type: 'session.end' }); |
|||
client?.close(); |
|||
client = null; |
|||
stopStateSub?.(); |
|||
stopStateSub = null; |
|||
stateStore?.destroy(); |
|||
stateStore = null; |
|||
listening = false; |
|||
status = 'closed'; |
|||
} |
|||
</script> |
|||
|
|||
<h1>Humanoid avatar-state demo</h1> |
|||
<p> |
|||
Exercises <code>$lib/humanoid</code>'s state derivation without a live API key: run |
|||
<code>REALTIME_PROVIDER=mock npm run dev</code> (dev-server restart required — see AGENTS.md's HMR |
|||
gotcha) and connect below to watch <code>avatarState</code> move through the mock provider's default |
|||
fixture. |
|||
</p> |
|||
<p>Status: {status}{statusMessage ? ` — ${statusMessage}` : ''}</p> |
|||
<p>Avatar state: <strong>{avatarState}</strong></p> |
|||
|
|||
{#if status === 'idle' || status === 'error' || status === 'closed'} |
|||
<button onclick={connect}>Connect</button> |
|||
{:else if status === 'ready'} |
|||
<button onclick={disconnect}>Disconnect</button> |
|||
<button onclick={toggleListening}>{listening ? 'Stop listening' : 'Start listening'}</button> |
|||
{/if} |
|||
Loading…
Reference in new issue