Browse Source
Wraps RealtimeClient with reconnect-on-drop (ResilientRealtimeClient), adds a MockRealtimeProvider so client/animation work doesn't need a live OpenAI key, and introduces "optional package" as a third module-layering kind alongside core/project — humanoid is the first instance, providing a renderer-agnostic avatar-state reducer and ARKit blend-shape animation compositor that future eye look-at/blink/lip-sync libraries can build on without forcing that surface onto non-humanoid (e.g. 2D) projects. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>main
26 changed files with 1329 additions and 48 deletions
@ -0,0 +1,59 @@ |
|||||
|
# Humanoid package |
||||
|
|
||||
|
`humanoid` is the first instance of the **optional package** convention (see "Module layering" in [ARCHITECTURE.md](ARCHITECTURE.md)) — a renderer-agnostic contract layer for 3D humanoid avatars rigged with ARKit blend shapes, scoped out of `core` because a 2D or non-humanoid project has no use for it. Nothing in `core` or `project` imports from it; a project opts in by importing directly from `$lib/humanoid` / `$lib/client/humanoid`, and if nothing imports it, it's simply not part of that project's bundle. |
||||
|
|
||||
|
This package ships the **contract** future animation libraries (eye look-at, blink, lip-sync, body/gesture) plug into — not the libraries themselves, and not a renderer. It has no idea what's rendering the avatar (Three.js, Babylon, a game engine WebGL bridge, anything else) — that stays entirely project-owned, per the base project's founding constraint that appearance/UI is never core's job. |
||||
|
|
||||
|
## `$lib/humanoid/state.ts` — avatar state |
||||
|
|
||||
|
```ts |
||||
|
type AvatarState = 'idle' | 'listening' | 'thinking' | 'talking'; |
||||
|
``` |
||||
|
|
||||
|
Deliberately just four states, not the `IdleActive`/`IdlePassive` split a humanoid rig's own animation design (e.g. a future eye look-at library) may eventually want — that distinction needs a user-presence signal (webcam/face-tracking) this package has no opinion on. A project with that signal can layer its own finer idle split on top; `idle` here just means "no signal that the user or the model is currently doing anything." |
||||
|
|
||||
|
`nextAvatarState(current, signal)` is a pure reducer — no WS/browser dependency, directly unit-tested in `state.spec.ts`. The signal set is small and closed (`listening_started`/`listening_stopped`, `response_requested`, `talking_started`/`talking_stopped`, `reset`). |
||||
|
|
||||
|
Note `listening_stopped` moves to `thinking`, not `idle` — ending listening almost always means a turn just completed and a response is expected next, so the avatar shouldn't visibly relax for the split second before the model's response starts arriving. |
||||
|
|
||||
|
## `$lib/client/humanoid/avatar-state.ts` — deriving it live |
||||
|
|
||||
|
`deriveAvatarState(source)` takes a `RealtimeClient` or `ResilientRealtimeClient` (accepted structurally, via an `AvatarStateSource` interface — this package doesn't care which transport variant is in use) and folds its `ServerEvent`s through `nextAvatarState`: |
||||
|
|
||||
|
| `ServerEvent` | Signal | |
||||
|
| ------------------------------------- | ----------------- | |
||||
|
| `session.ready` / `session.closed` | `reset` | |
||||
|
| `output_audio.chunk` | `talking_started` | |
||||
|
| `transcript.delta` (role: assistant) | `talking_started` | |
||||
|
| `output_audio.done` / `response.done` | `talking_stopped` | |
||||
|
| `transcript.done` (role: assistant) | `talking_stopped` | |
||||
|
|
||||
|
Two signals can't come from the event stream alone, so the returned store exposes them imperatively: |
||||
|
|
||||
|
- **`setListening(active)`** — call from whatever mic-capture code a project uses. That pipeline doesn't exist in `core` yet (see "Known limitations" below), and the client is the only thing that currently knows when the user starts/stops talking: the protocol doesn't surface OpenAI's `input_audio_buffer.speech_*` VAD events — see the "not part of our normalized vocabulary" comment in `server/core/providers/openai.ts`'s `fromOpenAiEvent`. |
||||
|
- **`notifyResponseRequested()`** — call right when sending a typed `input_text.send` (the voice path already gets this for free from `setListening(false)`). |
||||
|
|
||||
|
The returned `AvatarStateStore.subscribe()` follows the Svelte store contract (calls `run` immediately with the current value, then on every change), so it's usable directly as `$store` in a `.svelte` file without a hard Svelte dependency in this package itself. See `/demo/humanoid` (`src/routes/demo/humanoid/+page.svelte`) for the reference wiring, including the manual "listening" toggle standing in for real mic-capture UI. |
||||
|
|
||||
|
### Known limitations |
||||
|
|
||||
|
- **`thinking` is a heuristic**, not a precise reflection of server-side activity — the time between a request and the assistant's first output. The browser never sees server-executed tool calls at all (see [TOOLS.md](TOOLS.md)), so "the model is running a tool" isn't observable from here. |
||||
|
- **No presence-based idle split** (see above) — a project with webcam/face-tracking data can extend on top. |
||||
|
|
||||
|
## `$lib/humanoid/animation.ts` — animation-channel composition |
||||
|
|
||||
|
```ts |
||||
|
type ARKitBlendShape = 'jawOpen' | 'eyeBlinkLeft' | /* ...all 52 ARFaceAnchor.BlendShapeLocation names */; |
||||
|
type AnimationFrame = Partial<Record<ARKitBlendShape, number>>; // sparse, 0–1 weights |
||||
|
``` |
||||
|
|
||||
|
`compositeAnimationFrames(layers)` folds an ordered list of `AnimationLayer`s (`{ frame, mode: 'add' | 'override', weight? }`) into one `AnimationFrame`, clamping each channel to `[0, 1]`. This generalizes the eye look-at design doc's own additive-layers pattern — Macro, Micro, and Nano gaze offsets summed into one effective offset — to arbitrary channels, plus `'override'` for layers whose values can't be meaningfully summed (a blink closing an eyelid a gaze layer also touches, for instance). |
||||
|
|
||||
|
This is pure, isomorphic, and has no idea how the resulting `AnimationFrame` gets applied to a mesh — that binding (which bone/morph target each `ARKitBlendShape` name maps to in a given rig) is entirely the project's job. |
||||
|
|
||||
|
## What's not built here (on purpose) |
||||
|
|
||||
|
- The eye look-at, blink, lip-sync, and body/gesture libraries themselves — future work, built as consumers of this contract, not part of it. |
||||
|
- Mic-capture/audio-playback in the browser (still open — see `docs/REALTIME_SESSION.md`'s "What's deliberately not built yet"). |
||||
|
- Server-side viseme/phoneme generation — an open design fork (server-side, since the relay already touches raw audio, vs. client-side from the playback buffer) to decide when the lip-sync library actually gets built. |
||||
|
- Any protocol change (`$lib/core/protocol.ts` is untouched by this package) — `AvatarState` derivation works entirely from events the protocol already carries. |
||||
@ -0,0 +1,176 @@ |
|||||
|
import { describe, expect, it, vi } from 'vitest'; |
||||
|
import type { ClientEvent, ServerEvent } from '../../core/protocol.js'; |
||||
|
import { ResilientRealtimeClient, type RealtimeTransport } from './resilient-client.js'; |
||||
|
|
||||
|
/** Minimal stand-in for RealtimeClient — only what ResilientRealtimeClient touches. */ |
||||
|
class FakeTransport implements RealtimeTransport { |
||||
|
sent: ClientEvent[] = []; |
||||
|
closed = false; |
||||
|
connectCalls: string[] = []; |
||||
|
connectResult: 'resolve' | 'reject' = 'resolve'; |
||||
|
private eventHandler: ((event: ServerEvent) => void) | null = null; |
||||
|
private closeHandler: (() => void) | null = null; |
||||
|
|
||||
|
async connect(token: string): Promise<void> { |
||||
|
this.connectCalls.push(token); |
||||
|
if (this.connectResult === 'reject') throw new Error('connect failed'); |
||||
|
} |
||||
|
|
||||
|
send(event: ClientEvent): void { |
||||
|
this.sent.push(event); |
||||
|
} |
||||
|
|
||||
|
on(handler: (event: ServerEvent) => void): () => void { |
||||
|
this.eventHandler = handler; |
||||
|
return () => { |
||||
|
this.eventHandler = null; |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
onClose(handler: () => void): () => void { |
||||
|
this.closeHandler = handler; |
||||
|
return () => { |
||||
|
this.closeHandler = null; |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
close(): void { |
||||
|
this.closed = true; |
||||
|
} |
||||
|
|
||||
|
/** Simulates a ServerEvent arriving from the (fake) relay. */ |
||||
|
emit(event: ServerEvent): void { |
||||
|
this.eventHandler?.(event); |
||||
|
} |
||||
|
|
||||
|
/** Simulates the underlying socket closing, for any reason. */ |
||||
|
drop(): void { |
||||
|
this.closeHandler?.(); |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
function makeHarness( |
||||
|
opts: { |
||||
|
maxAttempts?: number; |
||||
|
baseDelayMs?: number; |
||||
|
configureTransport?: (t: FakeTransport) => void; |
||||
|
} = {} |
||||
|
) { |
||||
|
const transports: FakeTransport[] = []; |
||||
|
let tokenCount = 0; |
||||
|
const client = new ResilientRealtimeClient({ |
||||
|
getToken: async () => `token-${++tokenCount}`, |
||||
|
maxAttempts: opts.maxAttempts, |
||||
|
baseDelayMs: opts.baseDelayMs ?? 100, |
||||
|
createTransport: () => { |
||||
|
const t = new FakeTransport(); |
||||
|
opts.configureTransport?.(t); |
||||
|
transports.push(t); |
||||
|
return t; |
||||
|
} |
||||
|
}); |
||||
|
return { client, transports }; |
||||
|
} |
||||
|
|
||||
|
describe('ResilientRealtimeClient', () => { |
||||
|
it('connects, sends session.start, and proxies send()/on() to the live transport', async () => { |
||||
|
const { client, transports } = makeHarness(); |
||||
|
const states: string[] = []; |
||||
|
client.onConnectionState((s) => states.push(s)); |
||||
|
|
||||
|
await client.connect(); |
||||
|
|
||||
|
expect(transports).toHaveLength(1); |
||||
|
expect(transports[0].sent).toContainEqual({ type: 'session.start' }); |
||||
|
expect(states).toEqual(['connecting', 'open']); |
||||
|
|
||||
|
const received: ServerEvent[] = []; |
||||
|
client.on((e) => received.push(e)); |
||||
|
transports[0].emit({ type: 'session.ready', sessionId: 's1' }); |
||||
|
expect(received).toEqual([{ type: 'session.ready', sessionId: 's1' }]); |
||||
|
|
||||
|
client.send({ type: 'input_text.send', text: 'hi' }); |
||||
|
expect(transports[0].sent).toContainEqual({ type: 'input_text.send', text: 'hi' }); |
||||
|
}); |
||||
|
|
||||
|
it('does not reconnect after a clean session.closed', async () => { |
||||
|
const { client, transports } = makeHarness(); |
||||
|
const states: string[] = []; |
||||
|
client.onConnectionState((s) => states.push(s)); |
||||
|
await client.connect(); |
||||
|
|
||||
|
transports[0].emit({ type: 'session.closed', reason: 'client_requested' }); |
||||
|
// The underlying socket closing shortly after, as it does in practice —
|
||||
|
// must not be mistaken for an unexpected drop.
|
||||
|
transports[0].drop(); |
||||
|
|
||||
|
expect(states).toEqual(['connecting', 'open', 'closed']); |
||||
|
expect(transports).toHaveLength(1); |
||||
|
}); |
||||
|
|
||||
|
it('reconnects with backoff after an unexpected drop, fetching a fresh token and re-sending session.start', async () => { |
||||
|
vi.useFakeTimers(); |
||||
|
try { |
||||
|
const { client, transports } = makeHarness({ baseDelayMs: 100 }); |
||||
|
const states: string[] = []; |
||||
|
client.onConnectionState((s) => states.push(s)); |
||||
|
await client.connect(); |
||||
|
|
||||
|
transports[0].drop(); // no prior session.closed — unexpected
|
||||
|
|
||||
|
expect(states.at(-1)).toBe('reconnecting'); |
||||
|
expect(transports).toHaveLength(1); // waiting on backoff, hasn't retried yet
|
||||
|
|
||||
|
await vi.advanceTimersByTimeAsync(100); |
||||
|
|
||||
|
expect(transports).toHaveLength(2); |
||||
|
expect(transports[1].connectCalls).toEqual(['token-2']); // fresh token, not reused
|
||||
|
expect(transports[1].sent).toContainEqual({ type: 'session.start' }); |
||||
|
expect(states).toEqual(['connecting', 'open', 'reconnecting', 'open']); |
||||
|
} finally { |
||||
|
vi.useRealTimers(); |
||||
|
} |
||||
|
}); |
||||
|
|
||||
|
it('gives up after maxAttempts consecutive failures', async () => { |
||||
|
vi.useFakeTimers(); |
||||
|
try { |
||||
|
const { client, transports } = makeHarness({ |
||||
|
maxAttempts: 2, |
||||
|
baseDelayMs: 10, |
||||
|
configureTransport: (t) => { |
||||
|
t.connectResult = 'reject'; |
||||
|
} |
||||
|
}); |
||||
|
const states: string[] = []; |
||||
|
client.onConnectionState((s) => states.push(s)); |
||||
|
|
||||
|
await client.connect(); // initial attempt fails
|
||||
|
await vi.advanceTimersByTimeAsync(10); // retry 1 fails (delay = 10 * 2^0)
|
||||
|
await vi.advanceTimersByTimeAsync(20); // retry 2 fails (delay = 10 * 2^1) -> exceeds maxAttempts
|
||||
|
|
||||
|
expect(states.at(-1)).toBe('closed'); |
||||
|
expect(transports).toHaveLength(3); // initial attempt + 2 retries
|
||||
|
} finally { |
||||
|
vi.useRealTimers(); |
||||
|
} |
||||
|
}); |
||||
|
|
||||
|
it('close() tears down the transport and suppresses any pending reconnect', async () => { |
||||
|
vi.useFakeTimers(); |
||||
|
try { |
||||
|
const { client, transports } = makeHarness({ baseDelayMs: 100 }); |
||||
|
await client.connect(); |
||||
|
transports[0].drop(); |
||||
|
expect(transports[0].closed).toBe(true); |
||||
|
|
||||
|
client.close(); |
||||
|
await vi.advanceTimersByTimeAsync(1000); |
||||
|
|
||||
|
expect(transports).toHaveLength(1); // no reconnect happened after close()
|
||||
|
expect(() => client.send({ type: 'session.end' })).toThrow(); |
||||
|
} finally { |
||||
|
vi.useRealTimers(); |
||||
|
} |
||||
|
}); |
||||
|
}); |
||||
@ -0,0 +1,198 @@ |
|||||
|
import type { ClientEvent, ServerEvent } from '../../core/protocol.js'; |
||||
|
import { RealtimeClient, type RealtimeClientOptions } from './client.js'; |
||||
|
|
||||
|
export type ConnectionState = 'connecting' | 'open' | 'reconnecting' | 'closed'; |
||||
|
|
||||
|
type ServerEventHandler = (event: ServerEvent) => void; |
||||
|
type ConnectionStateHandler = (state: ConnectionState) => void; |
||||
|
|
||||
|
/** |
||||
|
* The subset of `RealtimeClient`'s surface `ResilientRealtimeClient` needs — |
||||
|
* exists so tests (and, in principle, an alternative transport) can supply a |
||||
|
* fake without touching a real WebSocket. `RealtimeClient` satisfies this |
||||
|
* structurally; no explicit `implements`. |
||||
|
*/ |
||||
|
export interface RealtimeTransport { |
||||
|
connect(token: string, options?: RealtimeClientOptions): Promise<void>; |
||||
|
send(event: ClientEvent): void; |
||||
|
on(handler: ServerEventHandler): () => void; |
||||
|
onClose(handler: () => void): () => void; |
||||
|
close(): void; |
||||
|
} |
||||
|
|
||||
|
export interface ResilientRealtimeClientOptions extends RealtimeClientOptions { |
||||
|
/** |
||||
|
* Re-fetches a fresh session token (e.g. `POST /api/realtime/session`). |
||||
|
* Required because tokens are short-lived — a stale one can't be reused |
||||
|
* across a reconnect — and because reconnecting has to happen without the |
||||
|
* caller's involvement for it to be worth wrapping in the first place. |
||||
|
*/ |
||||
|
getToken: () => Promise<string>; |
||||
|
/** Max consecutive reconnect attempts before giving up and reporting 'closed'. */ |
||||
|
maxAttempts?: number; |
||||
|
/** Base delay (ms) for exponential backoff between attempts. */ |
||||
|
baseDelayMs?: number; |
||||
|
/** For tests. Defaults to `() => new RealtimeClient()`. */ |
||||
|
createTransport?: () => RealtimeTransport; |
||||
|
} |
||||
|
|
||||
|
const DEFAULT_MAX_ATTEMPTS = 5; |
||||
|
const DEFAULT_BASE_DELAY_MS = 500; |
||||
|
|
||||
|
/** |
||||
|
* Wraps `RealtimeClient` with automatic reconnection on an *unexpected* |
||||
|
* transport drop — see docs/REALTIME_SESSION.md's "Resilience" section for |
||||
|
* the full reasoning. `RealtimeClient` itself is untouched (composition, not |
||||
|
* inheritance) and stays pure transport; `send()`/`on()`/`close()` here have |
||||
|
* the same shape, transparently proxied to whichever underlying transport is |
||||
|
* currently live across reconnects. Connection-state changes are a separate |
||||
|
* observable (`onConnectionState`) from the `ServerEvent`/protocol |
||||
|
* vocabulary — `protocol.ts` is the provider-relay contract, not the place |
||||
|
* for this transport's local retry bookkeeping. |
||||
|
* |
||||
|
* Critical design point: reconnecting the browser↔relay socket does not |
||||
|
* resurrect a dead relay↔OpenAI session. If the server proactively sends |
||||
|
* `session.closed`, that's a clean end — no auto-reconnect. Auto-reconnect |
||||
|
* only fires on an unexpected drop (the underlying socket closing with no |
||||
|
* preceding `session.closed`), and it always starts a **brand-new** session |
||||
|
* (fresh token, fresh `session.start`) — the model has no memory of the old |
||||
|
* one either way, so there's no conversation state worth trying to resume. |
||||
|
* Callers see this plainly as a `reconnecting` → `open` transition and a |
||||
|
* fresh `session.ready`, and decide for themselves how to present it (e.g. |
||||
|
* clear the transcript, show a toast) — this wrapper doesn't hide it. |
||||
|
*/ |
||||
|
export class ResilientRealtimeClient { |
||||
|
private readonly createTransport: () => RealtimeTransport; |
||||
|
private readonly maxAttempts: number; |
||||
|
private readonly baseDelayMs: number; |
||||
|
|
||||
|
private transport: RealtimeTransport | null = null; |
||||
|
private state: ConnectionState = 'closed'; |
||||
|
private attempt = 0; |
||||
|
private closedByCaller = false; |
||||
|
private reconnectTimer: ReturnType<typeof setTimeout> | null = null; |
||||
|
private stopEvents: (() => void) | null = null; |
||||
|
private stopClose: (() => void) | null = null; |
||||
|
|
||||
|
private readonly eventHandlers = new Set<ServerEventHandler>(); |
||||
|
private readonly stateHandlers = new Set<ConnectionStateHandler>(); |
||||
|
|
||||
|
constructor(private readonly options: ResilientRealtimeClientOptions) { |
||||
|
this.createTransport = options.createTransport ?? (() => new RealtimeClient()); |
||||
|
this.maxAttempts = options.maxAttempts ?? DEFAULT_MAX_ATTEMPTS; |
||||
|
this.baseDelayMs = options.baseDelayMs ?? DEFAULT_BASE_DELAY_MS; |
||||
|
} |
||||
|
|
||||
|
async connect(): Promise<void> { |
||||
|
this.closedByCaller = false; |
||||
|
this.attempt = 0; |
||||
|
await this.establish(); |
||||
|
} |
||||
|
|
||||
|
send(event: ClientEvent): void { |
||||
|
if (!this.transport) throw new Error('ResilientRealtimeClient is not connected.'); |
||||
|
this.transport.send(event); |
||||
|
} |
||||
|
|
||||
|
/** Returns an unsubscribe function. */ |
||||
|
on(handler: ServerEventHandler): () => void { |
||||
|
this.eventHandlers.add(handler); |
||||
|
return () => this.eventHandlers.delete(handler); |
||||
|
} |
||||
|
|
||||
|
/** Returns an unsubscribe function. */ |
||||
|
onConnectionState(handler: ConnectionStateHandler): () => void { |
||||
|
this.stateHandlers.add(handler); |
||||
|
return () => this.stateHandlers.delete(handler); |
||||
|
} |
||||
|
|
||||
|
close(): void { |
||||
|
this.closedByCaller = true; |
||||
|
if (this.reconnectTimer) { |
||||
|
clearTimeout(this.reconnectTimer); |
||||
|
this.reconnectTimer = null; |
||||
|
} |
||||
|
this.teardownTransport(); |
||||
|
this.setState('closed'); |
||||
|
} |
||||
|
|
||||
|
private async establish(): Promise<void> { |
||||
|
this.setState(this.attempt === 0 ? 'connecting' : 'reconnecting'); |
||||
|
|
||||
|
let token: string; |
||||
|
try { |
||||
|
token = await this.options.getToken(); |
||||
|
} catch { |
||||
|
this.handleUnexpectedDrop(); |
||||
|
return; |
||||
|
} |
||||
|
if (this.closedByCaller) return; // close() was called while awaiting the token
|
||||
|
|
||||
|
const nextTransport = this.createTransport(); |
||||
|
this.transport = nextTransport; |
||||
|
this.stopEvents = nextTransport.on((event) => { |
||||
|
this.forwardEvent(event); |
||||
|
if (event.type === 'session.closed' && this.transport === nextTransport) { |
||||
|
this.teardownTransport(); |
||||
|
this.setState('closed'); |
||||
|
} |
||||
|
}); |
||||
|
this.stopClose = nextTransport.onClose(() => { |
||||
|
if (this.transport !== nextTransport || this.state !== 'open') return; |
||||
|
this.handleUnexpectedDrop(); |
||||
|
}); |
||||
|
|
||||
|
try { |
||||
|
await nextTransport.connect(token, this.options); |
||||
|
} catch { |
||||
|
if (this.transport === nextTransport) this.handleUnexpectedDrop(); |
||||
|
return; |
||||
|
} |
||||
|
|
||||
|
// Torn down while connecting — a session.closed raced in, or close()
|
||||
|
// was called mid-handshake. Don't resurrect it as 'open'.
|
||||
|
if (this.transport !== nextTransport) return; |
||||
|
|
||||
|
this.attempt = 0; |
||||
|
this.setState('open'); |
||||
|
nextTransport.send({ type: 'session.start' }); |
||||
|
} |
||||
|
|
||||
|
private handleUnexpectedDrop(): void { |
||||
|
this.teardownTransport(); |
||||
|
if (this.closedByCaller) { |
||||
|
this.setState('closed'); |
||||
|
return; |
||||
|
} |
||||
|
this.attempt++; |
||||
|
if (this.attempt > this.maxAttempts) { |
||||
|
this.setState('closed'); |
||||
|
return; |
||||
|
} |
||||
|
this.setState('reconnecting'); |
||||
|
const delay = this.baseDelayMs * 2 ** (this.attempt - 1); |
||||
|
this.reconnectTimer = setTimeout(() => { |
||||
|
this.reconnectTimer = null; |
||||
|
void this.establish(); |
||||
|
}, delay); |
||||
|
} |
||||
|
|
||||
|
private teardownTransport(): void { |
||||
|
this.stopEvents?.(); |
||||
|
this.stopEvents = null; |
||||
|
this.stopClose?.(); |
||||
|
this.stopClose = null; |
||||
|
this.transport?.close(); |
||||
|
this.transport = null; |
||||
|
} |
||||
|
|
||||
|
private forwardEvent(event: ServerEvent): void { |
||||
|
for (const handler of this.eventHandlers) handler(event); |
||||
|
} |
||||
|
|
||||
|
private setState(state: ConnectionState): void { |
||||
|
if (this.state === state) return; |
||||
|
this.state = state; |
||||
|
for (const handler of this.stateHandlers) handler(state); |
||||
|
} |
||||
|
} |
||||
@ -0,0 +1,103 @@ |
|||||
|
import { describe, expect, it, vi } from 'vitest'; |
||||
|
import type { ServerEvent } from '../../core/protocol.js'; |
||||
|
import { deriveAvatarState, type AvatarStateSource } from './avatar-state.js'; |
||||
|
|
||||
|
/** Minimal stand-in for RealtimeClient/ResilientRealtimeClient — only .on() is used. */ |
||||
|
class FakeSource implements AvatarStateSource { |
||||
|
private handler: ((event: ServerEvent) => void) | null = null; |
||||
|
|
||||
|
on(handler: (event: ServerEvent) => void): () => void { |
||||
|
this.handler = handler; |
||||
|
return () => { |
||||
|
this.handler = null; |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
emit(event: ServerEvent): void { |
||||
|
this.handler?.(event); |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
describe('deriveAvatarState', () => { |
||||
|
it('subscribe() emits the current state immediately', () => { |
||||
|
const store = deriveAvatarState(new FakeSource()); |
||||
|
const run = vi.fn(); |
||||
|
store.subscribe(run); |
||||
|
expect(run).toHaveBeenCalledTimes(1); |
||||
|
expect(run).toHaveBeenCalledWith('idle'); |
||||
|
}); |
||||
|
|
||||
|
it('walks listening -> thinking -> talking -> idle across a full voice turn', () => { |
||||
|
const source = new FakeSource(); |
||||
|
const store = deriveAvatarState(source); |
||||
|
const seen: string[] = []; |
||||
|
store.subscribe((s) => seen.push(s)); |
||||
|
|
||||
|
store.setListening(true); |
||||
|
store.setListening(false); |
||||
|
source.emit({ type: 'output_audio.chunk', audio: '' }); |
||||
|
source.emit({ type: 'output_audio.done' }); |
||||
|
|
||||
|
expect(seen).toEqual(['idle', 'listening', 'thinking', 'talking', 'idle']); |
||||
|
}); |
||||
|
|
||||
|
it('notifyResponseRequested() drives thinking for the typed-text path', () => { |
||||
|
const source = new FakeSource(); |
||||
|
const store = deriveAvatarState(source); |
||||
|
const seen: string[] = []; |
||||
|
store.subscribe((s) => seen.push(s)); |
||||
|
|
||||
|
store.notifyResponseRequested(); |
||||
|
source.emit({ type: 'transcript.delta', role: 'assistant', text: 'hi' }); |
||||
|
source.emit({ type: 'transcript.done', role: 'assistant' }); |
||||
|
|
||||
|
expect(seen).toEqual(['idle', 'thinking', 'talking', 'idle']); |
||||
|
}); |
||||
|
|
||||
|
it('ignores user transcript events (no signal either way)', () => { |
||||
|
const source = new FakeSource(); |
||||
|
const store = deriveAvatarState(source); |
||||
|
const seen: string[] = []; |
||||
|
store.subscribe((s) => seen.push(s)); |
||||
|
|
||||
|
source.emit({ type: 'transcript.delta', role: 'user', text: 'hello' }); |
||||
|
|
||||
|
expect(seen).toEqual(['idle']); |
||||
|
}); |
||||
|
|
||||
|
it('session.ready and session.closed reset to idle from any state', () => { |
||||
|
const source = new FakeSource(); |
||||
|
const store = deriveAvatarState(source); |
||||
|
const seen: string[] = []; |
||||
|
store.subscribe((s) => seen.push(s)); |
||||
|
|
||||
|
source.emit({ type: 'output_audio.chunk', audio: '' }); // -> talking
|
||||
|
source.emit({ type: 'session.closed', reason: 'client_requested' }); |
||||
|
|
||||
|
expect(seen).toEqual(['idle', 'talking', 'idle']); |
||||
|
}); |
||||
|
|
||||
|
it('response.done is a fallback that stops talking even without output_audio.done', () => { |
||||
|
const source = new FakeSource(); |
||||
|
const store = deriveAvatarState(source); |
||||
|
const seen: string[] = []; |
||||
|
store.subscribe((s) => seen.push(s)); |
||||
|
|
||||
|
source.emit({ type: 'output_audio.chunk', audio: '' }); |
||||
|
source.emit({ type: 'response.done' }); |
||||
|
|
||||
|
expect(seen).toEqual(['idle', 'talking', 'idle']); |
||||
|
}); |
||||
|
|
||||
|
it('destroy() stops reacting to further events', () => { |
||||
|
const source = new FakeSource(); |
||||
|
const store = deriveAvatarState(source); |
||||
|
const seen: string[] = []; |
||||
|
store.subscribe((s) => seen.push(s)); |
||||
|
|
||||
|
store.destroy(); |
||||
|
source.emit({ type: 'output_audio.chunk', audio: '' }); |
||||
|
|
||||
|
expect(seen).toEqual(['idle']); |
||||
|
}); |
||||
|
}); |
||||
@ -0,0 +1,98 @@ |
|||||
|
import type { ServerEvent } from '../../core/protocol.js'; |
||||
|
import { nextAvatarState, type AvatarSignal, type AvatarState } from '../../humanoid/state.js'; |
||||
|
|
||||
|
/** |
||||
|
* The subset of `RealtimeClient`'s (or `ResilientRealtimeClient`'s) surface |
||||
|
* `deriveAvatarState` needs — accepted structurally so this package doesn't |
||||
|
* care which transport variant is in use. |
||||
|
*/ |
||||
|
export interface AvatarStateSource { |
||||
|
on(handler: (event: ServerEvent) => void): () => void; |
||||
|
} |
||||
|
|
||||
|
export interface AvatarStateStore { |
||||
|
/** Svelte-store-compatible: calls `run` immediately with the current value, then on every change. */ |
||||
|
subscribe(run: (state: AvatarState) => void): () => void; |
||||
|
/** |
||||
|
* Called by whatever mic-capture code a project uses (that pipeline |
||||
|
* doesn't exist in core yet — see docs/HUMANOID.md's non-goals). The |
||||
|
* client, not the server, is the only thing that currently knows when |
||||
|
* the user starts/stops talking: the protocol doesn't surface OpenAI's |
||||
|
* `input_audio_buffer.speech_*` VAD events — see the "not part of our |
||||
|
* normalized vocabulary" comment in |
||||
|
* `server/core/providers/openai.ts`'s `fromOpenAiEvent`. |
||||
|
*/ |
||||
|
setListening(active: boolean): void; |
||||
|
/** |
||||
|
* Call right when sending a request the server-event stream alone won't |
||||
|
* announce — chiefly a typed `input_text.send` (the voice path already |
||||
|
* gets this for free from `setListening(false)`). |
||||
|
*/ |
||||
|
notifyResponseRequested(): void; |
||||
|
/** Stops listening to the source and releases all subscribers. */ |
||||
|
destroy(): void; |
||||
|
} |
||||
|
|
||||
|
/** |
||||
|
* Derives a reactive `AvatarState` from a live `RealtimeClient` or |
||||
|
* `ResilientRealtimeClient`. Translates `ServerEvent`s (plus the imperative |
||||
|
* calls above, for signals the event stream alone can't carry) into |
||||
|
* `AvatarSignal`s and folds them through the pure `nextAvatarState` reducer |
||||
|
* from `$lib/humanoid/state.ts`. |
||||
|
* |
||||
|
* `thinking` is an explicit heuristic — the time between a request and the |
||||
|
* assistant's first output — not a precise reflection of server-side |
||||
|
* activity: the browser never sees server-executed tool calls at all (see |
||||
|
* docs/TOOLS.md), so "the model is running a tool" isn't observable here. |
||||
|
*/ |
||||
|
export function deriveAvatarState(source: AvatarStateSource): AvatarStateStore { |
||||
|
let state: AvatarState = 'idle'; |
||||
|
const subscribers = new Set<(state: AvatarState) => void>(); |
||||
|
|
||||
|
function apply(signal: AvatarSignal): void { |
||||
|
const next = nextAvatarState(state, signal); |
||||
|
if (next === state) return; |
||||
|
state = next; |
||||
|
for (const run of subscribers) run(state); |
||||
|
} |
||||
|
|
||||
|
const stopEvents = source.on((event) => { |
||||
|
switch (event.type) { |
||||
|
case 'session.ready': |
||||
|
case 'session.closed': |
||||
|
apply({ type: 'reset' }); |
||||
|
break; |
||||
|
case 'output_audio.chunk': |
||||
|
apply({ type: 'talking_started' }); |
||||
|
break; |
||||
|
case 'transcript.delta': |
||||
|
if (event.role === 'assistant') apply({ type: 'talking_started' }); |
||||
|
break; |
||||
|
case 'output_audio.done': |
||||
|
case 'response.done': |
||||
|
apply({ type: 'talking_stopped' }); |
||||
|
break; |
||||
|
case 'transcript.done': |
||||
|
if (event.role === 'assistant') apply({ type: 'talking_stopped' }); |
||||
|
break; |
||||
|
} |
||||
|
}); |
||||
|
|
||||
|
return { |
||||
|
subscribe(run) { |
||||
|
subscribers.add(run); |
||||
|
run(state); |
||||
|
return () => subscribers.delete(run); |
||||
|
}, |
||||
|
setListening(active) { |
||||
|
apply({ type: active ? 'listening_started' : 'listening_stopped' }); |
||||
|
}, |
||||
|
notifyResponseRequested() { |
||||
|
apply({ type: 'response_requested' }); |
||||
|
}, |
||||
|
destroy() { |
||||
|
stopEvents(); |
||||
|
subscribers.clear(); |
||||
|
} |
||||
|
}; |
||||
|
} |
||||
@ -0,0 +1,52 @@ |
|||||
|
import { describe, expect, it } from 'vitest'; |
||||
|
import { compositeAnimationFrames } from './animation.js'; |
||||
|
|
||||
|
describe('compositeAnimationFrames', () => { |
||||
|
it('sums add-mode layers onto the same channel', () => { |
||||
|
// 0.25 + 0.25, not 0.2 + 0.1, so the sum is exactly representable in
|
||||
|
// binary floating point and this isn't a disguised precision test.
|
||||
|
expect( |
||||
|
compositeAnimationFrames([ |
||||
|
{ frame: { eyeLookInLeft: 0.25 }, mode: 'add' }, |
||||
|
{ frame: { eyeLookInLeft: 0.25 }, mode: 'add' } |
||||
|
]) |
||||
|
).toEqual({ eyeLookInLeft: 0.5 }); |
||||
|
}); |
||||
|
|
||||
|
it('override replaces rather than sums, later layers winning', () => { |
||||
|
expect( |
||||
|
compositeAnimationFrames([ |
||||
|
{ frame: { eyeBlinkLeft: 0.1 }, mode: 'add' }, |
||||
|
{ frame: { eyeBlinkLeft: 1 }, mode: 'override' } |
||||
|
]) |
||||
|
).toEqual({ eyeBlinkLeft: 1 }); |
||||
|
}); |
||||
|
|
||||
|
it('applies a layer weight multiplier before composing', () => { |
||||
|
expect( |
||||
|
compositeAnimationFrames([{ frame: { jawOpen: 0.5 }, mode: 'add', weight: 0.5 }]) |
||||
|
).toEqual({ jawOpen: 0.25 }); |
||||
|
}); |
||||
|
|
||||
|
it('clamps summed channels to [0, 1]', () => { |
||||
|
expect( |
||||
|
compositeAnimationFrames([ |
||||
|
{ frame: { mouthSmileLeft: 0.8 }, mode: 'add' }, |
||||
|
{ frame: { mouthSmileLeft: 0.8 }, mode: 'add' } |
||||
|
]) |
||||
|
).toEqual({ mouthSmileLeft: 1 }); |
||||
|
}); |
||||
|
|
||||
|
it('leaves untouched channels from other layers alone', () => { |
||||
|
expect( |
||||
|
compositeAnimationFrames([ |
||||
|
{ frame: { eyeBlinkLeft: 1 }, mode: 'override' }, |
||||
|
{ frame: { jawOpen: 0.4 }, mode: 'add' } |
||||
|
]) |
||||
|
).toEqual({ eyeBlinkLeft: 1, jawOpen: 0.4 }); |
||||
|
}); |
||||
|
|
||||
|
it('returns an empty frame for no layers', () => { |
||||
|
expect(compositeAnimationFrames([])).toEqual({}); |
||||
|
}); |
||||
|
}); |
||||
@ -0,0 +1,114 @@ |
|||||
|
/** |
||||
|
* Isomorphic, renderer-agnostic animation-channel contract — the other half |
||||
|
* of the optional `humanoid` package alongside `state.ts` (see |
||||
|
* docs/HUMANOID.md). Defines a shared channel vocabulary (the standard |
||||
|
* ARKit blend shapes most 3D humanoid rigs in this project family share) and |
||||
|
* a compositor so independently-built animation sources (a future eye |
||||
|
* look-at library, blink, lip-sync, body/gesture) can layer onto the same |
||||
|
* frame without colliding — mirrors the additive-layers pattern from the |
||||
|
* eye look-at design doc (Macro/Micro/Nano summed into one gaze offset), |
||||
|
* generalized to arbitrary channels and to layers that override rather than |
||||
|
* sum. No rendering, no bone/mesh knowledge — a project's own rig-binding |
||||
|
* code is what actually applies a composited `AnimationFrame` to a mesh. |
||||
|
*/ |
||||
|
|
||||
|
/** The 52 standard ARKit face blend shapes (`ARFaceAnchor.BlendShapeLocation`). */ |
||||
|
export type ARKitBlendShape = |
||||
|
| 'browDownLeft' |
||||
|
| 'browDownRight' |
||||
|
| 'browInnerUp' |
||||
|
| 'browOuterUpLeft' |
||||
|
| 'browOuterUpRight' |
||||
|
| 'cheekPuff' |
||||
|
| 'cheekSquintLeft' |
||||
|
| 'cheekSquintRight' |
||||
|
| 'eyeBlinkLeft' |
||||
|
| 'eyeBlinkRight' |
||||
|
| 'eyeLookDownLeft' |
||||
|
| 'eyeLookDownRight' |
||||
|
| 'eyeLookInLeft' |
||||
|
| 'eyeLookInRight' |
||||
|
| 'eyeLookOutLeft' |
||||
|
| 'eyeLookOutRight' |
||||
|
| 'eyeLookUpLeft' |
||||
|
| 'eyeLookUpRight' |
||||
|
| 'eyeSquintLeft' |
||||
|
| 'eyeSquintRight' |
||||
|
| 'eyeWideLeft' |
||||
|
| 'eyeWideRight' |
||||
|
| 'jawForward' |
||||
|
| 'jawLeft' |
||||
|
| 'jawOpen' |
||||
|
| 'jawRight' |
||||
|
| 'mouthClose' |
||||
|
| 'mouthDimpleLeft' |
||||
|
| 'mouthDimpleRight' |
||||
|
| 'mouthFrownLeft' |
||||
|
| 'mouthFrownRight' |
||||
|
| 'mouthFunnel' |
||||
|
| 'mouthLeft' |
||||
|
| 'mouthLowerDownLeft' |
||||
|
| 'mouthLowerDownRight' |
||||
|
| 'mouthPressLeft' |
||||
|
| 'mouthPressRight' |
||||
|
| 'mouthPucker' |
||||
|
| 'mouthRight' |
||||
|
| 'mouthRollLower' |
||||
|
| 'mouthRollUpper' |
||||
|
| 'mouthShrugLower' |
||||
|
| 'mouthShrugUpper' |
||||
|
| 'mouthSmileLeft' |
||||
|
| 'mouthSmileRight' |
||||
|
| 'mouthStretchLeft' |
||||
|
| 'mouthStretchRight' |
||||
|
| 'mouthUpperUpLeft' |
||||
|
| 'mouthUpperUpRight' |
||||
|
| 'noseSneerLeft' |
||||
|
| 'noseSneerRight' |
||||
|
| 'tongueOut'; |
||||
|
|
||||
|
/** Sparse channel → weight (0–1) map for a single frame. */ |
||||
|
export type AnimationFrame = Partial<Record<ARKitBlendShape, number>>; |
||||
|
|
||||
|
export interface AnimationLayer { |
||||
|
frame: AnimationFrame; |
||||
|
/** |
||||
|
* `'add'` sums onto whatever the channel already holds (the eye look-at |
||||
|
* doc's Macro/Micro/Nano pattern — several small offsets combining into |
||||
|
* one). `'override'` replaces the channel outright — for layers whose |
||||
|
* values can't be meaningfully summed (e.g. a blink closing an eyelid |
||||
|
* that a gaze layer also touches). |
||||
|
*/ |
||||
|
mode: 'add' | 'override'; |
||||
|
/** Multiplies every weight in `frame` before composing. Defaults to 1. */ |
||||
|
weight?: number; |
||||
|
} |
||||
|
|
||||
|
/** |
||||
|
* Folds `layers` in order into one `AnimationFrame`, clamping each channel |
||||
|
* to [0, 1] at the end. Later layers in the array apply after earlier ones — |
||||
|
* order matters for `'override'` layers, not for `'add'` ones. |
||||
|
*/ |
||||
|
export function compositeAnimationFrames(layers: AnimationLayer[]): AnimationFrame { |
||||
|
const result: AnimationFrame = {}; |
||||
|
|
||||
|
for (const layer of layers) { |
||||
|
const weight = layer.weight ?? 1; |
||||
|
for (const [channel, value] of Object.entries(layer.frame) as Array< |
||||
|
[ARKitBlendShape, number] |
||||
|
>) { |
||||
|
const scaled = value * weight; |
||||
|
if (layer.mode === 'override') { |
||||
|
result[channel] = scaled; |
||||
|
} else { |
||||
|
result[channel] = (result[channel] ?? 0) + scaled; |
||||
|
} |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
for (const channel of Object.keys(result) as ARKitBlendShape[]) { |
||||
|
result[channel] = Math.min(1, Math.max(0, result[channel]!)); |
||||
|
} |
||||
|
|
||||
|
return result; |
||||
|
} |
||||
@ -0,0 +1,34 @@ |
|||||
|
import { describe, expect, it } from 'vitest'; |
||||
|
import { nextAvatarState } from './state.js'; |
||||
|
|
||||
|
describe('nextAvatarState', () => { |
||||
|
it('listening_started always moves to listening', () => { |
||||
|
expect(nextAvatarState('idle', { type: 'listening_started' })).toBe('listening'); |
||||
|
expect(nextAvatarState('talking', { type: 'listening_started' })).toBe('listening'); |
||||
|
}); |
||||
|
|
||||
|
it('listening_stopped moves to thinking (a response is expected next), only from listening', () => { |
||||
|
expect(nextAvatarState('listening', { type: 'listening_stopped' })).toBe('thinking'); |
||||
|
expect(nextAvatarState('idle', { type: 'listening_stopped' })).toBe('idle'); |
||||
|
}); |
||||
|
|
||||
|
it('response_requested moves to thinking unless already talking', () => { |
||||
|
expect(nextAvatarState('idle', { type: 'response_requested' })).toBe('thinking'); |
||||
|
expect(nextAvatarState('listening', { type: 'response_requested' })).toBe('thinking'); |
||||
|
expect(nextAvatarState('talking', { type: 'response_requested' })).toBe('talking'); |
||||
|
}); |
||||
|
|
||||
|
it('talking_started always moves to talking', () => { |
||||
|
expect(nextAvatarState('thinking', { type: 'talking_started' })).toBe('talking'); |
||||
|
}); |
||||
|
|
||||
|
it('talking_stopped only returns to idle from talking', () => { |
||||
|
expect(nextAvatarState('talking', { type: 'talking_stopped' })).toBe('idle'); |
||||
|
expect(nextAvatarState('idle', { type: 'talking_stopped' })).toBe('idle'); |
||||
|
}); |
||||
|
|
||||
|
it('reset always returns to idle', () => { |
||||
|
expect(nextAvatarState('talking', { type: 'reset' })).toBe('idle'); |
||||
|
expect(nextAvatarState('listening', { type: 'reset' })).toBe('idle'); |
||||
|
}); |
||||
|
}); |
||||
@ -0,0 +1,54 @@ |
|||||
|
/** |
||||
|
* Isomorphic, renderer-agnostic avatar state — part of the optional |
||||
|
* `humanoid` package (see docs/HUMANOID.md and the "Module layering" section |
||||
|
* of docs/ARCHITECTURE.md for what that means). Pure types and a pure |
||||
|
* reducer, no WS/browser dependency, so future animation libraries (eye |
||||
|
* look-at, blink, lip-sync, body/gesture) can all key off the same signal |
||||
|
* without each re-deriving turn-taking themselves. |
||||
|
* |
||||
|
* Deliberately just four states, not the `IdleActive`/`IdlePassive` split a |
||||
|
* humanoid rig's own animation design may eventually want — that |
||||
|
* distinction needs a user-presence signal (webcam/face-tracking) this |
||||
|
* package has no opinion on. A project with that signal can layer its own |
||||
|
* finer-grained idle split on top; `idle` here just means "no signal that |
||||
|
* the user or the model is currently doing anything." |
||||
|
*/ |
||||
|
export type AvatarState = 'idle' | 'listening' | 'thinking' | 'talking'; |
||||
|
|
||||
|
export type AvatarSignal = |
||||
|
| { type: 'listening_started' } |
||||
|
| { type: 'listening_stopped' } |
||||
|
| { type: 'response_requested' } |
||||
|
| { type: 'talking_started' } |
||||
|
| { type: 'talking_stopped' } |
||||
|
| { type: 'reset' }; |
||||
|
|
||||
|
/** |
||||
|
* Pure state transition — no side effects, directly unit-testable. The |
||||
|
* browser-facing translation from real `ServerEvent`s (and the imperative |
||||
|
* `setListening` a project's mic-capture code calls) into `AvatarSignal`s |
||||
|
* lives in `$lib/client/humanoid/avatar-state.ts`, which folds them through |
||||
|
* this function; this module doesn't know a `RealtimeClient` exists. |
||||
|
*/ |
||||
|
export function nextAvatarState(current: AvatarState, signal: AvatarSignal): AvatarState { |
||||
|
switch (signal.type) { |
||||
|
case 'listening_started': |
||||
|
return 'listening'; |
||||
|
// Ending listening almost always means a turn just completed and a
|
||||
|
// response is expected next — go straight to 'thinking' rather than
|
||||
|
// 'idle', so the avatar doesn't visibly relax for the split second
|
||||
|
// before the model's response starts arriving.
|
||||
|
case 'listening_stopped': |
||||
|
return current === 'listening' ? 'thinking' : current; |
||||
|
case 'response_requested': |
||||
|
return current === 'talking' ? current : 'thinking'; |
||||
|
case 'talking_started': |
||||
|
return 'talking'; |
||||
|
case 'talking_stopped': |
||||
|
return current === 'talking' ? 'idle' : current; |
||||
|
case 'reset': |
||||
|
return 'idle'; |
||||
|
default: |
||||
|
return current; |
||||
|
} |
||||
|
} |
||||
@ -0,0 +1,19 @@ |
|||||
|
import type { MockScript } from '../mock.js'; |
||||
|
|
||||
|
/** |
||||
|
* Short canned exchange for exercising transcript rendering and (via |
||||
|
* output_audio.chunk/done timing) humanoid avatar-state derivation without a |
||||
|
* real API key. `audio` is an empty string — nothing decodes/plays it yet (no |
||||
|
* client-side audio capture/playback pipeline exists in core today), only its |
||||
|
* start/stop timing matters here. |
||||
|
*/ |
||||
|
export const DEFAULT_MOCK_SCRIPT: MockScript = [ |
||||
|
{ delayMs: 300, event: { type: 'output_audio.chunk', audio: '' } }, |
||||
|
{ |
||||
|
delayMs: 200, |
||||
|
event: { type: 'transcript.delta', role: 'assistant', text: "Hey, I'm a mock avatar reply." } |
||||
|
}, |
||||
|
{ delayMs: 1200, event: { type: 'transcript.done', role: 'assistant' } }, |
||||
|
{ delayMs: 0, event: { type: 'output_audio.done' } }, |
||||
|
{ delayMs: 0, event: { type: 'response.done' } } |
||||
|
]; |
||||
@ -0,0 +1,17 @@ |
|||||
|
import type { AppConfig } from '../config.js'; |
||||
|
import type { RealtimeProvider } from '../provider.js'; |
||||
|
import { DEFAULT_MOCK_SCRIPT } from './fixtures/default.js'; |
||||
|
import { MockRealtimeProvider } from './mock.js'; |
||||
|
import { OpenAiRealtimeProvider } from './openai.js'; |
||||
|
|
||||
|
/** |
||||
|
* Picks the `RealtimeProvider` implementation for `config.realtimeProvider` |
||||
|
* — the one place both composition roots (`server.js`, `dev-plugin.ts`) call, |
||||
|
* instead of each hardcoding `new OpenAiRealtimeProvider(...)` directly. |
||||
|
*/ |
||||
|
export function selectProvider(config: AppConfig): RealtimeProvider { |
||||
|
if (config.realtimeProvider === 'mock') { |
||||
|
return new MockRealtimeProvider(DEFAULT_MOCK_SCRIPT); |
||||
|
} |
||||
|
return new OpenAiRealtimeProvider({ apiKey: config.openaiApiKey }); |
||||
|
} |
||||
@ -0,0 +1,56 @@ |
|||||
|
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; |
||||
|
import type { ServerEvent } from '../../../core/protocol.js'; |
||||
|
import { MockRealtimeProvider, type MockScript } from './mock.js'; |
||||
|
|
||||
|
const SCRIPT: MockScript = [ |
||||
|
{ delayMs: 100, event: { type: 'transcript.delta', role: 'assistant', text: 'hi' } }, |
||||
|
{ delayMs: 50, event: { type: 'transcript.done', role: 'assistant' } } |
||||
|
]; |
||||
|
|
||||
|
beforeEach(() => { |
||||
|
vi.useFakeTimers(); |
||||
|
}); |
||||
|
|
||||
|
afterEach(() => { |
||||
|
vi.useRealTimers(); |
||||
|
}); |
||||
|
|
||||
|
describe('MockRealtimeProvider', () => { |
||||
|
it('replays the script in order, at the cumulative scheduled delay', async () => { |
||||
|
const provider = new MockRealtimeProvider(SCRIPT); |
||||
|
const session = await provider.createSession(); |
||||
|
const received: ServerEvent[] = []; |
||||
|
session.onEvent((event) => received.push(event)); |
||||
|
|
||||
|
await vi.advanceTimersByTimeAsync(99); |
||||
|
expect(received).toEqual([]); |
||||
|
|
||||
|
await vi.advanceTimersByTimeAsync(1); // total 100ms — first step fires
|
||||
|
expect(received).toEqual([{ type: 'transcript.delta', role: 'assistant', text: 'hi' }]); |
||||
|
|
||||
|
await vi.advanceTimersByTimeAsync(50); // total 150ms — second step fires
|
||||
|
expect(received).toEqual([ |
||||
|
{ type: 'transcript.delta', role: 'assistant', text: 'hi' }, |
||||
|
{ type: 'transcript.done', role: 'assistant' } |
||||
|
]); |
||||
|
}); |
||||
|
|
||||
|
it('close() cancels any not-yet-fired steps', async () => { |
||||
|
const provider = new MockRealtimeProvider(SCRIPT); |
||||
|
const session = await provider.createSession(); |
||||
|
const received: ServerEvent[] = []; |
||||
|
session.onEvent((event) => received.push(event)); |
||||
|
|
||||
|
session.close(); |
||||
|
await vi.advanceTimersByTimeAsync(1000); |
||||
|
|
||||
|
expect(received).toEqual([]); |
||||
|
}); |
||||
|
|
||||
|
it('send() is a no-op — the script does not react to browser input', async () => { |
||||
|
const provider = new MockRealtimeProvider(SCRIPT); |
||||
|
const session = await provider.createSession(); |
||||
|
|
||||
|
expect(() => session.send({ type: 'input_text.send', text: 'hello' })).not.toThrow(); |
||||
|
}); |
||||
|
}); |
||||
@ -0,0 +1,54 @@ |
|||||
|
import type { ServerEvent } from '../../../core/protocol.js'; |
||||
|
import type { ProviderSession, RealtimeProvider } from '../provider.js'; |
||||
|
|
||||
|
/** One scripted event, replayed `delayMs` after the previous step (or after `createSession` for the first step). */ |
||||
|
export interface MockScriptStep { |
||||
|
delayMs: number; |
||||
|
event: ServerEvent; |
||||
|
} |
||||
|
|
||||
|
export type MockScript = MockScriptStep[]; |
||||
|
|
||||
|
/** |
||||
|
* `RealtimeProvider` that replays a canned `MockScript` on a timer instead of |
||||
|
* talking to a real upstream — lets client work (transcript rendering, avatar |
||||
|
* state, animation timing) happen without an API key or live usage. See |
||||
|
* docs/REALTIME_SESSION.md's "Development without a live API" section. |
||||
|
* |
||||
|
* Deliberately simple: linear, one-shot replay. `send()` is a no-op — the |
||||
|
* script doesn't react to what the browser sends. Interactive/branching |
||||
|
* fixtures are a plausible future enhancement, not needed yet. |
||||
|
*/ |
||||
|
export class MockRealtimeProvider implements RealtimeProvider { |
||||
|
constructor(private readonly script: MockScript) {} |
||||
|
|
||||
|
// Ignores `config` — the script is canned, not generated from session config.
|
||||
|
async createSession(): Promise<ProviderSession> { |
||||
|
return new MockProviderSession(this.script); |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
class MockProviderSession implements ProviderSession { |
||||
|
private handler: ((event: ServerEvent) => void) | null = null; |
||||
|
private readonly timers: NodeJS.Timeout[] = []; |
||||
|
|
||||
|
constructor(script: MockScript) { |
||||
|
let elapsed = 0; |
||||
|
for (const step of script) { |
||||
|
elapsed += step.delayMs; |
||||
|
this.timers.push(setTimeout(() => this.handler?.(step.event), elapsed)); |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// No-op — see the class doc comment on MockRealtimeProvider.
|
||||
|
send(): void {} |
||||
|
|
||||
|
onEvent(handler: (event: ServerEvent) => void): void { |
||||
|
this.handler = handler; |
||||
|
} |
||||
|
|
||||
|
close(): void { |
||||
|
for (const timer of this.timers) clearTimeout(timer); |
||||
|
this.timers.length = 0; |
||||
|
} |
||||
|
} |
||||
@ -0,0 +1,107 @@ |
|||||
|
<script lang="ts"> |
||||
|
import { |
||||
|
ResilientRealtimeClient, |
||||
|
type ConnectionState |
||||
|
} from '$lib/client/core/resilient-client.js'; |
||||
|
import { deriveAvatarState, type AvatarStateStore } from '$lib/client/humanoid/avatar-state.js'; |
||||
|
import type { ServerEvent } from '$lib/core/protocol.js'; |
||||
|
import type { AvatarState } from '$lib/humanoid/state.js'; |
||||
|
|
||||
|
type Status = 'idle' | 'connecting' | 'reconnecting' | 'ready' | 'error' | 'closed'; |
||||
|
|
||||
|
let status = $state<Status>('idle'); |
||||
|
let statusMessage = $state(''); |
||||
|
let avatarState = $state<AvatarState>('idle'); |
||||
|
let listening = $state(false); |
||||
|
|
||||
|
let client: ResilientRealtimeClient | null = null; |
||||
|
let stateStore: AvatarStateStore | null = null; |
||||
|
let stopStateSub: (() => void) | null = null; |
||||
|
|
||||
|
async function fetchToken(): Promise<string> { |
||||
|
const res = await fetch('/api/realtime/session', { method: 'POST' }); |
||||
|
if (!res.ok) throw new Error(`Failed to mint session token (${res.status})`); |
||||
|
const { token } = await res.json(); |
||||
|
return token; |
||||
|
} |
||||
|
|
||||
|
async function connect() { |
||||
|
status = 'connecting'; |
||||
|
statusMessage = ''; |
||||
|
try { |
||||
|
const nextClient = new ResilientRealtimeClient({ getToken: fetchToken }); |
||||
|
nextClient.onConnectionState(handleConnectionState); |
||||
|
nextClient.on(handleServerEvent); |
||||
|
await nextClient.connect(); |
||||
|
client = nextClient; |
||||
|
|
||||
|
// See $lib/humanoid — this is the "humanoid" optional package's |
||||
|
// contract layer, not tied to any renderer/UI. deriveAvatarState |
||||
|
// takes the client structurally (RealtimeClient or |
||||
|
// ResilientRealtimeClient both work). |
||||
|
const store = deriveAvatarState(nextClient); |
||||
|
stateStore = store; |
||||
|
stopStateSub = store.subscribe((s) => (avatarState = s)); |
||||
|
} catch (err) { |
||||
|
status = 'error'; |
||||
|
statusMessage = err instanceof Error ? err.message : 'Failed to connect.'; |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
function handleConnectionState(connState: ConnectionState) { |
||||
|
if (connState === 'connecting') { |
||||
|
status = 'connecting'; |
||||
|
} else if (connState === 'reconnecting') { |
||||
|
status = 'reconnecting'; |
||||
|
statusMessage = 'Connection lost — reconnecting…'; |
||||
|
} else if (connState === 'closed' && status !== 'error') { |
||||
|
status = 'closed'; |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
function handleServerEvent(event: ServerEvent) { |
||||
|
if (event.type === 'session.ready') { |
||||
|
status = 'ready'; |
||||
|
statusMessage = ''; |
||||
|
} else if (event.type === 'error') { |
||||
|
status = 'error'; |
||||
|
statusMessage = event.message; |
||||
|
} else if (event.type === 'session.closed') { |
||||
|
statusMessage = event.reason; |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
function toggleListening() { |
||||
|
listening = !listening; |
||||
|
stateStore?.setListening(listening); |
||||
|
} |
||||
|
|
||||
|
function disconnect() { |
||||
|
client?.send({ type: 'session.end' }); |
||||
|
client?.close(); |
||||
|
client = null; |
||||
|
stopStateSub?.(); |
||||
|
stopStateSub = null; |
||||
|
stateStore?.destroy(); |
||||
|
stateStore = null; |
||||
|
listening = false; |
||||
|
status = 'closed'; |
||||
|
} |
||||
|
</script> |
||||
|
|
||||
|
<h1>Humanoid avatar-state demo</h1> |
||||
|
<p> |
||||
|
Exercises <code>$lib/humanoid</code>'s state derivation without a live API key: run |
||||
|
<code>REALTIME_PROVIDER=mock npm run dev</code> (dev-server restart required — see AGENTS.md's HMR |
||||
|
gotcha) and connect below to watch <code>avatarState</code> move through the mock provider's default |
||||
|
fixture. |
||||
|
</p> |
||||
|
<p>Status: {status}{statusMessage ? ` — ${statusMessage}` : ''}</p> |
||||
|
<p>Avatar state: <strong>{avatarState}</strong></p> |
||||
|
|
||||
|
{#if status === 'idle' || status === 'error' || status === 'closed'} |
||||
|
<button onclick={connect}>Connect</button> |
||||
|
{:else if status === 'ready'} |
||||
|
<button onclick={disconnect}>Disconnect</button> |
||||
|
<button onclick={toggleListening}>{listening ? 'Stop listening' : 'Start listening'}</button> |
||||
|
{/if} |
||||
Loading…
Reference in new issue