webavatar-core/docs/HUMANOID.md

Humanoid package

humanoid is the first instance of the optional package convention (see "Module layering" in ARCHITECTURE.md) — a renderer-agnostic contract layer for 3D humanoid avatars rigged with ARKit blend shapes, scoped out of core because a 2D or non-humanoid project has no use for it. Nothing in core or project imports from it; a project opts in by importing directly from $lib/humanoid / $lib/client/humanoid, and if nothing imports it, it's simply not part of that project's bundle.

This package ships the contract future animation libraries (eye look-at, blink, lip-sync, body/gesture) plug into — not the libraries themselves, and not a renderer. It has no idea what's rendering the avatar (Three.js, Babylon, a game engine WebGL bridge, anything else) — that stays entirely project-owned, per the base project's founding constraint that appearance/UI is never core's job.

$lib/humanoid/state.ts — avatar state

type AvatarState = 'idleActive' | 'idlePassive' | 'listening' | 'thinking' | 'talking';

idle is split into idleActive/idlePassive from the start, since ported animation systems (this one's lineage traces to an Unreal Engine avatar app) commonly key their idle behavior off exactly this distinction: idleActive right after a turn ends (still "engaged"), settling into idlePassive once nothing has happened for a while (bored/ambient idle animation). The reducer itself has no timers — pure and isomorphic — so the active→passive transition rides in on an explicit idle_timeout signal; the client wrapper (deriveAvatarState, below) owns the actual clock, same as it owns setListening.

nextAvatarState(current, signal) is a pure reducer — no WS/browser dependency, directly unit-tested in state.spec.ts. The signal set is small and closed (listening_started/listening_stopped, response_requested, talking_started/talking_stopped, idle_timeout, reset).

Note listening_stopped moves to thinking, not back to idle — ending listening almost always means a turn just completed and a response is expected next, so the avatar shouldn't visibly relax for the split second before the model's response starts arriving. talking_stopped and reset both land on idleActive, never straight to idlePassive — passive idle is only ever reached via idle_timeout.

$lib/client/humanoid/avatar-state.ts — deriving it live

deriveAvatarState(source, options?) takes a RealtimeClient or ResilientRealtimeClient (accepted structurally, via an AvatarStateSource interface — this package doesn't care which transport variant is in use) and folds its ServerEvents through nextAvatarState:

ServerEvent	Signal
session.ready / session.closed	reset
output_audio.chunk	talking_started
transcript.delta (role: assistant)	talking_started
output_audio.done / response.done	talking_stopped
transcript.done (role: assistant)	talking_stopped

Two signals can't come from the event stream alone, so the returned store exposes them imperatively:

• setListening(active) — call from whatever mic-capture code a project uses. That pipeline doesn't exist in core yet (see "Known limitations" below), and the client is the only thing that currently knows when the user starts/stops talking: the protocol doesn't surface OpenAI's input_audio_buffer.speech_* VAD events — see the "not part of our normalized vocabulary" comment in server/core/providers/openai.ts's fromOpenAiEvent.
• notifyResponseRequested() — call right when sending a typed input_text.send (the voice path already gets this for free from setListening(false)).

This wrapper also owns the idleActive → idlePassive clock: the store starts (and every reset/talking_stopped returns to) idleActive, and a timer — options.idlePassiveTimeoutMs, default DEFAULT_IDLE_PASSIVE_TIMEOUT_MS (30s) — fires idle_timeout if nothing else happens first. Any other signal re-arms it on the next idleActive entry; destroy() cancels it.

The returned AvatarStateStore.subscribe() follows the Svelte store contract (calls run immediately with the current value, then on every change), so it's usable directly as $store in a .svelte file without a hard Svelte dependency in this package itself. See /demo/humanoid (src/routes/demo/humanoid/+page.svelte) for the reference wiring, including the manual "listening" toggle standing in for real mic-capture UI.

Known limitations

• thinking is a heuristic, not a precise reflection of server-side activity — the time between a request and the assistant's first output. The browser never sees server-executed tool calls at all (see TOOLS.md), so "the model is running a tool" isn't observable from here.
• The idle split is timer-based, not presence-based — idlePassive is reached purely by elapsed time with no other signal, not by knowing whether the user actually left. A project with webcam/face-tracking data can layer a real presence signal on top (e.g. calling back into idleActive early, or suppressing the timeout) — that stays project-owned.

$lib/humanoid/animation.ts — animation-channel composition

type ARKitBlendShape = 'jawOpen' | 'eyeBlinkLeft' | /* ...all 52 ARFaceAnchor.BlendShapeLocation names */;
type AnimationFrame = Partial<Record<ARKitBlendShape, number>>; // sparse, 0–1 weights

compositeAnimationFrames(layers) folds an ordered list of AnimationLayers ({ frame, mode: 'add' | 'override', weight? }) into one AnimationFrame, clamping each channel to [0, 1]. This generalizes the eye look-at design doc's own additive-layers pattern — Macro, Micro, and Nano gaze offsets summed into one effective offset — to arbitrary channels, plus 'override' for layers whose values can't be meaningfully summed (a blink closing an eyelid a gaze layer also touches, for instance).

This is pure, isomorphic, and has no idea how the resulting AnimationFrame gets applied to a mesh — that binding (which bone/morph target each ARKitBlendShape name maps to in a given rig) is entirely the project's job.

What's not built here (on purpose)

• The eye look-at, blink, lip-sync, and body/gesture libraries themselves — future work, built as consumers of this contract, not part of it.
• Mic-capture/audio-playback in the browser (still open — see docs/REALTIME_SESSION.md's "What's deliberately not built yet").
• Server-side viseme/phoneme generation — an open design fork (server-side, since the relay already touches raw audio, vs. client-side from the playback buffer) to decide when the lip-sync library actually gets built.
• Any protocol change ($lib/core/protocol.ts is untouched by this package) — AvatarState derivation works entirely from events the protocol already carries.