webavatar-core/docs/HUMANOID.md

Humanoid package

humanoid is the first instance of the optional package convention (see "Module layering" in ARCHITECTURE.md) — a renderer-agnostic contract layer for 3D humanoid avatars rigged with ARKit blend shapes, scoped out of core because a 2D or non-humanoid project has no use for it. Nothing in core or project imports from it; a project opts in by importing directly from $lib/humanoid / $lib/client/humanoid, and if nothing imports it, it's simply not part of that project's bundle.

This package ships the contract future animation libraries (eye look-at, blink, lip-sync, body/gesture) plug into — not the libraries themselves, and not a renderer. It has no idea what's rendering the avatar (Three.js, Babylon, a game engine WebGL bridge, anything else) — that stays entirely project-owned, per the base project's founding constraint that appearance/UI is never core's job.

$lib/humanoid/state.ts — avatar state

type AvatarState = 'idleActive' | 'idlePassive' | 'listening' | 'thinking' | 'talking';

idle is split into idleActive/idlePassive from the start, since ported animation systems (this one's lineage traces to an Unreal Engine avatar app) commonly key their idle behavior off exactly this distinction: idleActive right after a turn ends (still "engaged"), settling into idlePassive once nothing has happened for a while (bored/ambient idle animation). The reducer itself has no timers — pure and isomorphic — so the active→passive transition rides in on an explicit idle_timeout signal; the client wrapper (deriveAvatarState, below) owns the actual clock, same as it owns setListening.

nextAvatarState(current, signal) is a pure reducer — no WS/browser dependency, directly unit-tested in state.spec.ts. The signal set is small and closed (listening_started/listening_stopped, response_requested, talking_started/talking_stopped, idle_timeout, reset).

Note listening_stopped moves to thinking, not back to idle — ending listening almost always means a turn just completed and a response is expected next, so the avatar shouldn't visibly relax for the split second before the model's response starts arriving. talking_stopped and reset both land on idleActive, never straight to idlePassive — passive idle is only ever reached via idle_timeout.

$lib/humanoid/config.ts — shared tuning knobs

interface AnimationConfig {
	idlePassiveTimeoutMs: number; // default 30_000, see DEFAULT_ANIMATION_CONFIG
}

Every value this package's own state derivation exposes for tuning lives in this one AnimationConfig interface, not as a bespoke option on whichever function first needed it — idlePassiveTimeoutMs (consumed by deriveAvatarState, below) is the first entry, and it's where the next one goes too as more of the Unreal Engine avatar app's animation systems get ported over (further idle-escalation thresholds, turn-taking heuristics, etc.).

resolveAnimationConfig(overrides?) merges a project's partial overrides onto DEFAULT_ANIMATION_CONFIG — a project only names the knobs it wants to change. Every $lib/humanoid/$lib/client/humanoid function that reads a tunable value takes Partial<AnimationConfig> (default {}) and resolves it internally, so passing the same override object to several such functions keeps them in sync without each needing the whole config spelled out.

This interface is scoped to humanoid's own contract layer. A future animation library built as a consumer of this package (eye look-at, blink, lip-sync, body/gesture) is a separate package and gets its own config surface — follow this same three-piece shape there (XConfig interface, DEFAULT_X_CONFIG, resolveXConfig) rather than growing this interface across a package boundary. This mirrors the base project's existing convention of config living beside the subsystem it configures rather than in one shared blob (see CONFIGURATION.md's note on AnalyticsConfig/SessionDefaults).

$lib/client/humanoid/avatar-state.ts — deriving it live

deriveAvatarState(source, animationConfig?) takes a RealtimeClient or ResilientRealtimeClient (accepted structurally, via an AvatarStateSource interface — this package doesn't care which transport variant is in use) and folds its ServerEvents through nextAvatarState:

ServerEvent	Signal
session.ready / session.closed	reset
output_audio.chunk	talking_started
transcript.delta (role: assistant)	talking_started
output_audio.done / response.done	talking_stopped
transcript.done (role: assistant)	talking_stopped

Two signals can't come from the event stream alone, so the returned store exposes them imperatively:

• setListening(active) — call from whatever mic-capture code a project uses. That pipeline doesn't exist in core yet (see "Known limitations" below), and the client is the only thing that currently knows when the user starts/stops talking: the protocol doesn't surface OpenAI's input_audio_buffer.speech_* VAD events — see the "not part of our normalized vocabulary" comment in server/core/providers/openai.ts's fromOpenAiEvent.
• notifyResponseRequested() — call right when sending a typed input_text.send (the voice path already gets this for free from setListening(false)).

This wrapper also owns the idleActive → idlePassive clock: the store starts (and every reset/talking_stopped returns to) idleActive, and a timer — animationConfig.idlePassiveTimeoutMs, see config.ts above — fires idle_timeout if nothing else happens first. Any other signal re-arms it on the next idleActive entry; destroy() cancels it.

The returned AvatarStateStore.subscribe() follows the Svelte store contract (calls run immediately with the current value, then on every change), so it's usable directly as $store in a .svelte file without a hard Svelte dependency in this package itself. See /demo/humanoid (src/routes/demo/humanoid/+page.svelte) for the reference wiring, including the manual "listening" toggle standing in for real mic-capture UI.

Known limitations

• thinking is a heuristic, not a precise reflection of server-side activity — the time between a request and the assistant's first output. The browser never sees server-executed tool calls at all (see TOOLS.md), so "the model is running a tool" isn't observable from here.
• The idle split is timer-based, not presence-based — idlePassive is reached purely by elapsed time with no other signal, not by knowing whether the user actually left. A project with webcam/face-tracking data can layer a real presence signal on top (e.g. calling back into idleActive early, or suppressing the timeout) — that stays project-owned.

$lib/humanoid/animation.ts — animation-channel composition

type ARKitBlendShape = 'jawOpen' | 'eyeBlinkLeft' | /* ...all 52 ARFaceAnchor.BlendShapeLocation names */;
type AnimationFrame = Partial<Record<ARKitBlendShape, number>>; // sparse, 0–1 weights

compositeAnimationFrames(layers) folds an ordered list of AnimationLayers ({ frame, mode: 'add' | 'override', weight? }) into one AnimationFrame, clamping each channel to [0, 1]. This generalizes the eye look-at design doc's own additive-layers pattern — Macro, Micro, and Nano gaze offsets summed into one effective offset — to arbitrary channels, plus 'override' for layers whose values can't be meaningfully summed (a blink closing an eyelid a gaze layer also touches, for instance).

This is pure, isomorphic, and has no idea how the resulting AnimationFrame gets applied to a mesh — that binding (which bone/morph target each ARKitBlendShape name maps to in a given rig) is entirely the project's job.

What's not built here (on purpose)

• The eye look-at, blink, lip-sync, and body/gesture libraries themselves — future work, built as consumers of this contract, not part of it.
• Mic-capture/audio-playback in the browser (still open — see docs/REALTIME_SESSION.md's "What's deliberately not built yet").
• Server-side viseme/phoneme generation — an open design fork (server-side, since the relay already touches raw audio, vs. client-side from the playback buffer) to decide when the lip-sync library actually gets built.
• Any protocol change ($lib/core/protocol.ts is untouched by this package) — AvatarState derivation works entirely from events the protocol already carries.