webavatar-core/docs/HUMANOID.md

Humanoid package

humanoid — published as @webavatar/webavatar-humanoid (source in packages/humanoid/src/) — is the first instance of the optional package convention (see "Module layering" in ARCHITECTURE.md) — a renderer-agnostic contract layer for 3D humanoid avatars rigged with ARKit blend shapes, scoped out of core because a 2D or non-humanoid project has no use for it. Nothing in core or the app depends on it by default; a project opts in by installing the package and importing directly from @webavatar/webavatar-humanoid / @webavatar/webavatar-humanoid/client, and if a project doesn't import it, it's simply not part of that project's bundle.

This package ships the contract animation libraries plug into — @webavatar/webavatar-humanoid/eye (below) is the first one actually built, with blink/lip-sync/body-gesture still to come — not a renderer. It has no idea what's rendering the avatar (Three.js, Babylon, a game engine WebGL bridge, anything else) — that stays entirely project-owned, per the base project's founding constraint that appearance/UI is never core's job. apps/template/src/routes/demo/humanoid is where a project-owned renderer (Three.js, in the bundled demo) actually exists — see "/demo/humanoid" below.

@webavatar/webavatar-humanoid — avatar state (state.ts)

type AvatarState = 'idleActive' | 'idlePassive' | 'listening' | 'thinking' | 'talking';

idle is split into idleActive/idlePassive from the start, since ported animation systems (this one's lineage traces to an Unreal Engine avatar app) commonly key their idle behavior off exactly this distinction: idleActive right after a turn ends (still "engaged"), settling into idlePassive once nothing has happened for a while (bored/ambient idle animation). The reducer itself has no timers — pure and isomorphic — so the active→passive transition rides in on an explicit idle_timeout signal; the client wrapper (deriveAvatarState, below) owns the actual clock, same as it owns setListening.

nextAvatarState(current, signal) is a pure reducer — no WS/browser dependency, directly unit-tested in state.spec.ts. The signal set is small and closed (listening_started/listening_stopped, response_requested, talking_started/talking_stopped, idle_timeout, reset).

Note listening_stopped moves to thinking, not back to idle — ending listening almost always means a turn just completed and a response is expected next, so the avatar shouldn't visibly relax for the split second before the model's response starts arriving. talking_stopped and reset both land on idleActive, never straight to idlePassive — passive idle is only ever reached via idle_timeout.

@webavatar/webavatar-humanoid — shared tuning knobs (config.ts)

interface AnimationConfig {
	idlePassiveTimeoutMs: number; // default 30_000, see DEFAULT_ANIMATION_CONFIG
}

Every value this package's own state derivation exposes for tuning lives in this one AnimationConfig interface, not as a bespoke option on whichever function first needed it — idlePassiveTimeoutMs (consumed by deriveAvatarState, below) is the first entry, and it's where the next one goes too as more of the Unreal Engine avatar app's animation systems get ported over (further idle-escalation thresholds, turn-taking heuristics, etc.).

resolveAnimationConfig(overrides?) merges a project's partial overrides onto DEFAULT_ANIMATION_CONFIG — a project only names the knobs it wants to change. Every @webavatar/webavatar-humanoid/@webavatar/webavatar-humanoid/client function that reads a tunable value takes Partial<AnimationConfig> (default {}) and resolves it internally, so passing the same override object to several such functions keeps them in sync without each needing the whole config spelled out.

This interface is scoped to humanoid's own contract layer. An animation library built as a consumer of this package (eye look-at, blink, lip-sync, body/gesture) nests inside the package at packages/humanoid/src/<name>/ (exported as @webavatar/webavatar-humanoid/<name>, with a browser wrapper at <name>/client when it needs one) rather than becoming a separate top-level package — eye (below) is the first — and gets its own config surface rather than growing this interface: follow this same three-piece shape there (XConfig interface, DEFAULT_X_CONFIG, resolveXConfig). This mirrors the base project's existing convention of config living beside the subsystem it configures rather than in one shared blob (see CONFIGURATION.md's note on AnalyticsConfig/SessionDefaults).

@webavatar/webavatar-humanoid/client — deriving it live (client/avatar-state.ts)

deriveAvatarState(source, animationConfig?) takes a RealtimeClient or ResilientRealtimeClient (accepted structurally, via an AvatarStateSource interface — this package doesn't care which transport variant is in use) and folds its ServerEvents through nextAvatarState:

ServerEvent	Signal
session.ready / session.closed	reset
output_audio.chunk	talking_started
transcript.delta (role: assistant)	talking_started
output_audio.done / response.done	talking_stopped
transcript.done (role: assistant)	talking_stopped

Two signals can't come from the event stream alone, so the returned store exposes them imperatively:

• setListening(active) — call from whatever mic-capture code a project uses. That pipeline doesn't exist in core yet (see "Known limitations" below), and the client is the only thing that currently knows when the user starts/stops talking: the protocol doesn't surface OpenAI's input_audio_buffer.speech_* VAD events — see the "not part of our normalized vocabulary" comment in packages/core/src/server/providers/openai.ts's fromOpenAiEvent.
• notifyResponseRequested() — call right when sending a typed input_text.send (the voice path already gets this for free from setListening(false)).

This wrapper also owns the idleActive → idlePassive clock: the store starts (and every reset/talking_stopped returns to) idleActive, and a timer — animationConfig.idlePassiveTimeoutMs, see config.ts above — fires idle_timeout if nothing else happens first. Any other signal re-arms it on the next idleActive entry; destroy() cancels it.

The returned AvatarStateStore.subscribe() follows the Svelte store contract (calls run immediately with the current value, then on every change), so it's usable directly as $store in a .svelte file without a hard Svelte dependency in this package itself. See /demo/humanoid (apps/template/src/routes/demo/humanoid/+page.svelte) for the reference wiring, including the manual "listening" toggle standing in for real mic-capture UI.

Known limitations

• thinking is a heuristic, not a precise reflection of server-side activity — the time between a request and the assistant's first output. The browser never sees server-executed tool calls at all (see TOOLS.md), so "the model is running a tool" isn't observable from here.
• The idle split is timer-based, not presence-based — idlePassive is reached purely by elapsed time with no other signal, not by knowing whether the user actually left. A project with webcam/face-tracking data can layer a real presence signal on top (e.g. calling back into idleActive early, or suppressing the timeout) — that stays project-owned.

@webavatar/webavatar-humanoid — animation-channel composition (animation.ts)

type ARKitBlendShape = 'jawOpen' | 'eyeBlinkLeft' | /* ...all 52 ARFaceAnchor.BlendShapeLocation names */;
type AnimationFrame = Partial<Record<ARKitBlendShape, number>>; // sparse, 0–1 weights

compositeAnimationFrames(layers) folds an ordered list of AnimationLayers ({ frame, mode: 'add' | 'override', weight? }) into one AnimationFrame, clamping each channel to [0, 1]. This generalizes the eye look-at design doc's own additive-layers pattern — Macro, Micro, and Nano gaze offsets summed into one effective offset — to arbitrary channels, plus 'override' for layers whose values can't be meaningfully summed (a blink closing an eyelid a gaze layer also touches, for instance).

This is pure, isomorphic, and has no idea how the resulting AnimationFrame gets applied to a mesh — that binding (which bone/morph target each ARKitBlendShape name maps to in a given rig) is entirely the project's job.

@webavatar/webavatar-humanoid/eye — gaze animation

The first animation library built as a consumer of this package's contract, ported from an Unreal Engine avatar app's eye-gaze design doc. Three independent, continuously-ticking, additive 2D cm-offset layers — Macro (periodic look-aways, sometimes chaining "secondary" glances nearby before returning to center), Micro (small saccades simulating scanning a conversation partner's eyes/eyebrows), Nano (tiny constant jitter) — summed into one combinedOffsetCm on a plane in front of the avatar's face. Each layer's current value exponentially decays toward a randomly-redrawn goal (the doc's "SpringInterpolate"). Macro's look-at duration is keyed by the live AvatarState (@webavatar/webavatar-humanoid's state.ts) — this is the concrete use case AnimationConfig's doc comment anticipated when it said future ported systems would key off exactly that distinction. See packages/humanoid/src/eye/DESIGN_NOTES.md for what's a faithful port of the source doc's variables/logic versus a documented approximation (the three easing curves, reconstructed from screenshots of hand-authored Unreal splines with no exact keyframe data) versus a wholly new web-port adaptation (turning a cm-offset into ARKit eyeLook* blend-shape weights — the original drove real eye bones, not blend shapes).

function stepEyeGaze(
	state: EyeGazeState,
	input: { deltaSeconds: number; avatarState: AvatarState; lookAtDistanceCm: number },
	config: EyeConfig,
	rng: () => number, // never Math.random directly — see below
	toggles?: { macro?: boolean; micro?: boolean; nano?: boolean }
): EyeGazeState;

function gazeOffsetToBlendShapes(
	offsetCm: Vec2,
	lookAtDistanceCm: number,
	maxGazeAngleRad: number
): AnimationFrame;

stepEyeGaze is pure and directly unit-tested with a scripted rng (gaze.spec.ts) — every random draw anywhere in the module takes rng as a parameter, never calling Math.random() itself, the same "push side effects to the edge" shape as nextAvatarState. gazeOffsetToBlendShapes (blend-shapes.ts) is the separate, independently-tested conversion from EyeGazeState.combinedOffsetCm to the 8 ARKit eyeLook* channels — see its docstring for the left/right eye sign convention, the one place in this whole port that's easy to get backwards.

@webavatar/webavatar-humanoid/eye/client's createEyeGazeController(overrides?) — the module lives at packages/humanoid/src/eye/client.ts (renamed from gaze-controller.ts during the monorepo migration to match its export path) — is the client wrapper: owns the per-frame clock (requestAnimationFrame) and Math.random, mirroring deriveAvatarState's shape (a Svelte-store-compatible subscribe, plus imperative setters: setAvatarState, setLookAtDistanceCm, setLayerEnabled, setTimeScale, forceLookAway, destroy). See /demo/humanoid's "Eye system demo" section (apps/template/src/routes/demo/humanoid/EyeSystemDemo.svelte) for the reference wiring against a live Three.js-rendered avatar, including per-layer toggles, a forced-look-away button, and raw macro/micro/nano/combined offset and blend-shape readouts for debugging.

/demo/humanoid — playground

apps/template/src/routes/demo/humanoid/+page.svelte is a testing/debugging surface for this package's animation libraries generally, not just a demo of one — it composes AvatarStateDemo.svelte (the state-derivation demo above) and EyeSystemDemo.svelte side by side, structured so a future library's demo (blink, lip-sync, body/gesture) is another sibling component, not a rewrite. EyeSystemDemo.svelte's eye-demo/ subfolder holds its Three.js viewer and debug UI — ThreeAvatarViewer.svelte is the only file in this whole feature allowed to import three or touch mesh/bone data directly; everything in @webavatar/webavatar-humanoid (packages/humanoid/src/**, both its root contract and its client/eye entry points) stays renderer-agnostic. The bundled test avatar (apps/template/static/models/avatar.glb, a Ready Player Me-style rig) carries the same 52 ARKit morph targets this package's ARKitBlendShape type enumerates, on 4 of its meshes.

What's not built here (on purpose)

• The blink, lip-sync, and body/gesture libraries — future work, built as further consumers of this contract the same way eye was, not part of it.
• Mic-capture/audio-playback in the browser (still open — see docs/REALTIME_SESSION.md's "What's deliberately not built yet").
• Server-side viseme/phoneme generation — an open design fork (server-side, since the relay already touches raw audio, vs. client-side from the playback buffer) to decide when the lip-sync library actually gets built.
• Any protocol change (core's protocol.ts — packages/core/src/protocol.ts — is untouched by this package) — AvatarState derivation works entirely from events the protocol already carries.
• The LookAt system (whole-body/head aim toward a target) the source eye-gaze doc describes as eye's complement — eye's lookAtDistanceCm input assumes something else eventually owns that; EyeSystemDemo.svelte exposes it as a manual slider standing in for that not-yet-built system, the same way the avatar-state demo's "listening" toggle stands in for real mic-capture.