# Humanoid package

`humanoid` — published as `@webavatar/webavatar-humanoid` (source in `packages/humanoid/src/`) — is the first instance of the **optional package** convention (see "Module layering" in [ARCHITECTURE.md](ARCHITECTURE.md)) — a renderer-agnostic contract layer for 3D humanoid avatars rigged with ARKit blend shapes, scoped out of `core` because a 2D or non-humanoid project has no use for it. Nothing in `core` or the app depends on it by default; a project opts in by installing the package and importing directly from `@webavatar/webavatar-humanoid` / `@webavatar/webavatar-humanoid/client`, and if a project doesn't import it, it's simply not part of that project's bundle.

This package ships the **contract** animation libraries plug into — `@webavatar/webavatar-humanoid/eye` (below) is the first one actually built, with blink/lip-sync/body-gesture still to come — not a renderer. It has no idea what's rendering the avatar (Three.js, Babylon, a game engine WebGL bridge, anything else) — that stays entirely project-owned, per the base project's founding constraint that appearance/UI is never core's job. `apps/template/src/routes/demo/humanoid` is where a project-owned renderer (Three.js, in the bundled demo) actually exists — see "`/demo/humanoid`" below.

## `@webavatar/webavatar-humanoid` — avatar state (`state.ts`)

```ts
type AvatarState = 'idleActive' | 'idlePassive' | 'listening' | 'thinking' | 'talking';
```

`idle` is split into `idleActive`/`idlePassive` from the start, since ported animation systems (this one's lineage traces to an Unreal Engine avatar app) commonly key their idle behavior off exactly this distinction: `idleActive` right after a turn ends (still "engaged"), settling into `idlePassive` once nothing has happened for a while (bored/ambient idle animation). The reducer itself has no timers — pure and isomorphic — so the active→passive transition rides in on an explicit `idle_timeout` signal; the client wrapper (`deriveAvatarState`, below) owns the actual clock, same as it owns `setListening`.

`nextAvatarState(current, signal)` is a pure reducer — no WS/browser dependency, directly unit-tested in `state.spec.ts`. The signal set is small and closed (`listening_started`/`listening_stopped`, `response_requested`, `talking_started`/`talking_stopped`, `idle_timeout`, `reset`).

Note `listening_stopped` moves to `thinking`, not back to idle — ending listening almost always means a turn just completed and a response is expected next, so the avatar shouldn't visibly relax for the split second before the model's response starts arriving. `talking_stopped` and `reset` both land on `idleActive`, never straight to `idlePassive` — passive idle is only ever reached via `idle_timeout`.

## `@webavatar/webavatar-humanoid` — shared tuning knobs (`config.ts`)

```ts
interface AnimationConfig {
	idlePassiveTimeoutMs: number; // default 30_000, see DEFAULT_ANIMATION_CONFIG
}
```

Every value this package's own state derivation exposes for tuning lives in this one `AnimationConfig` interface, not as a bespoke option on whichever function first needed it — `idlePassiveTimeoutMs` (consumed by `deriveAvatarState`, below) is the first entry, and it's where the next one goes too as more of the Unreal Engine avatar app's animation systems get ported over (further idle-escalation thresholds, turn-taking heuristics, etc.).

`resolveAnimationConfig(overrides?)` merges a project's partial overrides onto `DEFAULT_ANIMATION_CONFIG` — a project only names the knobs it wants to change. Every `@webavatar/webavatar-humanoid`/`@webavatar/webavatar-humanoid/client` function that reads a tunable value takes `Partial<AnimationConfig>` (default `{}`) and resolves it internally, so passing the same override object to several such functions keeps them in sync without each needing the whole config spelled out.

This interface is scoped to `humanoid`'s own contract layer. An animation library built as a _consumer_ of this package (eye look-at, blink, lip-sync, body/gesture) nests inside the package at `packages/humanoid/src/<name>/` (exported as `@webavatar/webavatar-humanoid/<name>`, with a browser wrapper at `<name>/client` when it needs one) rather than becoming a separate top-level package — `eye` (below) is the first — and gets its own config surface rather than growing this interface: follow this same three-piece shape there (`XConfig` interface, `DEFAULT_X_CONFIG`, `resolveXConfig`). This mirrors the base project's existing convention of config living beside the subsystem it configures rather than in one shared blob (see [CONFIGURATION.md](CONFIGURATION.md)'s note on `AnalyticsConfig`/`SessionDefaults`).

## `@webavatar/webavatar-humanoid/client` — deriving it live (`client/avatar-state.ts`)

`deriveAvatarState(source, animationConfig?)` takes a `RealtimeClient` or `ResilientRealtimeClient` (accepted structurally, via an `AvatarStateSource` interface — this package doesn't care which transport variant is in use) and folds its `ServerEvent`s through `nextAvatarState`:

| `ServerEvent`                         | Signal            |
| ------------------------------------- | ----------------- |
| `session.ready` / `session.closed`    | `reset`           |
| `output_audio.chunk`                  | `talking_started` |
| `transcript.delta` (role: assistant)  | `talking_started` |
| `output_audio.done` / `response.done` | `talking_stopped` |
| `transcript.done` (role: assistant)   | `talking_stopped` |

Two signals can't come from the event stream alone, so the returned store exposes them imperatively:

- **`setListening(active)`** — call from whatever mic-capture code a project uses. That pipeline doesn't exist in `core` yet (see "Known limitations" below), and the client is the only thing that currently knows when the user starts/stops talking: the protocol doesn't surface OpenAI's `input_audio_buffer.speech_*` VAD events — see the "not part of our normalized vocabulary" comment in `packages/core/src/server/providers/openai.ts`'s `fromOpenAiEvent`.
- **`notifyResponseRequested()`** — call right when sending a typed `input_text.send` (the voice path already gets this for free from `setListening(false)`).

This wrapper also owns the `idleActive` → `idlePassive` clock: the store starts (and every `reset`/`talking_stopped` returns to) `idleActive`, and a timer — `animationConfig.idlePassiveTimeoutMs`, see `config.ts` above — fires `idle_timeout` if nothing else happens first. Any other signal re-arms it on the next `idleActive` entry; `destroy()` cancels it.

The returned `AvatarStateStore.subscribe()` follows the Svelte store contract (calls `run` immediately with the current value, then on every change), so it's usable directly as `$store` in a `.svelte` file without a hard Svelte dependency in this package itself. See `/demo/humanoid` (`apps/template/src/routes/demo/humanoid/+page.svelte`) for the reference wiring, including the manual "listening" toggle standing in for real mic-capture UI.

### Known limitations

- **`thinking` is a heuristic**, not a precise reflection of server-side activity — the time between a request and the assistant's first output. The browser never sees server-executed tool calls at all (see [TOOLS.md](TOOLS.md)), so "the model is running a tool" isn't observable from here.
- **The idle split is timer-based, not presence-based** — `idlePassive` is reached purely by elapsed time with no other signal, not by knowing whether the user actually left. A project with webcam/face-tracking data can layer a real presence signal on top (e.g. calling back into `idleActive` early, or suppressing the timeout) — that stays project-owned.

## `@webavatar/webavatar-humanoid` — animation-channel composition (`animation.ts`)

```ts
type ARKitBlendShape = 'jawOpen' | 'eyeBlinkLeft' | /* ...all 52 ARFaceAnchor.BlendShapeLocation names */;
type AnimationFrame = Partial<Record<ARKitBlendShape, number>>; // sparse, 0–1 weights
```

`compositeAnimationFrames(layers)` folds an ordered list of `AnimationLayer`s (`{ frame, mode: 'add' | 'override', weight? }`) into one `AnimationFrame`, clamping each channel to `[0, 1]`. This generalizes the eye look-at design doc's own additive-layers pattern — Macro, Micro, and Nano gaze offsets summed into one effective offset — to arbitrary channels, plus `'override'` for layers whose values can't be meaningfully summed (a blink closing an eyelid a gaze layer also touches, for instance).

This is pure, isomorphic, and has no idea how the resulting `AnimationFrame` gets applied to a mesh — that binding (which bone/morph target each `ARKitBlendShape` name maps to in a given rig) is entirely the project's job.

## `@webavatar/webavatar-humanoid/eye` — gaze animation

The first animation library built as a consumer of this package's contract, ported from an Unreal Engine avatar app's eye-gaze design doc. Three independent, continuously-ticking, additive 2D cm-offset layers — Macro (periodic look-aways, sometimes chaining "secondary" glances nearby before returning to center), Micro (small saccades simulating scanning a conversation partner's eyes/eyebrows), Nano (tiny constant jitter) — summed into one `combinedOffsetCm` on a plane in front of the avatar's face. Each layer's `current` value exponentially decays toward a randomly-redrawn `goal` (the doc's "SpringInterpolate"). Macro's look-at duration is keyed by the live `AvatarState` (`@webavatar/webavatar-humanoid`'s `state.ts`) — this is the concrete use case `AnimationConfig`'s doc comment anticipated when it said future ported systems would key off exactly that distinction. See `packages/humanoid/src/eye/DESIGN_NOTES.md` for what's a faithful port of the source doc's variables/logic versus a documented approximation (the three easing curves, reconstructed from screenshots of hand-authored Unreal splines with no exact keyframe data) versus a wholly new web-port adaptation (turning a cm-offset into ARKit `eyeLook*` blend-shape weights — the original drove real eye bones, not blend shapes).

```ts
function stepEyeGaze(
	state: EyeGazeState,
	input: { deltaSeconds: number; avatarState: AvatarState; lookAtDistanceCm: number },
	config: EyeConfig,
	rng: () => number, // never Math.random directly — see below
	toggles?: { macro?: boolean; micro?: boolean; nano?: boolean }
): EyeGazeState;

function gazeOffsetToBlendShapes(
	offsetCm: Vec2,
	lookAtDistanceCm: number,
	maxGazeAngleRad: number
): AnimationFrame;
```

`stepEyeGaze` is pure and directly unit-tested with a scripted `rng` (`gaze.spec.ts`) — every random draw anywhere in the module takes `rng` as a parameter, never calling `Math.random()` itself, the same "push side effects to the edge" shape as `nextAvatarState`. `gazeOffsetToBlendShapes` (`blend-shapes.ts`) is the separate, independently-tested conversion from `EyeGazeState.combinedOffsetCm` to the 8 ARKit `eyeLook*` channels — see its docstring for the left/right eye sign convention, the one place in this whole port that's easy to get backwards.

`@webavatar/webavatar-humanoid/eye/client`'s `createEyeGazeController(overrides?)` — the module lives at `packages/humanoid/src/eye/client.ts` (renamed from `gaze-controller.ts` during the monorepo migration to match its export path) — is the client wrapper: owns the per-frame clock (`requestAnimationFrame`) and `Math.random`, mirroring `deriveAvatarState`'s shape (a Svelte-store-compatible `subscribe`, plus imperative setters: `setAvatarState`, `setLookAtDistanceCm`, `setLayerEnabled`, `setTimeScale`, `forceLookAway`, `destroy`). See `/demo/humanoid`'s "Eye system demo" section (`apps/template/src/routes/demo/humanoid/EyeSystemDemo.svelte`) for the reference wiring against a live Three.js-rendered avatar, including per-layer toggles, a forced-look-away button, and raw macro/micro/nano/combined offset and blend-shape readouts for debugging.

## `/demo/humanoid` — playground

`apps/template/src/routes/demo/humanoid/+page.svelte` is a testing/debugging surface for this package's animation libraries generally, not just a demo of one — it composes `AvatarStateDemo.svelte` (the state-derivation demo above) and `EyeSystemDemo.svelte` side by side, structured so a future library's demo (blink, lip-sync, body/gesture) is another sibling component, not a rewrite. `EyeSystemDemo.svelte`'s `eye-demo/` subfolder holds its Three.js viewer and debug UI — `ThreeAvatarViewer.svelte` is the only file in this whole feature allowed to import `three` or touch mesh/bone data directly; everything in `@webavatar/webavatar-humanoid` (`packages/humanoid/src/**`, both its root contract and its `client`/`eye` entry points) stays renderer-agnostic. The bundled test avatar (`apps/template/static/models/avatar.glb`, a Ready Player Me-style rig) carries the same 52 ARKit morph targets this package's `ARKitBlendShape` type enumerates, on 4 of its meshes.

## What's not built here (on purpose)

- The blink, lip-sync, and body/gesture libraries — future work, built as further consumers of this contract the same way `eye` was, not part of it.
- Mic-capture/audio-playback in the browser (still open — see `docs/REALTIME_SESSION.md`'s "What's deliberately not built yet").
- Server-side viseme/phoneme generation — an open design fork (server-side, since the relay already touches raw audio, vs. client-side from the playback buffer) to decide when the lip-sync library actually gets built.
- Any protocol change (core's `protocol.ts` — `packages/core/src/protocol.ts` — is untouched by this package) — `AvatarState` derivation works entirely from events the protocol already carries.
- The LookAt system (whole-body/head aim toward a target) the source eye-gaze doc describes as `eye`'s complement — `eye`'s `lookAtDistanceCm` input assumes something else eventually owns that; `EyeSystemDemo.svelte` exposes it as a manual slider standing in for that not-yet-built system, the same way the avatar-state demo's "listening" toggle stands in for real mic-capture.