# Humanoid package

`humanoid` is the first instance of the **optional package** convention (see "Module layering" in [ARCHITECTURE.md](ARCHITECTURE.md)) — a renderer-agnostic contract layer for 3D humanoid avatars rigged with ARKit blend shapes, scoped out of `core` because a 2D or non-humanoid project has no use for it. Nothing in `core` or `project` imports from it; a project opts in by importing directly from `$lib/humanoid` / `$lib/client/humanoid`, and if nothing imports it, it's simply not part of that project's bundle.

This package ships the **contract** future animation libraries (eye look-at, blink, lip-sync, body/gesture) plug into — not the libraries themselves, and not a renderer. It has no idea what's rendering the avatar (Three.js, Babylon, a game engine WebGL bridge, anything else) — that stays entirely project-owned, per the base project's founding constraint that appearance/UI is never core's job.

## `$lib/humanoid/state.ts` — avatar state

```ts
type AvatarState = 'idleActive' | 'idlePassive' | 'listening' | 'thinking' | 'talking';
```

`idle` is split into `idleActive`/`idlePassive` from the start, since ported animation systems (this one's lineage traces to an Unreal Engine avatar app) commonly key their idle behavior off exactly this distinction: `idleActive` right after a turn ends (still "engaged"), settling into `idlePassive` once nothing has happened for a while (bored/ambient idle animation). The reducer itself has no timers — pure and isomorphic — so the active→passive transition rides in on an explicit `idle_timeout` signal; the client wrapper (`deriveAvatarState`, below) owns the actual clock, same as it owns `setListening`.

`nextAvatarState(current, signal)` is a pure reducer — no WS/browser dependency, directly unit-tested in `state.spec.ts`. The signal set is small and closed (`listening_started`/`listening_stopped`, `response_requested`, `talking_started`/`talking_stopped`, `idle_timeout`, `reset`).

Note `listening_stopped` moves to `thinking`, not back to idle — ending listening almost always means a turn just completed and a response is expected next, so the avatar shouldn't visibly relax for the split second before the model's response starts arriving. `talking_stopped` and `reset` both land on `idleActive`, never straight to `idlePassive` — passive idle is only ever reached via `idle_timeout`.

## `$lib/humanoid/config.ts` — shared tuning knobs

```ts
interface AnimationConfig {
	idlePassiveTimeoutMs: number; // default 30_000, see DEFAULT_ANIMATION_CONFIG
}
```

Every value this package's own state derivation exposes for tuning lives in this one `AnimationConfig` interface, not as a bespoke option on whichever function first needed it — `idlePassiveTimeoutMs` (consumed by `deriveAvatarState`, below) is the first entry, and it's where the next one goes too as more of the Unreal Engine avatar app's animation systems get ported over (further idle-escalation thresholds, turn-taking heuristics, etc.).

`resolveAnimationConfig(overrides?)` merges a project's partial overrides onto `DEFAULT_ANIMATION_CONFIG` — a project only names the knobs it wants to change. Every `$lib/humanoid`/`$lib/client/humanoid` function that reads a tunable value takes `Partial<AnimationConfig>` (default `{}`) and resolves it internally, so passing the same override object to several such functions keeps them in sync without each needing the whole config spelled out.

This interface is scoped to `humanoid`'s own contract layer. A future animation library built as a *consumer* of this package (eye look-at, blink, lip-sync, body/gesture) is a separate package and gets its own config surface — follow this same three-piece shape there (`XConfig` interface, `DEFAULT_X_CONFIG`, `resolveXConfig`) rather than growing this interface across a package boundary. This mirrors the base project's existing convention of config living beside the subsystem it configures rather than in one shared blob (see [CONFIGURATION.md](CONFIGURATION.md)'s note on `AnalyticsConfig`/`SessionDefaults`).

## `$lib/client/humanoid/avatar-state.ts` — deriving it live

`deriveAvatarState(source, animationConfig?)` takes a `RealtimeClient` or `ResilientRealtimeClient` (accepted structurally, via an `AvatarStateSource` interface — this package doesn't care which transport variant is in use) and folds its `ServerEvent`s through `nextAvatarState`:

| `ServerEvent`                         | Signal            |
| ------------------------------------- | ----------------- |
| `session.ready` / `session.closed`    | `reset`           |
| `output_audio.chunk`                  | `talking_started` |
| `transcript.delta` (role: assistant)  | `talking_started` |
| `output_audio.done` / `response.done` | `talking_stopped` |
| `transcript.done` (role: assistant)   | `talking_stopped` |

Two signals can't come from the event stream alone, so the returned store exposes them imperatively:

- **`setListening(active)`** — call from whatever mic-capture code a project uses. That pipeline doesn't exist in `core` yet (see "Known limitations" below), and the client is the only thing that currently knows when the user starts/stops talking: the protocol doesn't surface OpenAI's `input_audio_buffer.speech_*` VAD events — see the "not part of our normalized vocabulary" comment in `server/core/providers/openai.ts`'s `fromOpenAiEvent`.
- **`notifyResponseRequested()`** — call right when sending a typed `input_text.send` (the voice path already gets this for free from `setListening(false)`).

This wrapper also owns the `idleActive` → `idlePassive` clock: the store starts (and every `reset`/`talking_stopped` returns to) `idleActive`, and a timer — `animationConfig.idlePassiveTimeoutMs`, see `config.ts` above — fires `idle_timeout` if nothing else happens first. Any other signal re-arms it on the next `idleActive` entry; `destroy()` cancels it.

The returned `AvatarStateStore.subscribe()` follows the Svelte store contract (calls `run` immediately with the current value, then on every change), so it's usable directly as `$store` in a `.svelte` file without a hard Svelte dependency in this package itself. See `/demo/humanoid` (`src/routes/demo/humanoid/+page.svelte`) for the reference wiring, including the manual "listening" toggle standing in for real mic-capture UI.

### Known limitations

- **`thinking` is a heuristic**, not a precise reflection of server-side activity — the time between a request and the assistant's first output. The browser never sees server-executed tool calls at all (see [TOOLS.md](TOOLS.md)), so "the model is running a tool" isn't observable from here.
- **The idle split is timer-based, not presence-based** — `idlePassive` is reached purely by elapsed time with no other signal, not by knowing whether the user actually left. A project with webcam/face-tracking data can layer a real presence signal on top (e.g. calling back into `idleActive` early, or suppressing the timeout) — that stays project-owned.

## `$lib/humanoid/animation.ts` — animation-channel composition

```ts
type ARKitBlendShape = 'jawOpen' | 'eyeBlinkLeft' | /* ...all 52 ARFaceAnchor.BlendShapeLocation names */;
type AnimationFrame = Partial<Record<ARKitBlendShape, number>>; // sparse, 0–1 weights
```

`compositeAnimationFrames(layers)` folds an ordered list of `AnimationLayer`s (`{ frame, mode: 'add' | 'override', weight? }`) into one `AnimationFrame`, clamping each channel to `[0, 1]`. This generalizes the eye look-at design doc's own additive-layers pattern — Macro, Micro, and Nano gaze offsets summed into one effective offset — to arbitrary channels, plus `'override'` for layers whose values can't be meaningfully summed (a blink closing an eyelid a gaze layer also touches, for instance).

This is pure, isomorphic, and has no idea how the resulting `AnimationFrame` gets applied to a mesh — that binding (which bone/morph target each `ARKitBlendShape` name maps to in a given rig) is entirely the project's job.

## What's not built here (on purpose)

- The eye look-at, blink, lip-sync, and body/gesture libraries themselves — future work, built as consumers of this contract, not part of it.
- Mic-capture/audio-playback in the browser (still open — see `docs/REALTIME_SESSION.md`'s "What's deliberately not built yet").
- Server-side viseme/phoneme generation — an open design fork (server-side, since the relay already touches raw audio, vs. client-side from the playback buffer) to decide when the lip-sync library actually gets built.
- Any protocol change (`$lib/core/protocol.ts` is untouched by this package) — `AvatarState` derivation works entirely from events the protocol already carries.