# Humanoid package

`humanoid` is the first instance of the **optional package** convention (see "Module layering" in [ARCHITECTURE.md](ARCHITECTURE.md)) — a renderer-agnostic contract layer for 3D humanoid avatars rigged with ARKit blend shapes, scoped out of `core` because a 2D or non-humanoid project has no use for it. Nothing in `core` or `project` imports from it; a project opts in by importing directly from `$lib/humanoid` / `$lib/client/humanoid`, and if nothing imports it, it's simply not part of that project's bundle.

This package ships the **contract** future animation libraries (eye look-at, blink, lip-sync, body/gesture) plug into — not the libraries themselves, and not a renderer. It has no idea what's rendering the avatar (Three.js, Babylon, a game engine WebGL bridge, anything else) — that stays entirely project-owned, per the base project's founding constraint that appearance/UI is never core's job.

## `$lib/humanoid/state.ts` — avatar state

```ts
type AvatarState = 'idle' | 'listening' | 'thinking' | 'talking';
```

Deliberately just four states, not the `IdleActive`/`IdlePassive` split a humanoid rig's own animation design (e.g. a future eye look-at library) may eventually want — that distinction needs a user-presence signal (webcam/face-tracking) this package has no opinion on. A project with that signal can layer its own finer idle split on top; `idle` here just means "no signal that the user or the model is currently doing anything."

`nextAvatarState(current, signal)` is a pure reducer — no WS/browser dependency, directly unit-tested in `state.spec.ts`. The signal set is small and closed (`listening_started`/`listening_stopped`, `response_requested`, `talking_started`/`talking_stopped`, `reset`).

Note `listening_stopped` moves to `thinking`, not `idle` — ending listening almost always means a turn just completed and a response is expected next, so the avatar shouldn't visibly relax for the split second before the model's response starts arriving.

## `$lib/client/humanoid/avatar-state.ts` — deriving it live

`deriveAvatarState(source)` takes a `RealtimeClient` or `ResilientRealtimeClient` (accepted structurally, via an `AvatarStateSource` interface — this package doesn't care which transport variant is in use) and folds its `ServerEvent`s through `nextAvatarState`:

| `ServerEvent`                         | Signal            |
| ------------------------------------- | ----------------- |
| `session.ready` / `session.closed`    | `reset`           |
| `output_audio.chunk`                  | `talking_started` |
| `transcript.delta` (role: assistant)  | `talking_started` |
| `output_audio.done` / `response.done` | `talking_stopped` |
| `transcript.done` (role: assistant)   | `talking_stopped` |

Two signals can't come from the event stream alone, so the returned store exposes them imperatively:

- **`setListening(active)`** — call from whatever mic-capture code a project uses. That pipeline doesn't exist in `core` yet (see "Known limitations" below), and the client is the only thing that currently knows when the user starts/stops talking: the protocol doesn't surface OpenAI's `input_audio_buffer.speech_*` VAD events — see the "not part of our normalized vocabulary" comment in `server/core/providers/openai.ts`'s `fromOpenAiEvent`.
- **`notifyResponseRequested()`** — call right when sending a typed `input_text.send` (the voice path already gets this for free from `setListening(false)`).

The returned `AvatarStateStore.subscribe()` follows the Svelte store contract (calls `run` immediately with the current value, then on every change), so it's usable directly as `$store` in a `.svelte` file without a hard Svelte dependency in this package itself. See `/demo/humanoid` (`src/routes/demo/humanoid/+page.svelte`) for the reference wiring, including the manual "listening" toggle standing in for real mic-capture UI.

### Known limitations

- **`thinking` is a heuristic**, not a precise reflection of server-side activity — the time between a request and the assistant's first output. The browser never sees server-executed tool calls at all (see [TOOLS.md](TOOLS.md)), so "the model is running a tool" isn't observable from here.
- **No presence-based idle split** (see above) — a project with webcam/face-tracking data can extend on top.

## `$lib/humanoid/animation.ts` — animation-channel composition

```ts
type ARKitBlendShape = 'jawOpen' | 'eyeBlinkLeft' | /* ...all 52 ARFaceAnchor.BlendShapeLocation names */;
type AnimationFrame = Partial<Record<ARKitBlendShape, number>>; // sparse, 0–1 weights
```

`compositeAnimationFrames(layers)` folds an ordered list of `AnimationLayer`s (`{ frame, mode: 'add' | 'override', weight? }`) into one `AnimationFrame`, clamping each channel to `[0, 1]`. This generalizes the eye look-at design doc's own additive-layers pattern — Macro, Micro, and Nano gaze offsets summed into one effective offset — to arbitrary channels, plus `'override'` for layers whose values can't be meaningfully summed (a blink closing an eyelid a gaze layer also touches, for instance).

This is pure, isomorphic, and has no idea how the resulting `AnimationFrame` gets applied to a mesh — that binding (which bone/morph target each `ARKitBlendShape` name maps to in a given rig) is entirely the project's job.

## What's not built here (on purpose)

- The eye look-at, blink, lip-sync, and body/gesture libraries themselves — future work, built as consumers of this contract, not part of it.
- Mic-capture/audio-playback in the browser (still open — see `docs/REALTIME_SESSION.md`'s "What's deliberately not built yet").
- Server-side viseme/phoneme generation — an open design fork (server-side, since the relay already touches raw audio, vs. client-side from the playback buffer) to decide when the lip-sync library actually gets built.
- Any protocol change (`$lib/core/protocol.ts` is untouched by this package) — `AvatarState` derivation works entirely from events the protocol already carries.