diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index c477c54..bed85e3 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -18,7 +18,7 @@ A third kind of top-level directory, alongside `core` and `project`: a **package Same dependency rule as core/project, extended: **a package may import from `core`; `core` and `project` never import from a package; a project opts in by importing directly from the package** — no registration step, no runtime flag, no manifest. If nothing imports it, it's never bundled — that's what makes it free for a project that doesn't use it. -First (and, as of this writing, only) package: **`humanoid`** (`src/lib/humanoid/`, `src/lib/client/humanoid/`) — a renderer-agnostic avatar-state and animation-channel-composition contract for 3D humanoid avatars rigged with ARKit blend shapes. See [HUMANOID.md](HUMANOID.md). Future shared animation libraries (eye look-at, blink, lip-sync, body/gesture) are expected to build on top of it as further packages or as consumers of it, not to be folded into `core`. +First (and, as of this writing, only) package: **`humanoid`** (`src/lib/humanoid/`, `src/lib/client/humanoid/`) — a renderer-agnostic avatar-state and animation-channel-composition contract for 3D humanoid avatars rigged with ARKit blend shapes. See [HUMANOID.md](HUMANOID.md). Shared animation libraries built as consumers of that contract live as sub-packages nested inside it (`$lib/humanoid/`, `$lib/client/humanoid/`) rather than as further top-level packages — `eye` (gaze animation) is the first; blink, lip-sync, and body/gesture are expected to follow the same way, not to be folded into `core`. Routes (`src/routes/`) stay thin and delegate into these — a route file is glue, not where logic lives. diff --git a/docs/HUMANOID.md b/docs/HUMANOID.md index 8847091..1c7a2fc 100644 --- a/docs/HUMANOID.md +++ b/docs/HUMANOID.md @@ -2,7 +2,7 @@ `humanoid` is the first instance of the **optional package** convention (see "Module layering" in [ARCHITECTURE.md](ARCHITECTURE.md)) — a renderer-agnostic contract layer for 3D humanoid avatars rigged with ARKit blend shapes, scoped out of `core` because a 2D or non-humanoid project has no use for it. Nothing in `core` or `project` imports from it; a project opts in by importing directly from `$lib/humanoid` / `$lib/client/humanoid`, and if nothing imports it, it's simply not part of that project's bundle. -This package ships the **contract** future animation libraries (eye look-at, blink, lip-sync, body/gesture) plug into — not the libraries themselves, and not a renderer. It has no idea what's rendering the avatar (Three.js, Babylon, a game engine WebGL bridge, anything else) — that stays entirely project-owned, per the base project's founding constraint that appearance/UI is never core's job. +This package ships the **contract** animation libraries plug into — `$lib/humanoid/eye` (below) is the first one actually built, with blink/lip-sync/body-gesture still to come — not a renderer. It has no idea what's rendering the avatar (Three.js, Babylon, a game engine WebGL bridge, anything else) — that stays entirely project-owned, per the base project's founding constraint that appearance/UI is never core's job. `src/routes/demo/humanoid` is where a project-owned renderer (Three.js, in the bundled demo) actually exists — see "`/demo/humanoid`" below. ## `$lib/humanoid/state.ts` — avatar state @@ -28,7 +28,7 @@ Every value this package's own state derivation exposes for tuning lives in this `resolveAnimationConfig(overrides?)` merges a project's partial overrides onto `DEFAULT_ANIMATION_CONFIG` — a project only names the knobs it wants to change. Every `$lib/humanoid`/`$lib/client/humanoid` function that reads a tunable value takes `Partial` (default `{}`) and resolves it internally, so passing the same override object to several such functions keeps them in sync without each needing the whole config spelled out. -This interface is scoped to `humanoid`'s own contract layer. A future animation library built as a *consumer* of this package (eye look-at, blink, lip-sync, body/gesture) is a separate package and gets its own config surface — follow this same three-piece shape there (`XConfig` interface, `DEFAULT_X_CONFIG`, `resolveXConfig`) rather than growing this interface across a package boundary. This mirrors the base project's existing convention of config living beside the subsystem it configures rather than in one shared blob (see [CONFIGURATION.md](CONFIGURATION.md)'s note on `AnalyticsConfig`/`SessionDefaults`). +This interface is scoped to `humanoid`'s own contract layer. An animation library built as a _consumer_ of this package (eye look-at, blink, lip-sync, body/gesture) is a sub-package under `$lib/humanoid/` / `$lib/client/humanoid/` — `eye` (below) is the first — and gets its own config surface rather than growing this interface: follow this same three-piece shape there (`XConfig` interface, `DEFAULT_X_CONFIG`, `resolveXConfig`). This mirrors the base project's existing convention of config living beside the subsystem it configures rather than in one shared blob (see [CONFIGURATION.md](CONFIGURATION.md)'s note on `AnalyticsConfig`/`SessionDefaults`). ## `$lib/client/humanoid/avatar-state.ts` — deriving it live @@ -67,9 +67,38 @@ type AnimationFrame = Partial>; // sparse, 0–1 This is pure, isomorphic, and has no idea how the resulting `AnimationFrame` gets applied to a mesh — that binding (which bone/morph target each `ARKitBlendShape` name maps to in a given rig) is entirely the project's job. +## `$lib/humanoid/eye` — gaze animation + +The first animation library built as a consumer of this package's contract, ported from an Unreal Engine avatar app's eye-gaze design doc. Three independent, continuously-ticking, additive 2D cm-offset layers — Macro (periodic look-aways, sometimes chaining "secondary" glances nearby before returning to center), Micro (small saccades simulating scanning a conversation partner's eyes/eyebrows), Nano (tiny constant jitter) — summed into one `combinedOffsetCm` on a plane in front of the avatar's face. Each layer's `current` value exponentially decays toward a randomly-redrawn `goal` (the doc's "SpringInterpolate"). Macro's look-at duration is keyed by the live `AvatarState` (`$lib/humanoid/state.ts`) — this is the concrete use case `AnimationConfig`'s doc comment anticipated when it said future ported systems would key off exactly that distinction. See `src/lib/humanoid/eye/DESIGN_NOTES.md` for what's a faithful port of the source doc's variables/logic versus a documented approximation (the three easing curves, reconstructed from screenshots of hand-authored Unreal splines with no exact keyframe data) versus a wholly new web-port adaptation (turning a cm-offset into ARKit `eyeLook*` blend-shape weights — the original drove real eye bones, not blend shapes). + +```ts +function stepEyeGaze( + state: EyeGazeState, + input: { deltaSeconds: number; avatarState: AvatarState; lookAtDistanceCm: number }, + config: EyeConfig, + rng: () => number, // never Math.random directly — see below + toggles?: { macro?: boolean; micro?: boolean; nano?: boolean } +): EyeGazeState; + +function gazeOffsetToBlendShapes( + offsetCm: Vec2, + lookAtDistanceCm: number, + maxGazeAngleRad: number +): AnimationFrame; +``` + +`stepEyeGaze` is pure and directly unit-tested with a scripted `rng` (`gaze.spec.ts`) — every random draw anywhere in the module takes `rng` as a parameter, never calling `Math.random()` itself, the same "push side effects to the edge" shape as `nextAvatarState`. `gazeOffsetToBlendShapes` (`blend-shapes.ts`) is the separate, independently-tested conversion from `EyeGazeState.combinedOffsetCm` to the 8 ARKit `eyeLook*` channels — see its docstring for the left/right eye sign convention, the one place in this whole port that's easy to get backwards. + +`$lib/client/humanoid/eye/gaze-controller.ts`'s `createEyeGazeController(overrides?)` is the client wrapper — owns the per-frame clock (`requestAnimationFrame`) and `Math.random`, mirroring `deriveAvatarState`'s shape (a Svelte-store-compatible `subscribe`, plus imperative setters: `setAvatarState`, `setLookAtDistanceCm`, `setLayerEnabled`, `setTimeScale`, `forceLookAway`, `destroy`). See `/demo/humanoid`'s "Eye system demo" section (`src/routes/demo/humanoid/EyeSystemDemo.svelte`) for the reference wiring against a live Three.js-rendered avatar, including per-layer toggles, a forced-look-away button, and raw macro/micro/nano/combined offset and blend-shape readouts for debugging. + +## `/demo/humanoid` — playground + +`src/routes/demo/humanoid/+page.svelte` is a testing/debugging surface for this package's animation libraries generally, not just a demo of one — it composes `AvatarStateDemo.svelte` (the state-derivation demo above) and `EyeSystemDemo.svelte` side by side, structured so a future library's demo (blink, lip-sync, body/gesture) is another sibling component, not a rewrite. `EyeSystemDemo.svelte`'s `eye-demo/` subfolder holds its Three.js viewer and debug UI — `ThreeAvatarViewer.svelte` is the only file in this whole feature allowed to import `three` or touch mesh/bone data directly; everything under `$lib/humanoid/**` and `$lib/client/humanoid/**` stays renderer-agnostic. The bundled test avatar (`static/models/avatar.glb`, a Ready Player Me-style rig) carries the same 52 ARKit morph targets this package's `ARKitBlendShape` type enumerates, on 4 of its meshes. + ## What's not built here (on purpose) -- The eye look-at, blink, lip-sync, and body/gesture libraries themselves — future work, built as consumers of this contract, not part of it. +- The blink, lip-sync, and body/gesture libraries — future work, built as further consumers of this contract the same way `eye` was, not part of it. - Mic-capture/audio-playback in the browser (still open — see `docs/REALTIME_SESSION.md`'s "What's deliberately not built yet"). - Server-side viseme/phoneme generation — an open design fork (server-side, since the relay already touches raw audio, vs. client-side from the playback buffer) to decide when the lip-sync library actually gets built. - Any protocol change (`$lib/core/protocol.ts` is untouched by this package) — `AvatarState` derivation works entirely from events the protocol already carries. +- The LookAt system (whole-body/head aim toward a target) the source eye-gaze doc describes as `eye`'s complement — `eye`'s `lookAtDistanceCm` input assumes something else eventually owns that; `EyeSystemDemo.svelte` exposes it as a manual slider standing in for that not-yet-built system, the same way the avatar-state demo's "listening" toggle stands in for real mic-capture. diff --git a/package-lock.json b/package-lock.json index 528643c..0cd5082 100644 --- a/package-lock.json +++ b/package-lock.json @@ -19,6 +19,7 @@ "@sveltejs/vite-plugin-svelte": "^7.1.2", "@types/better-sqlite3": "^9.6.0", "@types/node": "^24", + "@types/three": "^0.185.4", "@types/ws": "^8.18.1", "@vitest/browser-playwright": "^4.1.8", "eslint": "^10.4.1", @@ -30,6 +31,7 @@ "prettier-plugin-svelte": "^4.1.0", "svelte": "^5.56.1", "svelte-check": "^4.6.0", + "three": "^0.185.1", "typescript": "^6.0.3", "typescript-eslint": "^8.60.1", "vite": "^8.0.16", @@ -44,6 +46,13 @@ "dev": true, "license": "MIT" }, + "node_modules/@dimforge/rapier3d-compat": { + "version": "0.12.0", + "resolved": "https://registry.npmjs.org/@dimforge/rapier3d-compat/-/rapier3d-compat-0.12.0.tgz", + "integrity": "sha512-uekIGetywIgopfD97oDL5PfeezkFpNhwlzlaEYNOA0N6ghdsOvh/HYjSMek5Q2O1PYvRSDFcqFVJl4r4ZBwOow==", + "dev": true, + "license": "Apache-2.0" + }, "node_modules/@eslint-community/eslint-utils": { "version": "4.10.1", "resolved": "https://registry.npmjs.org/@eslint-community/eslint-utils/-/eslint-utils-4.10.1.tgz", @@ -1281,6 +1290,13 @@ "svelte": "^3 || ^4 || ^5 || ^5.0.0-next.0" } }, + "node_modules/@tweenjs/tween.js": { + "version": "23.1.3", + "resolved": "https://registry.npmjs.org/@tweenjs/tween.js/-/tween.js-23.1.3.tgz", + "integrity": "sha512-vJmvvwFxYuGnF2axRtPYocag6Clbb5YS7kLL+SO/TeVFzHqDIWrNKYtcsPMibjDx9O+bu+psAy9NKfWklassUA==", + "dev": true, + "license": "MIT" + }, "node_modules/@types/better-sqlite3": { "version": "9.6.0", "resolved": "https://registry.npmjs.org/@types/better-sqlite3/-/better-sqlite3-9.6.0.tgz", @@ -1354,6 +1370,28 @@ "dev": true, "license": "MIT" }, + "node_modules/@types/stats.js": { + "version": "0.17.4", + "resolved": "https://registry.npmjs.org/@types/stats.js/-/stats.js-0.17.4.tgz", + "integrity": "sha512-jIBvWWShCvlBqBNIZt0KAshWpvSjhkwkEu4ZUcASoAvhmrgAUI2t1dXrjSL4xXVLB4FznPrIsX3nKXFl/Dt4vA==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/three": { + "version": "0.185.4", + "resolved": "https://registry.npmjs.org/@types/three/-/three-0.185.4.tgz", + "integrity": "sha512-gAsBIC07NIFrxjbf7tH2t71c38uulFfk/RFoC7FNBSjMRAQ8J1x/RBvusX0N5PJouaYFJawXQqfCQ0RKUx/1nA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@dimforge/rapier3d-compat": "~0.12.0", + "@tweenjs/tween.js": "~23.1.3", + "@types/stats.js": "*", + "@types/webxr": ">=0.5.17", + "fflate": "~0.8.2", + "meshoptimizer": "~1.1.1" + } + }, "node_modules/@types/trusted-types": { "version": "2.0.7", "resolved": "https://registry.npmjs.org/@types/trusted-types/-/trusted-types-2.0.7.tgz", @@ -1361,6 +1399,13 @@ "dev": true, "license": "MIT" }, + "node_modules/@types/webxr": { + "version": "0.5.24", + "resolved": "https://registry.npmjs.org/@types/webxr/-/webxr-0.5.24.tgz", + "integrity": "sha512-h8fgEd/DpoS9CBrjEQXR+dIDraopAEfu4wYVNY2tEPwk60stPWhvZMf4Foo5FakuQ7HFZoa8WceaWFervK2Ovg==", + "dev": true, + "license": "MIT" + }, "node_modules/@types/ws": { "version": "8.18.1", "resolved": "https://registry.npmjs.org/@types/ws/-/ws-8.18.1.tgz", @@ -2338,6 +2383,13 @@ } } }, + "node_modules/fflate": { + "version": "0.8.3", + "resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.3.tgz", + "integrity": "sha512-tbZNuJrLwGUp3zshBtdy4W+ORxZuIh8a5ilyIEQDC5rY1f3U20JMry0Ll3WBzU58EZKsEuJFXhb5gwv8CsPvgA==", + "dev": true, + "license": "MIT" + }, "node_modules/file-entry-cache": { "version": "8.0.0", "resolved": "https://registry.npmjs.org/file-entry-cache/-/file-entry-cache-8.0.0.tgz", @@ -2914,6 +2966,13 @@ "@jridgewell/sourcemap-codec": "^1.5.5" } }, + "node_modules/meshoptimizer": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/meshoptimizer/-/meshoptimizer-1.1.1.tgz", + "integrity": "sha512-oRFNWJRDA/WTrVj7NWvqa5HqE1t9MYDj2VaWirQCzCCrAd2GHrqR/sQezCxiWATPNlKTcRaPRHPJwIRoPBAp5g==", + "dev": true, + "license": "MIT" + }, "node_modules/minimatch": { "version": "10.2.6", "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.6.tgz", @@ -3702,6 +3761,13 @@ "url": "https://opencollective.com/eslint" } }, + "node_modules/three": { + "version": "0.185.1", + "resolved": "https://registry.npmjs.org/three/-/three-0.185.1.tgz", + "integrity": "sha512-5aojFCXKwnjBRZvUnt3WFfEcvUJgkN5LlijRFN95hMy8WVkG4I0QNcJE+OuWvuJ0bOdStrbfXn0pkd6/QyiAlg==", + "dev": true, + "license": "MIT" + }, "node_modules/tinybench": { "version": "2.9.0", "resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz", diff --git a/package.json b/package.json index 2236567..df9af94 100644 --- a/package.json +++ b/package.json @@ -25,6 +25,7 @@ "@sveltejs/vite-plugin-svelte": "^7.1.2", "@types/better-sqlite3": "^9.6.0", "@types/node": "^24", + "@types/three": "^0.185.4", "@types/ws": "^8.18.1", "@vitest/browser-playwright": "^4.1.8", "eslint": "^10.4.1", @@ -36,6 +37,7 @@ "prettier-plugin-svelte": "^4.1.0", "svelte": "^5.56.1", "svelte-check": "^4.6.0", + "three": "^0.185.1", "typescript": "^6.0.3", "typescript-eslint": "^8.60.1", "vite": "^8.0.16", diff --git a/src/lib/client/humanoid/eye/gaze-controller.spec.ts b/src/lib/client/humanoid/eye/gaze-controller.spec.ts new file mode 100644 index 0000000..901b48c --- /dev/null +++ b/src/lib/client/humanoid/eye/gaze-controller.spec.ts @@ -0,0 +1,223 @@ +import { describe, expect, it } from 'vitest'; +import type { EyeGazeState } from '../../../humanoid/eye/gaze.js'; +import { createEyeGazeController } from './gaze-controller.js'; + +/** A `scheduleFrame` stand-in the test drives manually, with a fully controlled `nowMs`. */ +function createFakeScheduler() { + let pending: ((nowMs: number) => void) | null = null; + return { + scheduleFrame(callback: (nowMs: number) => void): () => void { + pending = callback; + return () => { + if (pending === callback) pending = null; + }; + }, + fire(nowMs: number) { + const callback = pending; + callback?.(nowMs); + }, + hasPending() { + return pending !== null; + } + }; +} + +const FIXED_RNG = () => 0.4; + +describe('createEyeGazeController', () => { + it('subscribe() calls run immediately with the current state', () => { + const scheduler = createFakeScheduler(); + const controller = createEyeGazeController( + {}, + { rng: FIXED_RNG, scheduleFrame: scheduler.scheduleFrame } + ); + const calls: EyeGazeState[] = []; + controller.subscribe((s) => calls.push(s)); + expect(calls).toHaveLength(1); + expect(calls[0].macro.phase).toBe('atLookAt'); + controller.destroy(); + }); + + it('advances gaze state on a driven frame', () => { + const scheduler = createFakeScheduler(); + const controller = createEyeGazeController( + {}, + { rng: FIXED_RNG, scheduleFrame: scheduler.scheduleFrame } + ); + let latest: EyeGazeState | undefined; + controller.subscribe((s) => (latest = s)); + const initial = latest; + + scheduler.fire(0); // establishes lastFrameMs — no elapsed time yet + scheduler.fire(10_000); // 10 real seconds later — long enough to guarantee some layer redraws + + expect(latest).not.toEqual(initial); + controller.destroy(); + }); + + it("setLookAtDistanceCm affects subsequent frames' macro offset scaling", () => { + const makeController = (lookAtDistanceCm: number) => { + const scheduler = createFakeScheduler(); + const controller = createEyeGazeController( + {}, + { rng: FIXED_RNG, scheduleFrame: scheduler.scheduleFrame } + ); + controller.setLookAtDistanceCm(lookAtDistanceCm); + return { controller, scheduler }; + }; + + const near = makeController(10); + const far = makeController(610); + let nearState: EyeGazeState | undefined; + let farState: EyeGazeState | undefined; + near.controller.subscribe((s) => (nearState = s)); + far.controller.subscribe((s) => (farState = s)); + + near.controller.forceLookAway(); + far.controller.forceLookAway(); + near.scheduler.fire(0); + far.scheduler.fire(0); + // Short enough that the forced look-away's own timer (min 0.3s) can't + // have expired again — `goal` stays exactly the forced value on both, + // only `current` (and therefore the offset) differs by distance scale. + near.scheduler.fire(100); + far.scheduler.fire(100); + + expect(nearState!.macro.goal).toEqual(farState!.macro.goal); + expect(Math.abs(farState!.macroOffsetCm.x)).toBeGreaterThan( + Math.abs(nearState!.macroOffsetCm.x) + ); + + near.controller.destroy(); + far.controller.destroy(); + }); + + it('setAvatarState changes which look-at timer range a subsequent return-to-center draws from', () => { + // A fixed rng > secondaryProbability(0.5) always fails the secondary + // roll, so a forced look-away always returns straight to center on its + // very next expiry — deterministic and avoids depending on how many + // secondaries a lower rng value would chain first. + const rng = () => 0.9; + const makeController = () => { + const scheduler = createFakeScheduler(); + const controller = createEyeGazeController( + {}, + { rng, scheduleFrame: scheduler.scheduleFrame } + ); + return { controller, scheduler }; + }; + + const listening = makeController(); + const thinking = makeController(); + let listeningState: EyeGazeState | undefined; + let thinkingState: EyeGazeState | undefined; + listening.controller.subscribe((s) => (listeningState = s)); + thinking.controller.subscribe((s) => (thinkingState = s)); + + listening.controller.setAvatarState('listening'); + thinking.controller.setAvatarState('thinking'); + listening.controller.forceLookAway(); + thinking.controller.forceLookAway(); + listening.scheduler.fire(0); + thinking.scheduler.fire(0); + // The forced look-away timer draws to 1.2s with rng=0.9 — 2s is + // enough for exactly one return-to-center transition, not two. + listening.scheduler.fire(2000); + thinking.scheduler.fire(2000); + + expect(listeningState!.macro.phase).toBe('atLookAt'); + expect(thinkingState!.macro.phase).toBe('atLookAt'); + expect(listeningState!.macro.timer).not.toBeCloseTo(thinkingState!.macro.timer, 3); + + listening.controller.destroy(); + thinking.controller.destroy(); + }); + + it('setTimeScale scales the effective per-frame deltaSeconds', () => { + const makeController = (scale: number) => { + const scheduler = createFakeScheduler(); + const controller = createEyeGazeController( + {}, + { rng: FIXED_RNG, scheduleFrame: scheduler.scheduleFrame } + ); + controller.setTimeScale(scale); + return { controller, scheduler }; + }; + + const slow = makeController(1); + const fast = makeController(2); + let slowState: EyeGazeState | undefined; + let fastState: EyeGazeState | undefined; + slow.controller.subscribe((s) => (slowState = s)); + fast.controller.subscribe((s) => (fastState = s)); + + slow.controller.forceLookAway(); + fast.controller.forceLookAway(); + slow.scheduler.fire(0); + fast.scheduler.fire(0); + slow.scheduler.fire(100); + fast.scheduler.fire(100); + + expect(slowState!.macro.goal).toEqual(fastState!.macro.goal); + expect(Math.abs(fastState!.macro.current.x)).toBeGreaterThan( + Math.abs(slowState!.macro.current.x) + ); + + slow.controller.destroy(); + fast.controller.destroy(); + }); + + it('setLayerEnabled(false) freezes that layer across subsequent ticks', () => { + const scheduler = createFakeScheduler(); + const controller = createEyeGazeController( + {}, + { rng: FIXED_RNG, scheduleFrame: scheduler.scheduleFrame } + ); + let latest: EyeGazeState | undefined; + controller.subscribe((s) => (latest = s)); + + controller.setLayerEnabled('nano', false); + scheduler.fire(0); + const frozenNano = latest!.nano; + scheduler.fire(5_000); + + expect(latest!.nano).toEqual(frozenNano); + controller.destroy(); + }); + + it('forceLookAway is reflected immediately, without waiting for a tick', () => { + const scheduler = createFakeScheduler(); + const controller = createEyeGazeController( + {}, + { rng: FIXED_RNG, scheduleFrame: scheduler.scheduleFrame } + ); + let latest: EyeGazeState | undefined; + controller.subscribe((s) => (latest = s)); + + expect(latest!.macro.phase).toBe('atLookAt'); + controller.forceLookAway(); + expect(latest!.macro.phase).toBe('lookingAway'); + + controller.destroy(); + }); + + it('destroy() cancels the pending frame and silences further notifications', () => { + const scheduler = createFakeScheduler(); + const controller = createEyeGazeController( + {}, + { rng: FIXED_RNG, scheduleFrame: scheduler.scheduleFrame } + ); + let callCount = 0; + controller.subscribe(() => { + callCount++; + }); + scheduler.fire(0); + const afterFirstFrame = callCount; + + controller.destroy(); + expect(scheduler.hasPending()).toBe(false); + + controller.forceLookAway(); + expect(callCount).toBe(afterFirstFrame); + }); +}); diff --git a/src/lib/client/humanoid/eye/gaze-controller.ts b/src/lib/client/humanoid/eye/gaze-controller.ts new file mode 100644 index 0000000..4f42459 --- /dev/null +++ b/src/lib/client/humanoid/eye/gaze-controller.ts @@ -0,0 +1,130 @@ +import { + resolveEyeConfig, + type DeepPartial, + type EyeConfig +} from '../../../humanoid/eye/config.js'; +import { + createInitialEyeGazeState, + forceMacroLookAway, + stepEyeGaze, + type EyeGazeState, + type LayerToggles +} from '../../../humanoid/eye/gaze.js'; +import type { AvatarState } from '../../../humanoid/state.js'; + +export interface EyeGazeStore { + /** Svelte-store-compatible: calls `run` immediately with the current value, then on every change. */ + subscribe(run: (state: EyeGazeState) => void): () => void; + /** Feeds Macro's per-state look-at timer range (see `$lib/humanoid/eye/config.ts`). */ + setAvatarState(state: AvatarState): void; + /** Feeds Macro's distance-scale correction and the cm-offset -> gaze-angle conversion. */ + setLookAtDistanceCm(cm: number): void; + setLayerEnabled(layer: 'macro' | 'micro' | 'nano', enabled: boolean): void; + /** Multiplies the per-frame `deltaSeconds` fed into `stepEyeGaze` — 1 is real time. */ + setTimeScale(scale: number): void; + /** Forces an immediate macro look-away — see `forceMacroLookAway` in `$lib/humanoid/eye/gaze.ts`. */ + forceLookAway(): void; + /** Stops the frame loop and releases all subscribers. */ + destroy(): void; +} + +export interface EyeGazeControllerOptions { + /** Defaults to `Math.random` — the only place this whole package calls it directly. */ + rng?: () => number; + /** + * Defaults to a real `requestAnimationFrame` loop. Injectable so tests can + * drive frames manually (with a fully controlled `nowMs`) without a real + * browser or DOM — see `gaze-controller.spec.ts`. + */ + scheduleFrame?: (callback: (nowMs: number) => void) => () => void; +} + +const DEFAULT_LOOK_AT_DISTANCE_CM = 80; + +function defaultScheduleFrame(callback: (nowMs: number) => void): () => void { + const id = requestAnimationFrame(callback); + return () => cancelAnimationFrame(id); +} + +/** + * Owns the eye gaze system's per-frame clock and `Math.random` — the + * `client/humanoid/eye` counterpart to `$lib/humanoid/eye`'s pure + * `stepEyeGaze`, mirroring how `client/humanoid/avatar-state.ts` wraps + * `$lib/humanoid/state.ts`'s pure `nextAvatarState`. `overrides` resolves + * through `resolveEyeConfig` (a project only names the knobs it wants to + * change); `options` only exists for testability (`rng`/`scheduleFrame`) + * and isn't meant to be set by real callers. + */ +export function createEyeGazeController( + overrides: DeepPartial = {}, + options: EyeGazeControllerOptions = {} +): EyeGazeStore { + const config = resolveEyeConfig(overrides); + const rng = options.rng ?? Math.random; + const scheduleFrame = options.scheduleFrame ?? defaultScheduleFrame; + + let avatarState: AvatarState = 'idleActive'; + let lookAtDistanceCm = DEFAULT_LOOK_AT_DISTANCE_CM; + let timeScale = 1; + let toggles: LayerToggles = { macro: true, micro: true, nano: true }; + let state: EyeGazeState = createInitialEyeGazeState(config, rng); + + const subscribers = new Set<(state: EyeGazeState) => void>(); + let lastFrameMs: number | null = null; + let destroyed = false; + let cancelScheduled: (() => void) | null = null; + + function notify(): void { + for (const run of subscribers) run(state); + } + + function tick(nowMs: number): void { + if (destroyed) return; + // The very first scheduled frame has no prior timestamp to diff against — + // treat it as a zero-length tick rather than advancing by `nowMs` itself. + const deltaSeconds = lastFrameMs === null ? 0 : Math.max(0, (nowMs - lastFrameMs) / 1000); + lastFrameMs = nowMs; + + state = stepEyeGaze( + state, + { deltaSeconds: deltaSeconds * timeScale, avatarState, lookAtDistanceCm }, + config, + rng, + toggles + ); + notify(); + + cancelScheduled = scheduleFrame(tick); + } + + cancelScheduled = scheduleFrame(tick); + + return { + subscribe(run) { + subscribers.add(run); + run(state); + return () => subscribers.delete(run); + }, + setAvatarState(next) { + avatarState = next; + }, + setLookAtDistanceCm(cm) { + lookAtDistanceCm = cm; + }, + setLayerEnabled(layer, enabled) { + toggles = { ...toggles, [layer]: enabled }; + }, + setTimeScale(scale) { + timeScale = scale; + }, + forceLookAway() { + state = forceMacroLookAway(state, config, rng); + notify(); + }, + destroy() { + destroyed = true; + cancelScheduled?.(); + subscribers.clear(); + } + }; +} diff --git a/src/lib/humanoid/eye/DESIGN_NOTES.md b/src/lib/humanoid/eye/DESIGN_NOTES.md new file mode 100644 index 0000000..d0acd9f --- /dev/null +++ b/src/lib/humanoid/eye/DESIGN_NOTES.md @@ -0,0 +1,24 @@ +# Eye gaze system — design notes + +Rationale that doesn't belong in code comments spread across four files, kept in one place for whoever next has to retune or re-derive this. See `docs/HUMANOID.md`'s "`$lib/humanoid/eye`" section for the user-facing summary this backs up. + +## Source material and its limits + +The port comes from an Unreal Engine avatar project's control-rig design doc: a "Eye-System Variablen & Logik" section giving exact variable names, ranges, defaults, and branching logic for three additive gaze layers (Macro/Micro/Nano), plus three screenshots of hand-authored curve-editor splines. The variable/logic side is complete and ported faithfully (see `gaze.ts`'s docstrings, which cite the doc's own state-machine description directly). The curves are not: a screenshot shows a spline's shape, not its keyframe positions or tangent handles, so `curves.ts`'s `smoothstepSigned`/`biasedSmoothstepSigned`/`easeOutCubic` are documented best-effort approximations chosen to match the _described and visible_ behavior, not pixel-exact reconstructions. If exact curve data ever surfaces (an exported `.curve` asset, or numeric keyframe values), these three functions are the only place that needs revisiting — everything else in `gaze.ts` is agnostic to their exact shape. + +## The spring/"SpringInterpolate" formula + +The doc names a `SpringInterpolate` operation for every layer's `current -> goal` smoothing, described only qualitatively ("initially snappy, then increasingly smooth as it nears the goal") plus a ballpark duration ("should take about 0.125 seconds"), no formula and no mention of overshoot or oscillation. That description is exactly frame-rate-independent exponential decay, not a literal damped spring (which can overshoot) — implemented in `curves.ts` as `current += (goal - current) * (1 - exp(-dt / tau))` with `tau = 0.125` (the doc's ballpark duration, reinterpreted as a decay time constant rather than a fixed completion time, since exponential decay has no finite completion time). + +## cm-offset -> ARKit blend shapes + +This entire conversion (`blend-shapes.ts`) has no source-doc precedent. The original system positions a 2D offset on a plane in front of the avatar's face and (per the doc's `ControlRig` hierarchy — `LookAt_ctrl > LookAtMacroScale_ctrl > LookAtMacro_ctrl > ...`) drives real eye bones from it. This project's avatars are ARKit-blend-shape-only, so the port instead treats the plane offset as subtending a gaze angle (`Math.atan2(offsetCm, lookAtDistanceCm)`) and normalizes that angle by a new `maxGazeAngleRad` config knob (no equivalent in the source doc) to get a `[-1, 1]` blend-weight input. + +Two simplifications worth knowing about: + +- **No per-eye vergence.** A real "aim each eye bone at the same 3D point" rig naturally gives each eye a slightly different angle for a near target (interpupillary distance). This port computes one shared angle and mirrors it onto both eyes' In/Out channels — consistent with the doc's own single-shared-2D-offset framing, but not physically exact for very close look-at targets. +- **`lookAtDistanceCm` is one value, reused twice.** It's the same physical distance the doc uses for Macro's `distance/300 + 0.66` scale correction _and_ the distance this port uses for the angle conversion — deliberately, since both describe "how far away is whatever the avatar is looking at," not two separate concepts. It's always a live runtime value (`EyeGazeInput.lookAtDistanceCm`), never a config default, because the (out-of-scope, not-yet-built) LookAt/body-aim system is what should ultimately own it. + +## Sign convention + +Positive `x` = the avatar's own right; positive `y` = up. See the docstring on `gazeOffsetToBlendShapes` in `blend-shapes.ts` for the left/right eye In/Out derivation — flagged there and here because it's the one place in this whole port that's easy to get backwards without a live model to check against. diff --git a/src/lib/humanoid/eye/blend-shapes.spec.ts b/src/lib/humanoid/eye/blend-shapes.spec.ts new file mode 100644 index 0000000..f37e964 --- /dev/null +++ b/src/lib/humanoid/eye/blend-shapes.spec.ts @@ -0,0 +1,67 @@ +import { describe, expect, it } from 'vitest'; +import { gazeOffsetToAngleRad, gazeOffsetToBlendShapes } from './blend-shapes.js'; + +const MAX_ANGLE = (30 * Math.PI) / 180; + +describe('gazeOffsetToAngleRad', () => { + it('matches a hand-computed atan2 for a known offset/distance pair', () => { + const result = gazeOffsetToAngleRad({ x: 20, y: -10 }, 80); + expect(result.x).toBeCloseTo(Math.atan2(20, 80), 10); + expect(result.y).toBeCloseTo(Math.atan2(-10, 80), 10); + }); + + it('is (0, 0) for a zero offset', () => { + expect(gazeOffsetToAngleRad({ x: 0, y: 0 }, 80)).toEqual({ x: 0, y: 0 }); + }); +}); + +describe('gazeOffsetToBlendShapes', () => { + it('returns an empty frame for a zero offset', () => { + expect(gazeOffsetToBlendShapes({ x: 0, y: 0 }, 80, MAX_ANGLE)).toEqual({}); + }); + + it('a positive-X-only offset drives eyeLookInLeft/eyeLookOutRight, and nothing else', () => { + const frame = gazeOffsetToBlendShapes({ x: 5, y: 0 }, 80, MAX_ANGLE); + expect(frame.eyeLookInLeft).toBeGreaterThan(0); + expect(frame.eyeLookOutRight).toBe(frame.eyeLookInLeft); + expect(frame.eyeLookOutLeft).toBeUndefined(); + expect(frame.eyeLookInRight).toBeUndefined(); + expect(frame.eyeLookUpLeft).toBeUndefined(); + expect(frame.eyeLookDownLeft).toBeUndefined(); + }); + + it('a negative-X-only offset drives eyeLookOutLeft/eyeLookInRight', () => { + const frame = gazeOffsetToBlendShapes({ x: -5, y: 0 }, 80, MAX_ANGLE); + expect(frame.eyeLookOutLeft).toBeGreaterThan(0); + expect(frame.eyeLookInRight).toBe(frame.eyeLookOutLeft); + expect(frame.eyeLookInLeft).toBeUndefined(); + expect(frame.eyeLookOutRight).toBeUndefined(); + }); + + it('a positive-Y-only offset drives both eyeLookUp channels equally', () => { + const frame = gazeOffsetToBlendShapes({ x: 0, y: 5 }, 80, MAX_ANGLE); + expect(frame.eyeLookUpLeft).toBeGreaterThan(0); + expect(frame.eyeLookUpRight).toBe(frame.eyeLookUpLeft); + expect(frame.eyeLookDownLeft).toBeUndefined(); + expect(frame.eyeLookDownRight).toBeUndefined(); + }); + + it('a negative-Y-only offset drives both eyeLookDown channels equally', () => { + const frame = gazeOffsetToBlendShapes({ x: 0, y: -5 }, 80, MAX_ANGLE); + expect(frame.eyeLookDownLeft).toBeGreaterThan(0); + expect(frame.eyeLookDownRight).toBe(frame.eyeLookDownLeft); + expect(frame.eyeLookUpLeft).toBeUndefined(); + }); + + it('clamps an offset beyond maxGazeAngleRad to exactly 1', () => { + const frame = gazeOffsetToBlendShapes({ x: 1000, y: 0 }, 80, MAX_ANGLE); + expect(frame.eyeLookInLeft).toBe(1); + expect(frame.eyeLookOutRight).toBe(1); + }); + + it('combines independent X and Y channels for a diagonal offset', () => { + const frame = gazeOffsetToBlendShapes({ x: 5, y: 5 }, 80, MAX_ANGLE); + expect(frame.eyeLookInLeft).toBeGreaterThan(0); + expect(frame.eyeLookUpLeft).toBeGreaterThan(0); + }); +}); diff --git a/src/lib/humanoid/eye/blend-shapes.ts b/src/lib/humanoid/eye/blend-shapes.ts new file mode 100644 index 0000000..d9fd5a6 --- /dev/null +++ b/src/lib/humanoid/eye/blend-shapes.ts @@ -0,0 +1,72 @@ +import type { AnimationFrame } from '../animation.js'; +import { clamp, type Vec2 } from './curves.js'; + +/** + * Converts a cm-offset on the gaze plane into a gaze angle (radians) per + * axis, treating `offsetCm` as a point on a plane `lookAtDistanceCm` in + * front of the eyes. `Math.atan2` rather than a linear divide, so the + * relationship is physically correct (the same cm offset subtends a smaller + * angle at a greater distance) rather than an approximation valid only for + * small angles. + */ +export function gazeOffsetToAngleRad(offsetCm: Vec2, lookAtDistanceCm: number): Vec2 { + return { + x: Math.atan2(offsetCm.x, lookAtDistanceCm), + y: Math.atan2(offsetCm.y, lookAtDistanceCm) + }; +} + +/** + * Converts a gaze-plane cm-offset into ARKit `eyeLook*` blend-shape weights. + * This whole conversion is a web-port adaptation with no Unreal precedent — + * the original design doc drives real eye bones via a control rig, not + * blend shapes; this project's avatars are ARKit-blend-shape-only (see + * docs/HUMANOID.md), so the plane offset is instead turned into a gaze + * angle (`gazeOffsetToAngleRad`) and normalized by `maxGazeAngleRad`, the + * angle at which a channel reaches full weight. + * + * Sign convention (the one place this is easy to get backwards): positive + * `offsetCm.x` means the avatar looks toward its own right. That's a + * conjugate gaze shift — both eyes rotate the same rotational direction — + * so the *left* eye (sitting on the avatar's left) rotates toward the nose + * (`eyeLookInLeft`), while the *right* eye (already on the right) rotates + * further from the nose (`eyeLookOutRight`). Negative `x` mirrors this onto + * `eyeLookOutLeft`/`eyeLookInRight`. Positive `offsetCm.y` (up) drives + * `eyeLookUpLeft`/`eyeLookUpRight` equally; negative (down) drives + * `eyeLookDownLeft`/`eyeLookDownRight` equally — no left/right ambiguity on + * this axis. Per-eye vergence (a closer look-at target producing a larger + * angle for one eye than the other) isn't modeled, matching the source + * doc's single shared 2D offset — see DESIGN_NOTES.md. + * + * Output is sparse: only the channels a nonzero offset actually drives are + * present. + */ +export function gazeOffsetToBlendShapes( + offsetCm: Vec2, + lookAtDistanceCm: number, + maxGazeAngleRad: number +): AnimationFrame { + const angle = gazeOffsetToAngleRad(offsetCm, lookAtDistanceCm); + const horizontal = clamp(angle.x / maxGazeAngleRad, -1, 1); + const vertical = clamp(angle.y / maxGazeAngleRad, -1, 1); + + const frame: AnimationFrame = {}; + + if (horizontal > 0) { + frame.eyeLookInLeft = horizontal; + frame.eyeLookOutRight = horizontal; + } else if (horizontal < 0) { + frame.eyeLookOutLeft = -horizontal; + frame.eyeLookInRight = -horizontal; + } + + if (vertical > 0) { + frame.eyeLookUpLeft = vertical; + frame.eyeLookUpRight = vertical; + } else if (vertical < 0) { + frame.eyeLookDownLeft = -vertical; + frame.eyeLookDownRight = -vertical; + } + + return frame; +} diff --git a/src/lib/humanoid/eye/config.spec.ts b/src/lib/humanoid/eye/config.spec.ts new file mode 100644 index 0000000..cf28475 --- /dev/null +++ b/src/lib/humanoid/eye/config.spec.ts @@ -0,0 +1,63 @@ +import { describe, expect, it } from 'vitest'; +import { DEFAULT_EYE_CONFIG, resolveEyeConfig } from './config.js'; + +describe('resolveEyeConfig', () => { + it('returns DEFAULT_EYE_CONFIG when called with no overrides', () => { + expect(resolveEyeConfig()).toEqual(DEFAULT_EYE_CONFIG); + }); + + it('merges a leaf override in macro without disturbing its siblings', () => { + const resolved = resolveEyeConfig({ macro: { secondaryProbability: 0.9 } }); + expect(resolved.macro).toEqual({ ...DEFAULT_EYE_CONFIG.macro, secondaryProbability: 0.9 }); + expect(resolved.micro).toEqual(DEFAULT_EYE_CONFIG.micro); + expect(resolved.nano).toEqual(DEFAULT_EYE_CONFIG.nano); + }); + + it('merges a leaf override inside macro.offsetStrengthCm without dropping the other axis', () => { + const resolved = resolveEyeConfig({ macro: { offsetStrengthCm: { x: 99 } } }); + expect(resolved.macro.offsetStrengthCm).toEqual({ + x: 99, + y: DEFAULT_EYE_CONFIG.macro.offsetStrengthCm.y + }); + }); + + it('merges a partial lookAtLengthMinMaxByState override without dropping the other states', () => { + const resolved = resolveEyeConfig({ + macro: { lookAtLengthMinMaxByState: { talking: [1, 2] } } + }); + expect(resolved.macro.lookAtLengthMinMaxByState.talking).toEqual([1, 2]); + expect(resolved.macro.lookAtLengthMinMaxByState.listening).toEqual( + DEFAULT_EYE_CONFIG.macro.lookAtLengthMinMaxByState.listening + ); + }); + + it('merges a leaf override in micro without disturbing macro/nano', () => { + const resolved = resolveEyeConfig({ micro: { offsetStrengthCm: { y: 5 } } }); + expect(resolved.micro.offsetStrengthCm).toEqual({ + x: DEFAULT_EYE_CONFIG.micro.offsetStrengthCm.x, + y: 5 + }); + expect(resolved.macro).toEqual(DEFAULT_EYE_CONFIG.macro); + }); + + it('merges a leaf override in nano', () => { + const resolved = resolveEyeConfig({ nano: { offsetStrengthCm: 1.5 } }); + expect(resolved.nano).toEqual({ ...DEFAULT_EYE_CONFIG.nano, offsetStrengthCm: 1.5 }); + }); + + it('merges top-level scalar overrides', () => { + const resolved = resolveEyeConfig({ springTimeConstantSeconds: 0.5, maxGazeAngleRad: 1 }); + expect(resolved.springTimeConstantSeconds).toBe(0.5); + expect(resolved.maxGazeAngleRad).toBe(1); + }); + + it('does not mutate DEFAULT_EYE_CONFIG (deep check)', () => { + const snapshot = JSON.parse(JSON.stringify(DEFAULT_EYE_CONFIG)); + resolveEyeConfig({ + macro: { secondaryProbability: 0.1, offsetStrengthCm: { x: 1 } }, + micro: { offsetStrengthCm: { x: 1 } }, + nano: { offsetStrengthCm: 1 } + }); + expect(DEFAULT_EYE_CONFIG).toEqual(snapshot); + }); +}); diff --git a/src/lib/humanoid/eye/config.ts b/src/lib/humanoid/eye/config.ts new file mode 100644 index 0000000..292bbce --- /dev/null +++ b/src/lib/humanoid/eye/config.ts @@ -0,0 +1,145 @@ +import type { AvatarState } from '../state.js'; + +/** + * Recursively-partial variant of a config type, used by `resolveEyeConfig`'s + * overrides parameter. Arrays/tuples (e.g. a `[min, max]` range) are treated + * as atomic — overriding one means supplying the whole tuple, not patching + * one element — everything else recurses field-by-field. + */ +export type DeepPartial = T extends readonly unknown[] + ? T + : T extends object + ? { [K in keyof T]?: DeepPartial } + : T; + +export interface EyeMacroConfig { + /** Seconds spent at the look-at target before the next look-away, keyed by the current `AvatarState`. */ + lookAtLengthMinMaxByState: Record; + /** Seconds spent looking away before returning or chaining a secondary. */ + lookAwayLengthMinMax: [number, number]; + /** Seconds spent looking away while chaining a secondary glance. */ + lookAwayLengthSecondaryMinMax: [number, number]; + /** Chance (0-1) a secondary glance is chained once a look-away expires. */ + secondaryProbability: number; + /** A secondary can only be chained while `|goal.x|` and `|goal.y|` are both within this. */ + secondaryCutoff: number; + /** Max magnitude of the additional offset a secondary glance adds onto the current goal. */ + secondaryMaxStrength: number; + /** Max cm offset on each axis at `goal` magnitude 1, before distance scaling. */ + offsetStrengthCm: { x: number; y: number }; + /** Reference distance (cm) in the `distance/reference + floor` scale formula. */ + distanceScaleReferenceCm: number; + /** Minimum distance scale (reached as distance approaches 0). */ + distanceScaleFloor: number; + /** Bias passed to `biasedSmoothstepSigned` when drawing the Y axis of a macro goal. */ + yCurveBias: number; +} + +export interface EyeMicroConfig { + /** Seconds range for the X-axis micro timer, remapped through `easeOutCubic` on redraw. */ + xTimerMinMax: [number, number]; + /** Seconds range for the Y-axis micro timer, remapped through `easeOutCubic` on redraw. */ + yTimerMinMax: [number, number]; + /** Max cm offset on each axis (goal is always -1, 0, or 1). */ + offsetStrengthCm: { x: number; y: number }; +} + +export interface EyeNanoConfig { + /** Seconds range for the (single, unremapped) nano timer. */ + timerMinMax: [number, number]; + /** Max cm offset magnitude (goal direction is unit-length, so this is the only strength term). */ + offsetStrengthCm: number; +} + +export interface EyeConfig { + macro: EyeMacroConfig; + micro: EyeMicroConfig; + nano: EyeNanoConfig; + /** Shared exponential-decay time constant (seconds) all three layers' `current` springs toward `goal` with. */ + springTimeConstantSeconds: number; + /** + * Gaze angle (radians) at which `gazeOffsetToBlendShapes` (blend-shapes.ts) + * saturates a channel to weight 1. No `lookAtDistanceCm` default lives + * here — that's always a live runtime value (see `EyeGazeInput` in + * gaze.ts), not a config constant, since it's also what Macro's distance + * scaling reads every tick. + */ + maxGazeAngleRad: number; +} + +/** + * Defaults taken directly from the ported design doc's "Eye-System + * Variablen & Logik" section, except `yCurveBias` and `maxGazeAngleRad` + * (both new web-port knobs — see DESIGN_NOTES.md). + */ +export const DEFAULT_EYE_CONFIG: EyeConfig = { + macro: { + lookAtLengthMinMaxByState: { + listening: [7, 15], + thinking: [3, 10], + talking: [3, 10], + idleActive: [3, 9], + idlePassive: [2, 9] + }, + lookAwayLengthMinMax: [0.3, 1.3], + lookAwayLengthSecondaryMinMax: [0.3, 0.8], + secondaryProbability: 0.5, + secondaryCutoff: 0.7, + secondaryMaxStrength: 0.2, + offsetStrengthCm: { x: 20, y: 15 }, + distanceScaleReferenceCm: 300, + distanceScaleFloor: 0.66, + yCurveBias: 0.5 + }, + micro: { + xTimerMinMax: [0.05, 6], + yTimerMinMax: [0.05, 4], + offsetStrengthCm: { x: 3, y: 1 } + }, + nano: { + timerMinMax: [0.05, 2.0], + offsetStrengthCm: 0.5 + }, + springTimeConstantSeconds: 0.125, + maxGazeAngleRad: (30 * Math.PI) / 180 +}; + +/** + * Merges a project's partial overrides onto `DEFAULT_EYE_CONFIG`. Unlike + * `resolveAnimationConfig`'s flat `{...DEFAULT, ...overrides}` (every field + * of `AnimationConfig` is a scalar), `EyeConfig`'s fields are themselves + * objects, so this merges each nested section independently — a caller + * overriding only `macro.secondaryProbability` must not blank out the rest + * of `macro`, or `micro`/`nano`/the top-level scalars. + */ +export function resolveEyeConfig(overrides: DeepPartial = {}): EyeConfig { + return { + macro: { + ...DEFAULT_EYE_CONFIG.macro, + ...overrides.macro, + lookAtLengthMinMaxByState: { + ...DEFAULT_EYE_CONFIG.macro.lookAtLengthMinMaxByState, + ...overrides.macro?.lookAtLengthMinMaxByState + }, + offsetStrengthCm: { + ...DEFAULT_EYE_CONFIG.macro.offsetStrengthCm, + ...overrides.macro?.offsetStrengthCm + } + }, + micro: { + ...DEFAULT_EYE_CONFIG.micro, + ...overrides.micro, + offsetStrengthCm: { + ...DEFAULT_EYE_CONFIG.micro.offsetStrengthCm, + ...overrides.micro?.offsetStrengthCm + } + }, + nano: { + ...DEFAULT_EYE_CONFIG.nano, + ...overrides.nano + }, + springTimeConstantSeconds: + overrides.springTimeConstantSeconds ?? DEFAULT_EYE_CONFIG.springTimeConstantSeconds, + maxGazeAngleRad: overrides.maxGazeAngleRad ?? DEFAULT_EYE_CONFIG.maxGazeAngleRad + }; +} diff --git a/src/lib/humanoid/eye/curves.spec.ts b/src/lib/humanoid/eye/curves.spec.ts new file mode 100644 index 0000000..9c746fd --- /dev/null +++ b/src/lib/humanoid/eye/curves.spec.ts @@ -0,0 +1,143 @@ +import { describe, expect, it } from 'vitest'; +import { + biasedSmoothstepSigned, + clamp, + easeOutCubic, + normalizeVec2, + smoothstepSigned, + springTowardScalar, + springTowardVec2 +} from './curves.js'; + +describe('clamp', () => { + it('passes values inside the range through unchanged', () => { + expect(clamp(0.5, -1, 1)).toBe(0.5); + }); + + it('clamps values outside the range to the nearest bound', () => { + expect(clamp(-2, -1, 1)).toBe(-1); + expect(clamp(2, -1, 1)).toBe(1); + }); +}); + +describe('smoothstepSigned', () => { + it('passes through the three fixed points', () => { + expect(smoothstepSigned(-1)).toBe(-1); + expect(smoothstepSigned(0)).toBe(0); + expect(smoothstepSigned(1)).toBe(1); + }); + + it('is odd-symmetric', () => { + for (const x of [0.1, 0.3, 0.7, 0.95]) { + expect(smoothstepSigned(-x)).toBeCloseTo(-smoothstepSigned(x), 10); + } + }); + + it('is monotonically increasing', () => { + const samples = Array.from({ length: 21 }, (_, i) => -1 + (i / 20) * 2); + const values = samples.map(smoothstepSigned); + for (let i = 1; i < values.length; i++) { + expect(values[i]).toBeGreaterThanOrEqual(values[i - 1]); + } + }); + + it('biases a uniform-random input toward the extremes (steep center, flat ends)', () => { + // Equal-width input bands near the extremes map to a wider output band + // than the same-width band at the center — the "makes values more + // extreme" effect described in the doc. + const edgeSpread = smoothstepSigned(-0.9) - smoothstepSigned(-1); + const centerSpread = smoothstepSigned(0.1) - smoothstepSigned(-0.1); + expect(centerSpread).toBeGreaterThan(edgeSpread); + }); +}); + +describe('biasedSmoothstepSigned', () => { + it('passes through the three fixed points', () => { + expect(biasedSmoothstepSigned(-1, 0.5)).toBeCloseTo(-1, 10); + expect(biasedSmoothstepSigned(0, 0.5)).toBe(0); + expect(biasedSmoothstepSigned(1, 0.5)).toBeCloseTo(1, 10); + }); + + it('matches smoothstepSigned exactly at and below the origin', () => { + for (const x of [-1, -0.7, -0.3, 0]) { + expect(biasedSmoothstepSigned(x, 0.5)).toBe(smoothstepSigned(x)); + } + }); + + it('exceeds smoothstepSigned above the origin (the documented upward shift)', () => { + for (const x of [0.1, 0.3, 0.5, 0.8]) { + expect(biasedSmoothstepSigned(x, 0.5)).toBeGreaterThan(smoothstepSigned(x)); + } + }); + + it('reaches its plateau near 1 well before x=1', () => { + expect(biasedSmoothstepSigned(0.8, 0.5)).toBeGreaterThan(0.95); + }); + + it('guards against a non-positive bias instead of producing NaN/Infinity', () => { + expect(Number.isFinite(biasedSmoothstepSigned(0.5, 0))).toBe(true); + expect(Number.isFinite(biasedSmoothstepSigned(0.5, -1))).toBe(true); + }); +}); + +describe('easeOutCubic', () => { + it('passes through the two fixed points', () => { + expect(easeOutCubic(0)).toBe(0); + expect(easeOutCubic(1)).toBe(1); + }); + + it('is monotonically increasing', () => { + const samples = Array.from({ length: 11 }, (_, i) => i / 10); + const values = samples.map(easeOutCubic); + for (let i = 1; i < values.length; i++) { + expect(values[i]).toBeGreaterThan(values[i - 1]); + } + }); + + it('front-loads growth, biasing a uniform input toward larger outputs', () => { + expect(easeOutCubic(0.5)).toBeGreaterThan(0.5); + }); + + it('clamps input outside [0, 1]', () => { + expect(easeOutCubic(-1)).toBe(0); + expect(easeOutCubic(2)).toBe(1); + }); +}); + +describe('springTowardScalar', () => { + it('does not move for dt=0', () => { + expect(springTowardScalar(0, 10, 0, 0.125)).toBe(0); + }); + + it('moves partway toward the goal for a finite dt', () => { + const result = springTowardScalar(0, 10, 0.125, 0.125); + expect(result).toBeGreaterThan(0); + expect(result).toBeLessThan(10); + }); + + it('converges to the goal as dt grows large relative to tau', () => { + const result = springTowardScalar(0, 10, 100 * 0.125, 0.125); + expect(result).toBeCloseTo(10, 5); + }); +}); + +describe('springTowardVec2', () => { + it('applies springTowardScalar independently per axis', () => { + const result = springTowardVec2({ x: 0, y: 5 }, { x: 10, y: -5 }, 0.125, 0.125); + expect(result.x).toBe(springTowardScalar(0, 10, 0.125, 0.125)); + expect(result.y).toBe(springTowardScalar(5, -5, 0.125, 0.125)); + }); +}); + +describe('normalizeVec2', () => { + it('returns a unit-length vector for a nonzero input', () => { + const result = normalizeVec2({ x: 3, y: 4 }); + expect(Math.hypot(result.x, result.y)).toBeCloseTo(1, 10); + expect(result.x).toBeCloseTo(0.6, 10); + expect(result.y).toBeCloseTo(0.8, 10); + }); + + it('passes a zero-length input through unchanged instead of dividing by zero', () => { + expect(normalizeVec2({ x: 0, y: 0 })).toEqual({ x: 0, y: 0 }); + }); +}); diff --git a/src/lib/humanoid/eye/curves.ts b/src/lib/humanoid/eye/curves.ts new file mode 100644 index 0000000..798118e --- /dev/null +++ b/src/lib/humanoid/eye/curves.ts @@ -0,0 +1,99 @@ +/** + * Small math/easing primitives for the eye gaze system's Macro/Micro/Nano + * layers (see gaze.ts and DESIGN_NOTES.md). Kept separate from gaze.ts so + * each curve's shape can be asserted directly and retuned without touching + * the state machine. + */ + +export interface Vec2 { + x: number; + y: number; +} + +export function clamp(value: number, min: number, max: number): number { + return Math.min(max, Math.max(min, value)); +} + +/** + * Symmetric ease through (-1,-1) -> (0,0) -> (1,1) with zero tangent at both + * ends (a signed remap of the classic 3t^2-2t^3 smoothstep). Approximates + * the Unreal "LookAt_Macro_Strength_X" curve asset — only a screenshot of + * the hand-authored spline was available, not its keyframe/tangent data, so + * this is a best-effort match to the visible shape rather than a pixel-exact + * port (see DESIGN_NOTES.md). Because the derivative is near zero at the + * extremes and highest at the center, remapping a uniform-random input + * through this curve biases the *output* toward the extremes, not the + * center — matching the doc's documented intent ("tends to make values more + * extreme"). + */ +export function smoothstepSigned(x: number): number { + const t = clamp((x + 1) / 2, 0, 1); + const s = 3 * t * t - 2 * t * t * t; + return 2 * s - 1; +} + +/** + * Approximates the "LookAt_Macro_Strength_Y" curve, whose screenshot shows + * it tracking `smoothstepSigned` closely below the origin but plateauing at + * +1 well before x=1 — the documented "shifts values further upward, so the + * avatar looks up more than down". Modeled as `smoothstepSigned` for x<=0, + * and for x>0 a Perlin-style input bias (`t' = t^bias`) applied before the + * smoothstep evaluation so the plateau arrives early while endpoints (0->0, + * 1->1) and the lower half stay untouched. `bias` is clamped away from zero + * to avoid a `pow` degeneracy. + */ +export function biasedSmoothstepSigned(x: number, bias: number): number { + if (x <= 0) return smoothstepSigned(x); + const safeBias = Math.max(bias, 1e-3); + const t = clamp(Math.pow(x, safeBias), 0, 1); + return 3 * t * t - 2 * t * t * t; +} + +/** + * Ease-out cubic on [0,1]: steep at t=0, flattening to a plateau at t=1. + * Approximates the "LookAt_Micro_Timer" curve, whose documented effect is + * that a uniform-random `t` remapped through it is "somewhat less likely to + * land on a small value than a large one". + */ +export function easeOutCubic(t: number): number { + const clamped = clamp(t, 0, 1); + return 1 - Math.pow(1 - clamped, 3); +} + +/** + * Frame-rate-independent exponential-decay smoothing of `current` toward + * `goal` — the doc's "SpringInterpolate" ("initially snappy, then + * increasingly smooth as it nears the goal", no overshoot described). No + * exact formula is given, only a ballpark "should take about 0.125 + * seconds"; that value is treated here as the decay time constant `tau`, + * not a fixed completion duration — see DESIGN_NOTES.md. + */ +export function springTowardScalar( + current: number, + goal: number, + dtSeconds: number, + tauSeconds: number +): number { + if (dtSeconds <= 0) return current; + const alpha = 1 - Math.exp(-dtSeconds / Math.max(tauSeconds, 1e-6)); + return current + (goal - current) * alpha; +} + +export function springTowardVec2( + current: Vec2, + goal: Vec2, + dtSeconds: number, + tauSeconds: number +): Vec2 { + return { + x: springTowardScalar(current.x, goal.x, dtSeconds, tauSeconds), + y: springTowardScalar(current.y, goal.y, dtSeconds, tauSeconds) + }; +} + +/** Zero-length input passes through unchanged rather than dividing by zero. */ +export function normalizeVec2(v: Vec2): Vec2 { + const length = Math.hypot(v.x, v.y); + if (length === 0) return { x: 0, y: 0 }; + return { x: v.x / length, y: v.y / length }; +} diff --git a/src/lib/humanoid/eye/gaze.spec.ts b/src/lib/humanoid/eye/gaze.spec.ts new file mode 100644 index 0000000..c303485 --- /dev/null +++ b/src/lib/humanoid/eye/gaze.spec.ts @@ -0,0 +1,423 @@ +import { describe, expect, it } from 'vitest'; +import { DEFAULT_EYE_CONFIG } from './config.js'; +import { + biasedSmoothstepSigned, + clamp, + easeOutCubic, + normalizeVec2, + smoothstepSigned, + springTowardVec2 +} from './curves.js'; +import { + createInitialEyeGazeState, + forceMacroLookAway, + stepEyeGaze, + type EyeGazeState +} from './gaze.js'; + +/** Returns a fixed sequence of values; throws if called more times than scripted (proves "no extra draw happened"). */ +function scriptedRng(values: number[]): () => number { + let i = 0; + return () => { + if (i >= values.length) throw new Error(`scriptedRng exhausted after ${i} calls`); + return values[i++]; + }; +} + +const ZERO = { x: 0, y: 0 }; + +/** A state where nothing is about to expire (timer=100) unless a test overrides a specific layer. */ +function baseState(overrides: Partial = {}): EyeGazeState { + return { + macro: { phase: 'atLookAt', timer: 100, goal: { ...ZERO }, current: { ...ZERO } }, + micro: { + x: { goal: 0, current: 0, timer: 100 }, + y: { goal: 0, current: 0, timer: 100 } + }, + nano: { goal: { ...ZERO }, current: { ...ZERO }, timer: 100 }, + macroOffsetCm: { ...ZERO }, + microOffsetCm: { ...ZERO }, + nanoOffsetCm: { ...ZERO }, + combinedOffsetCm: { ...ZERO }, + ...overrides + }; +} + +const config = DEFAULT_EYE_CONFIG; + +describe('createInitialEyeGazeState', () => { + it("starts Macro atLookAt, at rest, per the doc's startup rule", () => { + const state = createInitialEyeGazeState(config, scriptedRng([0.5, 0.5, 0.5, 0.5])); + expect(state.macro.phase).toBe('atLookAt'); + expect(state.macro.goal).toEqual(ZERO); + expect(state.macro.current).toEqual(ZERO); + expect(state.macro.timer).toBeGreaterThan(0); + }); + + it('starts Micro and Nano at rest with a randomly-drawn first timer', () => { + const state = createInitialEyeGazeState(config, scriptedRng([0.5, 0.5, 0.5, 0.5])); + expect(state.micro.x.goal).toBe(0); + expect(state.micro.y.goal).toBe(0); + expect(state.nano.goal).toEqual(ZERO); + expect(state.micro.x.timer).toBeGreaterThan(0); + expect(state.micro.y.timer).toBeGreaterThan(0); + expect(state.nano.timer).toBeGreaterThan(0); + }); + + it('starts with all offsets at zero', () => { + const state = createInitialEyeGazeState(config, scriptedRng([0.5, 0.5, 0.5, 0.5])); + expect(state.combinedOffsetCm).toEqual(ZERO); + }); +}); + +describe('stepEyeGaze — macro', () => { + it('transitions atLookAt -> lookingAway on expiry, drawing a goal via the curve functions', () => { + const state = baseState({ + macro: { phase: 'atLookAt', timer: 0.05, goal: { ...ZERO }, current: { ...ZERO } } + }); + const rng = scriptedRng([0.75, 0.25, 0.5]); + const result = stepEyeGaze( + state, + { deltaSeconds: 0.1, avatarState: 'idleActive', lookAtDistanceCm: 80 }, + config, + rng, + { micro: false, nano: false } + ); + + const expectedGoal = { + x: smoothstepSigned(0.75 * 2 - 1), + y: biasedSmoothstepSigned(0.25 * 2 - 1, config.macro.yCurveBias) + }; + const [min, max] = config.macro.lookAwayLengthMinMax; + + expect(result.macro.phase).toBe('lookingAway'); + expect(result.macro.goal.x).toBeCloseTo(expectedGoal.x, 10); + expect(result.macro.goal.y).toBeCloseTo(expectedGoal.y, 10); + expect(result.macro.timer).toBeCloseTo(0.05 - 0.1 + (min + 0.5 * (max - min)), 10); + expect(result.macro.current).toEqual( + springTowardVec2(ZERO, result.macro.goal, 0.1, config.springTimeConstantSeconds) + ); + }); + + it('ignores avatarState changes while the current countdown has not expired', () => { + const state = baseState({ + macro: { phase: 'atLookAt', timer: 5, goal: { ...ZERO }, current: { ...ZERO } } + }); + const rng = scriptedRng([]); // any draw here would mean avatarState triggered a redraw it shouldn't have + const result = stepEyeGaze( + state, + { deltaSeconds: 1, avatarState: 'talking', lookAtDistanceCm: 80 }, + config, + rng, + { micro: false, nano: false } + ); + expect(result.macro.phase).toBe('atLookAt'); + expect(result.macro.timer).toBeCloseTo(4, 10); + }); + + it('keys the next look-at timer range by the current avatarState on return-to-center', () => { + const runWithState = (avatarState: 'listening' | 'thinking') => + stepEyeGaze( + baseState({ + macro: { + phase: 'lookingAway', + timer: -0.009, + goal: { x: 0.1, y: 0.1 }, + current: { ...ZERO } + } + }), + { deltaSeconds: 0, avatarState, lookAtDistanceCm: 80 }, + config, + scriptedRng([0.9, 0.5]), // r=0.9 fails the secondary roll -> return-to-center branch + { micro: false, nano: false } + ); + + const listening = runWithState('listening'); + const thinking = runWithState('thinking'); + const [listenMin, listenMax] = config.macro.lookAtLengthMinMaxByState.listening; + const [thinkMin, thinkMax] = config.macro.lookAtLengthMinMaxByState.thinking; + + expect(listening.macro.phase).toBe('atLookAt'); + expect(listening.macro.goal).toEqual(ZERO); + expect(listening.macro.timer).toBeCloseTo( + -0.009 + (listenMin + 0.5 * (listenMax - listenMin)), + 10 + ); + expect(thinking.macro.timer).toBeCloseTo(-0.009 + (thinkMin + 0.5 * (thinkMax - thinkMin)), 10); + expect(listening.macro.timer).not.toBeCloseTo(thinking.macro.timer, 3); + }); + + it('returns to center if the roll succeeds but the goal exceeds the secondary cutoff', () => { + const state = baseState({ + macro: { phase: 'lookingAway', timer: -0.01, goal: { x: 0.9, y: 0 }, current: { ...ZERO } } + }); + const rng = scriptedRng([0.0, 0.5]); // r=0 rolls secondary, but goal.x=0.9 exceeds the 0.7 cutoff + const result = stepEyeGaze( + state, + { deltaSeconds: 0, avatarState: 'idlePassive', lookAtDistanceCm: 80 }, + config, + rng, + { micro: false, nano: false } + ); + expect(result.macro.phase).toBe('atLookAt'); + expect(result.macro.goal).toEqual(ZERO); + }); + + it('chains a secondary glance when the roll succeeds and the goal is within the cutoff', () => { + const initialGoal = { x: 0.5, y: 0.5 }; + const state = baseState({ + macro: { phase: 'lookingAway', timer: -0.01, goal: initialGoal, current: { ...ZERO } } + }); + const rng = scriptedRng([0.1, 0.75, 0.25, 0.5]); // r=0.1 rolls secondary; within cutoff + const result = stepEyeGaze( + state, + { deltaSeconds: 0, avatarState: 'talking', lookAtDistanceCm: 80 }, + config, + rng, + { micro: false, nano: false } + ); + + const deltaX = smoothstepSigned(0.75 * 2 - 1) * config.macro.secondaryMaxStrength; + const deltaY = smoothstepSigned(0.25 * 2 - 1) * config.macro.secondaryMaxStrength; + const [secMin, secMax] = config.macro.lookAwayLengthSecondaryMinMax; + + expect(result.macro.phase).toBe('lookingAway'); + expect(result.macro.goal.x).toBeCloseTo(clamp(initialGoal.x + deltaX, -1, 1), 10); + expect(result.macro.goal.y).toBeCloseTo(clamp(initialGoal.y + deltaY, -1, 1), 10); + expect(result.macro.timer).toBeCloseTo(-0.01 + (secMin + 0.5 * (secMax - secMin)), 10); + }); + + it('scales the macro cm-offset by distance, and never below the floor', () => { + const state = baseState({ + macro: { phase: 'atLookAt', timer: 100, goal: { ...ZERO }, current: { x: 1, y: 1 } } + }); + const rng = scriptedRng([]); + const near = stepEyeGaze( + state, + { deltaSeconds: 0, avatarState: 'idleActive', lookAtDistanceCm: 0 }, + config, + rng, + { + micro: false, + nano: false + } + ); + const far = stepEyeGaze( + state, + { deltaSeconds: 0, avatarState: 'idleActive', lookAtDistanceCm: 600 }, + config, + rng, + { + micro: false, + nano: false + } + ); + expect(near.macroOffsetCm.x).toBeCloseTo( + 1 * config.macro.offsetStrengthCm.x * config.macro.distanceScaleFloor, + 10 + ); + expect(far.macroOffsetCm.x).toBeGreaterThan(near.macroOffsetCm.x); + }); +}); + +describe('stepEyeGaze — micro', () => { + it('advances the X and Y timers independently', () => { + const state = baseState({ + micro: { x: { goal: 0, current: 0, timer: 0.01 }, y: { goal: 0, current: 0, timer: 100 } } + }); + const rng = scriptedRng([0.5, 0.5]); + const result = stepEyeGaze( + state, + { deltaSeconds: 0.02, avatarState: 'idleActive', lookAtDistanceCm: 80 }, + config, + rng, + { macro: false, nano: false } + ); + expect(result.micro.x.goal).not.toBe(0); + expect(result.micro.y.goal).toBe(0); + expect(result.micro.y.timer).toBeCloseTo(100 - 0.02, 10); + }); + + it('always redraws a goal different from the current value', () => { + const fromZero = baseState({ + micro: { x: { goal: 0, current: 0, timer: -0.01 }, y: { goal: 0, current: 0, timer: 100 } } + }); + const pickLow = stepEyeGaze( + fromZero, + { deltaSeconds: 0, avatarState: 'idleActive', lookAtDistanceCm: 80 }, + config, + scriptedRng([0.1, 0.5]), + { macro: false, nano: false } + ); + const pickHigh = stepEyeGaze( + fromZero, + { deltaSeconds: 0, avatarState: 'idleActive', lookAtDistanceCm: 80 }, + config, + scriptedRng([0.9, 0.5]), + { macro: false, nano: false } + ); + expect(pickLow.micro.x.goal).not.toBe(0); + expect(pickHigh.micro.x.goal).not.toBe(0); + expect(pickLow.micro.x.goal).not.toBe(pickHigh.micro.x.goal); + + const fromNegativeOne = baseState({ + micro: { x: { goal: -1, current: -1, timer: -0.01 }, y: { goal: 0, current: 0, timer: 100 } } + }); + const redrawn = stepEyeGaze( + fromNegativeOne, + { deltaSeconds: 0, avatarState: 'idleActive', lookAtDistanceCm: 80 }, + config, + scriptedRng([0.1, 0.5]), + { macro: false, nano: false } + ); + expect(redrawn.micro.x.goal).not.toBe(-1); + }); + + it('remaps the redrawn timer duration through easeOutCubic, not linearly', () => { + const state = baseState({ + micro: { x: { goal: 0, current: 0, timer: -0.01 }, y: { goal: 0, current: 0, timer: 100 } } + }); + const rawT = 0.3; + const result = stepEyeGaze( + state, + { deltaSeconds: 0, avatarState: 'idleActive', lookAtDistanceCm: 80 }, + config, + scriptedRng([0.9, rawT]), + { macro: false, nano: false } + ); + const [min, max] = config.micro.xTimerMinMax; + const curvedDuration = min + easeOutCubic(rawT) * (max - min); + const linearDuration = min + rawT * (max - min); + expect(result.micro.x.timer).toBeCloseTo(-0.01 + curvedDuration, 10); + expect(curvedDuration).not.toBeCloseTo(linearDuration, 5); + }); +}); + +describe('stepEyeGaze — nano', () => { + it('draws a unit-length direction and a linearly (uncurved) mapped timer duration', () => { + const state = baseState({ nano: { goal: { ...ZERO }, current: { ...ZERO }, timer: -0.01 } }); + const rng = scriptedRng([0.75, 0.25, 0.4]); + const result = stepEyeGaze( + state, + { deltaSeconds: 0, avatarState: 'idleActive', lookAtDistanceCm: 80 }, + config, + rng, + { macro: false, micro: false } + ); + + const expectedGoal = normalizeVec2({ x: 0.75 * 2 - 1, y: 0.25 * 2 - 1 }); + expect(result.nano.goal.x).toBeCloseTo(expectedGoal.x, 10); + expect(result.nano.goal.y).toBeCloseTo(expectedGoal.y, 10); + expect(Math.hypot(result.nano.goal.x, result.nano.goal.y)).toBeCloseTo(1, 10); + + const [min, max] = config.nano.timerMinMax; + expect(result.nano.timer).toBeCloseTo(-0.01 + (min + 0.4 * (max - min)), 10); + }); +}); + +describe('stepEyeGaze — composition', () => { + it('combinedOffsetCm is the exact sum of the three layer offsets', () => { + const state = baseState({ + macro: { phase: 'atLookAt', timer: 100, goal: { ...ZERO }, current: { x: 0.2, y: -0.1 } }, + micro: { + x: { goal: 1, current: 0.5, timer: 100 }, + y: { goal: -1, current: -0.3, timer: 100 } + }, + nano: { goal: { x: 0.6, y: 0.8 }, current: { x: 0.6, y: 0.8 }, timer: 100 } + }); + const result = stepEyeGaze( + state, + { deltaSeconds: 0, avatarState: 'idleActive', lookAtDistanceCm: 80 }, + config, + scriptedRng([]) + ); + expect(result.combinedOffsetCm.x).toBeCloseTo( + result.macroOffsetCm.x + result.microOffsetCm.x + result.nanoOffsetCm.x, + 10 + ); + expect(result.combinedOffsetCm.y).toBeCloseTo( + result.macroOffsetCm.y + result.microOffsetCm.y + result.nanoOffsetCm.y, + 10 + ); + }); +}); + +describe('forceMacroLookAway', () => { + it('immediately transitions to lookingAway with a fresh goal, regardless of remaining timer', () => { + const state = baseState({ + macro: { phase: 'atLookAt', timer: 50, goal: { ...ZERO }, current: { ...ZERO } } + }); + const rng = scriptedRng([0.75, 0.25, 0.5]); + const result = forceMacroLookAway(state, config, rng); + + expect(result.macro.phase).toBe('lookingAway'); + expect(result.macro.goal.x).toBeCloseTo(smoothstepSigned(0.5), 10); + expect(result.macro.goal.y).toBeCloseTo( + biasedSmoothstepSigned(-0.5, config.macro.yCurveBias), + 10 + ); + const [min, max] = config.macro.lookAwayLengthMinMax; + expect(result.macro.timer).toBeCloseTo(min + 0.5 * (max - min), 10); + }); + + it('is a no-op while already lookingAway', () => { + const state = baseState({ + macro: { + phase: 'lookingAway', + timer: 3, + goal: { x: 0.4, y: 0.4 }, + current: { x: 0.1, y: 0.1 } + } + }); + const result = forceMacroLookAway(state, config, scriptedRng([])); + expect(result).toBe(state); + }); +}); + +describe('layer toggles', () => { + it('freezes a disabled layer entirely and zeroes its offset contribution', () => { + const state = baseState({ + macro: { + phase: 'atLookAt', + timer: 0.01, + goal: { x: 0.3, y: 0.3 }, + current: { x: 0.2, y: 0.2 } + } + }); + const rng = scriptedRng([]); // the frozen layer's timer looks expired but must not draw + const result = stepEyeGaze( + state, + { deltaSeconds: 1, avatarState: 'idleActive', lookAtDistanceCm: 80 }, + config, + rng, + { + macro: false + } + ); + expect(result.macro).toEqual(state.macro); + expect(result.macroOffsetCm).toEqual(ZERO); + }); + + it('resumes a re-enabled layer from where it left off, not from a reset state', () => { + const frozen = baseState({ + macro: { phase: 'atLookAt', timer: 5, goal: { ...ZERO }, current: { ...ZERO } } + }); + const stillFrozen = stepEyeGaze( + frozen, + { deltaSeconds: 1, avatarState: 'idleActive', lookAtDistanceCm: 80 }, + config, + scriptedRng([]), + { macro: false } + ); + expect(stillFrozen.macro.timer).toBe(5); + + const resumed = stepEyeGaze( + stillFrozen, + { deltaSeconds: 1, avatarState: 'idleActive', lookAtDistanceCm: 80 }, + config, + scriptedRng([]), + { macro: true } + ); + expect(resumed.macro.timer).toBeCloseTo(4, 10); + }); +}); diff --git a/src/lib/humanoid/eye/gaze.ts b/src/lib/humanoid/eye/gaze.ts new file mode 100644 index 0000000..f07e946 --- /dev/null +++ b/src/lib/humanoid/eye/gaze.ts @@ -0,0 +1,358 @@ +import type { AvatarState } from '../state.js'; +import { + biasedSmoothstepSigned, + clamp, + easeOutCubic, + normalizeVec2, + smoothstepSigned, + springTowardScalar, + springTowardVec2, + type Vec2 +} from './curves.js'; +import type { EyeConfig, EyeMacroConfig, EyeMicroConfig, EyeNanoConfig } from './config.js'; + +export interface EyeGazeInput { + deltaSeconds: number; + avatarState: AvatarState; + /** Distance (cm) from the avatar's eyes to wherever the (out of scope) LookAt system is currently aiming. */ + lookAtDistanceCm: number; +} + +/** Each layer defaults to enabled; disabling one freezes it (see `stepEyeGaze`). */ +export type LayerToggles = { macro?: boolean; micro?: boolean; nano?: boolean }; + +interface MacroLayerState { + phase: 'atLookAt' | 'lookingAway'; + /** Seconds remaining in the current phase. */ + timer: number; + goal: Vec2; + current: Vec2; +} + +interface MicroAxisState { + goal: -1 | 0 | 1; + current: number; + timer: number; +} + +interface MicroLayerState { + x: MicroAxisState; + y: MicroAxisState; +} + +interface NanoLayerState { + goal: Vec2; + current: Vec2; + timer: number; +} + +export interface EyeGazeState { + macro: MacroLayerState; + micro: MicroLayerState; + nano: NanoLayerState; + /** Each layer's effective cm-offset this tick — exposed for debug UIs, not just internal use. */ + macroOffsetCm: Vec2; + microOffsetCm: Vec2; + nanoOffsetCm: Vec2; + combinedOffsetCm: Vec2; +} + +const ZERO: Vec2 = { x: 0, y: 0 }; + +function randomSigned(rng: () => number): number { + return rng() * 2 - 1; +} + +function randomRange(min: number, max: number, rng: () => number): number { + return min + rng() * (max - min); +} + +function drawMacroGoal(config: EyeMacroConfig, rng: () => number): Vec2 { + return { + x: smoothstepSigned(randomSigned(rng)), + y: biasedSmoothstepSigned(randomSigned(rng), config.yCurveBias) + }; +} + +/** Micro's goal must differ from its current value on every redraw — pick uniformly among the 2 remaining options. */ +function drawMicroGoal(current: -1 | 0 | 1, rng: () => number): -1 | 0 | 1 { + const options = ([-1, 0, 1] as const).filter((value) => value !== current); + return options[rng() < 0.5 ? 0 : 1]; +} + +function drawMicroTimerDuration(timerMinMax: [number, number], rng: () => number): number { + const t = easeOutCubic(rng()); + return timerMinMax[0] + t * (timerMinMax[1] - timerMinMax[0]); +} + +function macroDistanceScale(config: EyeMacroConfig, lookAtDistanceCm: number): number { + return lookAtDistanceCm / config.distanceScaleReferenceCm + config.distanceScaleFloor; +} + +function macroOffsetFromCurrent( + current: Vec2, + config: EyeMacroConfig, + lookAtDistanceCm: number +): Vec2 { + const scale = macroDistanceScale(config, lookAtDistanceCm); + return { + x: current.x * config.offsetStrengthCm.x * scale, + y: current.y * config.offsetStrengthCm.y * scale + }; +} + +function microOffsetFromCurrent(current: Vec2, config: EyeMicroConfig): Vec2 { + return { + x: current.x * config.offsetStrengthCm.x, + y: current.y * config.offsetStrengthCm.y + }; +} + +function nanoOffsetFromCurrent(current: Vec2, config: EyeNanoConfig): Vec2 { + return { + x: current.x * config.offsetStrengthCm, + y: current.y * config.offsetStrengthCm + }; +} + +/** + * Advances the Macro phase machine by `dtSeconds`. A `while` loop (not + * `if`) handles a `dtSeconds` large enough to span more than one phase + * transition (e.g. a demo's time-scale multiplier cranked way up) — each + * iteration always draws a strictly-positive next duration, so this + * terminates. `current` springs toward the *final* goal over the full + * `dtSeconds` once, rather than re-springing after every intermediate + * transition — an intentional simplification that only matters in that same + * large-dt edge case. + */ +function stepMacro( + layer: MacroLayerState, + dtSeconds: number, + avatarState: AvatarState, + config: EyeConfig, + rng: () => number +): MacroLayerState { + let phase = layer.phase; + let timer = layer.timer - dtSeconds; + let goal = layer.goal; + + while (timer <= 0) { + if (phase === 'atLookAt') { + goal = drawMacroGoal(config.macro, rng); + timer += randomRange( + config.macro.lookAwayLengthMinMax[0], + config.macro.lookAwayLengthMinMax[1], + rng + ); + phase = 'lookingAway'; + } else { + const withinCutoff = + Math.abs(goal.x) <= config.macro.secondaryCutoff && + Math.abs(goal.y) <= config.macro.secondaryCutoff; + const rollsSecondary = rng() <= config.macro.secondaryProbability; + + if (rollsSecondary && withinCutoff) { + const delta = { + x: smoothstepSigned(randomSigned(rng)) * config.macro.secondaryMaxStrength, + y: smoothstepSigned(randomSigned(rng)) * config.macro.secondaryMaxStrength + }; + goal = { + x: clamp(goal.x + delta.x, -1, 1), + y: clamp(goal.y + delta.y, -1, 1) + }; + timer += randomRange( + config.macro.lookAwayLengthSecondaryMinMax[0], + config.macro.lookAwayLengthSecondaryMinMax[1], + rng + ); + // stays 'lookingAway' — another secondary check happens when this timer next expires. + } else { + goal = { x: 0, y: 0 }; + const [min, max] = config.macro.lookAtLengthMinMaxByState[avatarState]; + timer += randomRange(min, max, rng); + phase = 'atLookAt'; + } + } + } + + const current = springTowardVec2( + layer.current, + goal, + dtSeconds, + config.springTimeConstantSeconds + ); + return { phase, timer, goal, current }; +} + +function stepMicroAxis( + axis: MicroAxisState, + dtSeconds: number, + timerMinMax: [number, number], + tauSeconds: number, + rng: () => number +): MicroAxisState { + let goal = axis.goal; + let timer = axis.timer - dtSeconds; + + while (timer <= 0) { + goal = drawMicroGoal(goal, rng); + timer += drawMicroTimerDuration(timerMinMax, rng); + } + + const current = springTowardScalar(axis.current, goal, dtSeconds, tauSeconds); + return { goal, timer, current }; +} + +function stepNano( + layer: NanoLayerState, + dtSeconds: number, + config: EyeConfig, + rng: () => number +): NanoLayerState { + let goal = layer.goal; + let timer = layer.timer - dtSeconds; + + while (timer <= 0) { + goal = normalizeVec2({ x: randomSigned(rng), y: randomSigned(rng) }); + timer += randomRange(config.nano.timerMinMax[0], config.nano.timerMinMax[1], rng); + } + + const current = springTowardVec2( + layer.current, + goal, + dtSeconds, + config.springTimeConstantSeconds + ); + return { goal, timer, current }; +} + +/** + * Initial state per the doc's startup rule ("LookAtMacroOffsetStartup"): + * Macro begins `atLookAt` with `goal`/`current` at rest rather than an + * immediate look-away. Every layer's first timer is still randomly drawn + * (from the same distributions a normal redraw would use) so the three + * layers don't all expire on the same tick. Macro's very first look-at + * timer is seeded using the `idleActive` range — the `AvatarState` store's + * own documented starting value (`avatar-state.ts`) — since no live + * `AvatarState` is available yet at construction time; this only affects + * the first countdown; every countdown after that reads the live state + * passed into `stepEyeGaze`, per that state's own range. + */ +export function createInitialEyeGazeState(config: EyeConfig, rng: () => number): EyeGazeState { + const [macroMin, macroMax] = config.macro.lookAtLengthMinMaxByState.idleActive; + + return { + macro: { + phase: 'atLookAt', + timer: randomRange(macroMin, macroMax, rng), + goal: { x: 0, y: 0 }, + current: { x: 0, y: 0 } + }, + micro: { + x: { goal: 0, current: 0, timer: drawMicroTimerDuration(config.micro.xTimerMinMax, rng) }, + y: { goal: 0, current: 0, timer: drawMicroTimerDuration(config.micro.yTimerMinMax, rng) } + }, + nano: { + goal: { x: 0, y: 0 }, + current: { x: 0, y: 0 }, + timer: randomRange(config.nano.timerMinMax[0], config.nano.timerMinMax[1], rng) + }, + macroOffsetCm: ZERO, + microOffsetCm: ZERO, + nanoOffsetCm: ZERO, + combinedOffsetCm: ZERO + }; +} + +/** + * Advances all enabled layers by `input.deltaSeconds` and recomputes their + * effective cm-offsets. A disabled layer (`toggles. === false`) + * *freezes* — its timer/goal/current stop advancing entirely, rather than + * continuing to tick while its offset is silently zeroed — so re-enabling + * it resumes exactly where it left off instead of jumping. Every random + * draw anywhere in this module goes through `rng`; nothing here calls + * `Math.random()` directly, which is what makes this whole stepper + * deterministically testable (`Math.random` is only ever supplied at the + * `client/humanoid/eye/gaze-controller.ts` boundary). + */ +export function stepEyeGaze( + state: EyeGazeState, + input: EyeGazeInput, + config: EyeConfig, + rng: () => number, + toggles: LayerToggles = {} +): EyeGazeState { + const macroEnabled = toggles.macro ?? true; + const microEnabled = toggles.micro ?? true; + const nanoEnabled = toggles.nano ?? true; + + const macro = macroEnabled + ? stepMacro(state.macro, input.deltaSeconds, input.avatarState, config, rng) + : state.macro; + const micro = microEnabled + ? { + x: stepMicroAxis( + state.micro.x, + input.deltaSeconds, + config.micro.xTimerMinMax, + config.springTimeConstantSeconds, + rng + ), + y: stepMicroAxis( + state.micro.y, + input.deltaSeconds, + config.micro.yTimerMinMax, + config.springTimeConstantSeconds, + rng + ) + } + : state.micro; + const nano = nanoEnabled ? stepNano(state.nano, input.deltaSeconds, config, rng) : state.nano; + + const macroOffsetCm = macroEnabled + ? macroOffsetFromCurrent(macro.current, config.macro, input.lookAtDistanceCm) + : ZERO; + const microOffsetCm = microEnabled + ? microOffsetFromCurrent({ x: micro.x.current, y: micro.y.current }, config.micro) + : ZERO; + const nanoOffsetCm = nanoEnabled ? nanoOffsetFromCurrent(nano.current, config.nano) : ZERO; + + return { + macro, + micro, + nano, + macroOffsetCm, + microOffsetCm, + nanoOffsetCm, + combinedOffsetCm: { + x: macroOffsetCm.x + microOffsetCm.x + nanoOffsetCm.x, + y: macroOffsetCm.y + microOffsetCm.y + nanoOffsetCm.y + } + }; +} + +/** + * Demo/debug helper: immediately starts a macro look-away (fresh goal, full + * look-away timer), regardless of how much time is left in the current + * `atLookAt` countdown. A no-op while already `lookingAway` — chaining a + * secondary on demand isn't exposed, only triggering the first look-away. + */ +export function forceMacroLookAway( + state: EyeGazeState, + config: EyeConfig, + rng: () => number +): EyeGazeState { + if (state.macro.phase === 'lookingAway') return state; + + const goal = drawMacroGoal(config.macro, rng); + const timer = randomRange( + config.macro.lookAwayLengthMinMax[0], + config.macro.lookAwayLengthMinMax[1], + rng + ); + + return { + ...state, + macro: { phase: 'lookingAway', timer, goal, current: state.macro.current } + }; +} diff --git a/src/lib/humanoid/eye/index.ts b/src/lib/humanoid/eye/index.ts new file mode 100644 index 0000000..8ae28f0 --- /dev/null +++ b/src/lib/humanoid/eye/index.ts @@ -0,0 +1,4 @@ +export * from './config.js'; +export * from './curves.js'; +export * from './gaze.js'; +export * from './blend-shapes.js'; diff --git a/src/routes/demo/humanoid/+page.svelte b/src/routes/demo/humanoid/+page.svelte index a69516b..b56a87a 100644 --- a/src/routes/demo/humanoid/+page.svelte +++ b/src/routes/demo/humanoid/+page.svelte @@ -1,107 +1,17 @@ - function toggleListening() { - listening = !listening; - stateStore?.setListening(listening); - } +

Humanoid animation playground

+

A testing/debugging surface for the humanoid package's animation libraries.

- function disconnect() { - client?.send({ type: 'session.end' }); - client?.close(); - client = null; - stopStateSub?.(); - stopStateSub = null; - stateStore?.destroy(); - stateStore = null; - listening = false; - status = 'closed'; - } - +
+ +
-

Humanoid avatar-state demo

-

- Exercises $lib/humanoid's state derivation without a live API key: run - REALTIME_PROVIDER=mock npm run dev (dev-server restart required — see AGENTS.md's HMR - gotcha) and connect below to watch avatarState move through the mock provider's default - fixture. -

-

Status: {status}{statusMessage ? ` — ${statusMessage}` : ''}

-

Avatar state: {avatarState}

+
-{#if status === 'idle' || status === 'error' || status === 'closed'} - -{:else if status === 'ready'} - - -{/if} +
+ +
diff --git a/src/routes/demo/humanoid/AvatarStateDemo.svelte b/src/routes/demo/humanoid/AvatarStateDemo.svelte new file mode 100644 index 0000000..c0ab3da --- /dev/null +++ b/src/routes/demo/humanoid/AvatarStateDemo.svelte @@ -0,0 +1,107 @@ + + +

Avatar-state demo

+

+ Exercises $lib/humanoid's state derivation without a live API key: run + REALTIME_PROVIDER=mock npm run dev (dev-server restart required — see AGENTS.md's HMR + gotcha) and connect below to watch avatarState move through the mock provider's default + fixture. +

+

Status: {status}{statusMessage ? ` — ${statusMessage}` : ''}

+

Avatar state: {avatarState}

+ +{#if status === 'idle' || status === 'error' || status === 'closed'} + +{:else if status === 'ready'} + + +{/if} diff --git a/src/routes/demo/humanoid/EyeSystemDemo.svelte b/src/routes/demo/humanoid/EyeSystemDemo.svelte new file mode 100644 index 0000000..4f614dc --- /dev/null +++ b/src/routes/demo/humanoid/EyeSystemDemo.svelte @@ -0,0 +1,121 @@ + + +

Eye system demo

+

+ Exercises $lib/humanoid/eye's Macro/Micro/Nano gaze layers live, against the bundled + ARKit-rigged test avatar — see docs/HUMANOID.md and + src/lib/humanoid/eye/DESIGN_NOTES.md. +

+ +{#if gazeState} +
+ +
+ + +
+
+{/if} + + diff --git a/src/routes/demo/humanoid/eye-demo/GazeDebugPanel.svelte b/src/routes/demo/humanoid/eye-demo/GazeDebugPanel.svelte new file mode 100644 index 0000000..8f85677 --- /dev/null +++ b/src/routes/demo/humanoid/eye-demo/GazeDebugPanel.svelte @@ -0,0 +1,229 @@ + + +
+
+ + + + + + + +
+ +
+ + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Layerx (cm)y (cm)
Macro{fmt(state.macroOffsetCm.x)}{fmt(state.macroOffsetCm.y)}
Micro{fmt(state.microOffsetCm.x)}{fmt(state.microOffsetCm.y)}
Nano{fmt(state.nanoOffsetCm.x)}{fmt(state.nanoOffsetCm.y)}
Combined{fmt(state.combinedOffsetCm.x)}{fmt(state.combinedOffsetCm.y)}
+ +

+ Macro phase: {state.macro.phase} · Gaze angle: {fmt(angleDeg.x)}°, {fmt( + angleDeg.y + )}° +

+ +

Active blend shapes:

+ {#if frameEntries.length === 0} +

none

+ {:else} +
    + {#each frameEntries as [name, weight] (name)} +
  • {name}: {weight.toFixed(3)}
    • + {/each} +
+ {/if} +
+ + diff --git a/src/routes/demo/humanoid/eye-demo/GazePlaneVisualizer.svelte b/src/routes/demo/humanoid/eye-demo/GazePlaneVisualizer.svelte new file mode 100644 index 0000000..bfcb6c2 --- /dev/null +++ b/src/routes/demo/humanoid/eye-demo/GazePlaneVisualizer.svelte @@ -0,0 +1,101 @@ + + + + + + + + + + + + + +
    +
  • Macro
    • +
  • Micro
    • +
  • Nano
    • +
  • Combined
    • +
+ + diff --git a/src/routes/demo/humanoid/eye-demo/ThreeAvatarViewer.svelte b/src/routes/demo/humanoid/eye-demo/ThreeAvatarViewer.svelte new file mode 100644 index 0000000..4d1b5de --- /dev/null +++ b/src/routes/demo/humanoid/eye-demo/ThreeAvatarViewer.svelte @@ -0,0 +1,156 @@ + + +
+ {#if loading} +

Loading avatar…

+ {:else if loadError} +

{loadError}

+ {/if} +
+ + diff --git a/static/models/avatar.glb b/static/models/avatar.glb new file mode 100644 index 0000000..372beff --- /dev/null +++ b/static/models/avatar.glb @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b3b15585a521362aadb3714028a2643f2a4d84554ebafc68e0543823a373fc66 +size 14198100 diff --git a/vite.config.ts b/vite.config.ts index 41adc1a..7f92364 100644 --- a/vite.config.ts +++ b/vite.config.ts @@ -29,6 +29,19 @@ export default defineConfig({ // module scope would run it there as well, with no env vars available. realtimeDevWsPlugin({ persona: SESSION_PERSONA, toolRegistry }) ], + // three/addons/* (GLTFLoader, OrbitControls — see the eye-demo Three.js + // viewer) are deep imports Vite's dependency scanner doesn't always + // discover on its own; without this, dev mode can end up serving two + // separate module instances of `three` (one pre-bundled, one not) and + // warns "Multiple instances of Three.js being imported". Doesn't affect + // the production build, which Rollup already dedupes correctly. + optimizeDeps: { + include: [ + 'three', + 'three/addons/loaders/GLTFLoader.js', + 'three/addons/controls/OrbitControls.js' + ] + }, test: { expect: { requireAssertions: true }, projects: [