3.8 KiB

Raw Blame History

Eye gaze system — design notes

Rationale that doesn't belong in code comments spread across four files, kept in one place for whoever next has to retune or re-derive this. See docs/HUMANOID.md's "$lib/humanoid/eye" section for the user-facing summary this backs up.

Source material and its limits

The port comes from an Unreal Engine avatar project's control-rig design doc: a "Eye-System Variablen & Logik" section giving exact variable names, ranges, defaults, and branching logic for three additive gaze layers (Macro/Micro/Nano), plus three screenshots of hand-authored curve-editor splines. The variable/logic side is complete and ported faithfully (see gaze.ts's docstrings, which cite the doc's own state-machine description directly). The curves are not: a screenshot shows a spline's shape, not its keyframe positions or tangent handles, so curves.ts's smoothstepSigned/biasedSmoothstepSigned/easeOutCubic are documented best-effort approximations chosen to match the described and visible behavior, not pixel-exact reconstructions. If exact curve data ever surfaces (an exported .curve asset, or numeric keyframe values), these three functions are the only place that needs revisiting — everything else in gaze.ts is agnostic to their exact shape.

The spring/"SpringInterpolate" formula

The doc names a SpringInterpolate operation for every layer's current -> goal smoothing, described only qualitatively ("initially snappy, then increasingly smooth as it nears the goal") plus a ballpark duration ("should take about 0.125 seconds"), no formula and no mention of overshoot or oscillation. That description is exactly frame-rate-independent exponential decay, not a literal damped spring (which can overshoot) — implemented in curves.ts as current += (goal - current) * (1 - exp(-dt / tau)) with tau = 0.125 (the doc's ballpark duration, reinterpreted as a decay time constant rather than a fixed completion time, since exponential decay has no finite completion time).

cm-offset -> ARKit blend shapes

This entire conversion (blend-shapes.ts) has no source-doc precedent. The original system positions a 2D offset on a plane in front of the avatar's face and (per the doc's ControlRig hierarchy — LookAt_ctrl > LookAtMacroScale_ctrl > LookAtMacro_ctrl > ...) drives real eye bones from it. This project's avatars are ARKit-blend-shape-only, so the port instead treats the plane offset as subtending a gaze angle (Math.atan2(offsetCm, lookAtDistanceCm)) and normalizes that angle by a new maxGazeAngleRad config knob (no equivalent in the source doc) to get a [-1, 1] blend-weight input.

Two simplifications worth knowing about:

No per-eye vergence. A real "aim each eye bone at the same 3D point" rig naturally gives each eye a slightly different angle for a near target (interpupillary distance). This port computes one shared angle and mirrors it onto both eyes' In/Out channels — consistent with the doc's own single-shared-2D-offset framing, but not physically exact for very close look-at targets.
lookAtDistanceCm is one value, reused twice. It's the same physical distance the doc uses for Macro's distance/300 + 0.66 scale correction and the distance this port uses for the angle conversion — deliberately, since both describe "how far away is whatever the avatar is looking at," not two separate concepts. It's always a live runtime value (EyeGazeInput.lookAtDistanceCm), never a config default, because the (out-of-scope, not-yet-built) LookAt/body-aim system is what should ultimately own it.

Sign convention

Positive x = the avatar's own right; positive y = up. See the docstring on gazeOffsetToBlendShapes in blend-shapes.ts for the left/right eye In/Out derivation — flagged there and here because it's the one place in this whole port that's easy to get backwards without a live model to check against.