Server- and client-side infrastructure other subsystems (humanoid,
analytics, project surfaces) can log into: an open-category Logger
contract, per-session WS forwarding gated by DEBUG_ACCESS_KEY in prod
(frictionless in dev), and a production-shippable DebugPanel.svelte.
Also logs full conversation content (system prompt, user/assistant
messages) under its own `messages` category so a session's history is
trackable while debugging, with DEBUG_LOG_MESSAGES to keep that content
out of the unconditional console/stdout mirror when desired.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
They're core-owned verification scaffolding (the live smoke-test surface
for the realtime session layer and humanoid package), not client-project
UI, so a branched project shouldn't need to remove them to keep them out
of production.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ports an Unreal Engine avatar project's Macro/Micro/Nano additive gaze
design to ARKit blend shapes: pure, seedable state machine in
src/lib/humanoid/eye/, a client wrapper owning the frame loop in
src/lib/client/humanoid/eye/, and a Three.js demo/debug playground at
/demo/humanoid (split into AvatarStateDemo + EyeSystemDemo so future
animation libraries get their own sibling demo). Updates HUMANOID.md
and ARCHITECTURE.md to describe animation libraries as sub-packages
nested under humanoid/<name> rather than separate top-level packages.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
idlePassiveTimeoutMs moves out of a one-off deriveAvatarState options
type into $lib/humanoid/config.ts's AnimationConfig, with
DEFAULT_ANIMATION_CONFIG and resolveAnimationConfig(overrides) merging
partial overrides onto it. Documents the convention in HUMANOID.md so
future knobs ported from the Unreal Engine avatar app's animation
systems land in this same interface instead of scattered options.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ported animation systems (from an Unreal Engine avatar app) key idle
behavior directly off this distinction, so it moves from an unbuilt
extension point to a core state. The pure state.ts reducer stays
timer-free (idleActive -> idlePassive rides in on an explicit
idle_timeout signal); deriveAvatarState in the client wrapper owns the
actual clock via a configurable idlePassiveTimeoutMs (default 30s).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wraps RealtimeClient with reconnect-on-drop (ResilientRealtimeClient), adds
a MockRealtimeProvider so client/animation work doesn't need a live OpenAI
key, and introduces "optional package" as a third module-layering kind
alongside core/project — humanoid is the first instance, providing a
renderer-agnostic avatar-state reducer and ARKit blend-shape animation
compositor that future eye look-at/blink/lip-sync libraries can build on
without forcing that surface onto non-humanoid (e.g. 2D) projects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Centralizes env parsing/validation into a single loadConfig() (config.ts),
replacing scattered process.env reads across server.js and dev-plugin.ts and
making dev fail fast on a missing OPENAI_API_KEY the same way prod already
did. Analytics gains a master on/off switch (a new NoopAnalyticsRecorder,
selected when ANALYTICS_ENABLED=false) plus independent category toggles for
messages/tool calls/events, so a deployment can control what gets logged
without writing a custom recorder. Session persona (model/instructions)
moves out of the core defaults.ts into project-owned persona.ts, mirroring
the existing tools.ts split and fixing a prior inconsistency where a
branched project was expected to hand-edit a core file. Client tool timeout,
token TTL, model override, and default voice are now env-configurable
instead of hardcoded constants.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Records session start/end, conversation messages, tool calls, and
errors via better-sqlite3, hooked into RealtimeSession's existing
choke points so no client-side changes are needed. Also fixes a
pre-existing gap where an upstream-initiated disconnect only forwarded
session.closed to the browser without ever calling close(), so no
session end time was ever recorded for that path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends the realtime session protocol with tool.call/tool.result/response.trigger
so the model can invoke functions that run either on our server or, relayed
through the session, in the browser (e.g. DOM interaction) — a single
outstanding-call counter lets a turn mix both without racing the model's
continuation. createToolRegistry enforces at startup that a client-only tool
can never accidentally be backed by server logic.
Also introduces a project/ namespace alongside core/, server/core/, and
client/core/, for code specific to whichever client project branches from
this template. Core never imports project; the two composition roots
(server.js, vite.config.ts) do the merge instead, so template updates can
still be pulled into a branched project without touching core files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds docs/ARCHITECTURE.md (module layering, foundational infra decisions,
subsystem index) and docs/REALTIME_SESSION.md (the WS relay design, the
provider extensibility seam, the trust boundary, SvelteKit's WS plumbing
workaround, and the OpenAI event-naming gotcha found via live testing).
Restructures AGENTS.md around orientation, conventions, and process, and
establishes documentation as part of the same change as the work itself
rather than a follow-up pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Jonathan Hinkel