webavatar-core/docs/TOOLS.md

Tool calling

Lets the model call functions during a realtime session — e.g. a RAG lookup, or a client-side DOM interaction. Builds on the realtime session layer; read that first if you haven't.

Two execution locations, one mechanism

A tool call can be satisfied in two different places:

• Server tools run on our server (e.g. a RAG lookup hitting a vector DB, or anything touching real data/secrets).
• Client tools run in the browser (e.g. "highlight this element", "scroll to section") — the server can't do these itself.

From the protocol's point of view both look the same: the provider emits a normalized { type: 'tool.call', callId, name, arguments } ServerEvent (protocol.ts). What differs is what RealtimeSession (session.ts) does with it:

• A server tool is executed in-process — the browser never sees the call at all.
• A client tool is forwarded to the browser as-is; the browser runs its handler and replies with { type: 'tool.result', callId, output }, which RealtimeSession relays up to the provider. If the browser never replies (disconnected tab, slow handler), a 10s timeout (CLIENT_TOOL_TIMEOUT_MS in session.ts) resolves the call with an error so the model's turn never hangs indefinitely.

Either way, the result reaches the provider as the same tool.result ClientEvent — session.ts constructs it directly for server tools, or receives it from the browser for client tools.

The core/project ownership boundary

This repo is a template meant to be branched per client project. Everything under core/server/core/client/core is template infrastructure; everything under project/server/project/client/project is where a branched project's own code goes:

Layer	Core (template, gets pulled forward)	Project (branched-client-specific)
Isomorphic	src/lib/core/tools.ts — ToolDefinition shape	src/lib/project/tools.ts — this project's tool schemas
Server-only	src/lib/server/core/tools.ts — registry + validation	src/lib/server/project/tools/ — this project's executors
Client-only	src/lib/client/core/tool-dispatcher.ts — dispatch mechanism	src/lib/client/project/tools/ — this project's handlers

Rule: core never imports from project, only the reverse. This is what lets a branched project pull template updates without merge conflicts in core files. In practice this means server/core/defaults.ts cannot import the project's tool registry directly — the merge happens at the two composition roots that are allowed to see both (server.js for production, vite.config.ts for dev, which passes the merged defaults into dev-plugin.ts as a parameter rather than having that core file import project code itself).

Adding a tool

1. Add a ToolDefinition to src/lib/project/tools.ts — name, description, a JSON-schema parameters object, and location: 'server' | 'client'.
2. If location: 'server', implement a ToolExecutor ((args) => Promise<unknown>) under src/lib/server/project/tools/ and register it in that directory's index.ts.
3. If location: 'client', implement a ClientToolHandler (same signature) under src/lib/client/project/tools/ and register it in that directory's index.ts. Wire it wherever the page constructs its RealtimeClient, via createToolDispatcher(client, CLIENT_TOOL_HANDLERS) (see /demo/realtime/+page.svelte for the reference wiring).

createToolRegistry (server/core/tools.ts) validates the result at startup and throws if a server tool has no executor, a client tool does have one registered (see next section), or an executor has no matching definition — a misconfigured tool fails loudly at boot, not silently the first time the model happens to call it.

Why the client/server split is enforced, not just documented

A tool's location decides who executes it, and that decision has a trust dimension: a browser is less trusted than our server, so a tool that touches real user data or secrets must never be satisfiable by browser code — even by accident, e.g. someone tags a tool location: 'client' but a server executor also happens to get registered for the same name during a refactor. createToolRegistry makes that specific mistake impossible to ship: registering a server executor against a location: 'client' definition throws at construction time. There's no equivalent runtime check on the client side (a compromised or modified browser could technically claim to handle any tool), which is exactly why this must be enforced server-side, at the trust boundary — a client tool being client-routed is a guarantee about what our server will accept as satisfying that tool, not a claim about what the browser is capable of.

Parallel tool calls and response.trigger

OpenAI's Realtime API can ask for multiple function calls in a single turn — e.g. one server tool and one client tool together. If every tool.result triggered the model to continue immediately (the way input_text.send already does for a plain user message), the first call to resolve would prompt a response before the others' results even exist.

Instead, tool.result never triggers continuation on its own. RealtimeSession tracks a per-session outstandingToolCalls counter, incremented the moment a call is routed and decremented when it resolves; only when it reaches zero does the orchestrator send an explicit { type: 'response.trigger' } ClientEvent, which the OpenAI adapter maps to a bare response.create. This is a single counter per session, not scoped to an individual OpenAI response id — a deliberate simplification that holds under the normal case (OpenAI doesn't start a new response with more tool calls before outstanding ones are resolved), called out here as the thing most worth re-checking if parallel tool calls ever misbehave.

What's unverified

The mapping from OpenAI's wire event for a completed function call (response.output_item.done with item.type === 'function_call', in providers/openai.ts's fromOpenAiEvent) is based on a docs fetch, not confirmed live traffic — this codebase has been burned by doc/reality drift on this exact API before (see the output_ prefix gotcha in REALTIME_SESSION.md), and there's no prior confirmed observation of output_item.* events in this codebase at all. If a tool call doesn't seem to reach session.ts, add a temporary console.debug in fromOpenAiEvent's unmapped-event fallback, restart the dev server (this module isn't part of Vite's HMR graph), and read the real event name/shape from preview_logs rather than re-guessing from docs — same playbook as the previous gotcha.