# Tool calling

Lets the model call functions during a realtime session — e.g. a RAG lookup, or a client-side DOM interaction. Builds on the [realtime session layer](REALTIME_SESSION.md); read that first if you haven't.

## Two execution locations, one mechanism

A tool call can be satisfied in two different places:

- **Server tools** run on our server (e.g. a RAG lookup hitting a vector DB, or anything touching real data/secrets).
- **Client tools** run in the browser (e.g. "highlight this element", "scroll to section") — the server can't do these itself.

From the protocol's point of view both look the same: the provider emits a normalized `{ type: 'tool.call', callId, name, arguments }` `ServerEvent` ([`protocol.ts`](../packages/core/src/protocol.ts)). What differs is what `RealtimeSession` (`session.ts`) does with it:

- A **server** tool is executed in-process — the browser never sees the call at all.
- A **client** tool is forwarded to the browser as-is; the browser runs its handler and replies with `{ type: 'tool.result', callId, output }`, which `RealtimeSession` relays up to the provider. If the browser never replies (disconnected tab, slow handler), a 10s timeout (`CLIENT_TOOL_TIMEOUT_MS` in `session.ts`) resolves the call with an error so the model's turn never hangs indefinitely.

Either way, the result reaches the provider as the same `tool.result` `ClientEvent` — `session.ts` constructs it directly for server tools, or receives it from the browser for client tools.

## The core/project ownership boundary

This repo is an npm-workspaces monorepo split between the `@webavatar/webavatar-core` package and the branched client project. Everything in `packages/core` is template infrastructure, published as a package and pulled forward into every client project; everything under `apps/template/src/lib/project`, `apps/template/src/lib/server/project`, `apps/template/src/lib/client/project` is where a branched project's own code goes:

| Layer       | Core (`@webavatar/webavatar-core`, gets pulled forward)                                      | Project (branched-client-specific)                                       |
| ----------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| Isomorphic  | `packages/core/src/tools.ts` — `ToolDefinition` shape, exported from the package root `.`    | `apps/template/src/lib/project/tools.ts` — this project's tool schemas   |
| Server-only | `packages/core/src/server/tools.ts` — registry + validation, exported from `./server`        | `apps/template/src/lib/server/project/tools/` — this project's executors |
| Client-only | `packages/core/src/client/tool-dispatcher.ts` — dispatch mechanism, exported from `./client` | `apps/template/src/lib/client/project/tools/` — this project's handlers  |

**Rule: core never imports from project, only the reverse.** This is what lets a branched project pull package updates without merge conflicts in core files. In practice this means `packages/core/src/server/defaults.ts` cannot import the project's tool registry directly — the merge happens at the composition roots that are allowed to see both, which now share a single `createRealtimeRuntime(options)` factory in `packages/core/src/server/index.ts` (exported from `./server`) rather than each doing its own wiring: `apps/template/server.js` calls it for production, and `apps/template/vite.config.ts` calls it (via the `./vite` plugin) for dev, each passing in the project's merged defaults as an option rather than having core import project code itself.

## Adding a tool

1. Add a `ToolDefinition` to `apps/template/src/lib/project/tools.ts` — `name`, `description`, a JSON-schema `parameters` object, and `location: 'server' | 'client'`.
2. If `location: 'server'`, implement a `ToolExecutor` (`(args) => Promise<unknown>`) under `apps/template/src/lib/server/project/tools/` and register it in that directory's `index.ts`.
3. If `location: 'client'`, implement a `ClientToolHandler` (same signature) under `apps/template/src/lib/client/project/tools/` and register it in that directory's `index.ts`. Wire it wherever the page constructs its `RealtimeClient`, via `createToolDispatcher(client, CLIENT_TOOL_HANDLERS)` (see `/demo/realtime/+page.svelte` for the reference wiring).

`createToolRegistry` (`packages/core/src/server/tools.ts`, exported from `@webavatar/webavatar-core/server`) validates the result at startup and throws if a server tool has no executor, a client tool _does_ have one registered (see next section), or an executor has no matching definition — a misconfigured tool fails loudly at boot, not silently the first time the model happens to call it.

## Why the client/server split is enforced, not just documented

A tool's `location` decides who executes it, and that decision has a trust dimension: a browser is less trusted than our server, so a tool that touches real user data or secrets must never be satisfiable by browser code — even by accident, e.g. someone tags a tool `location: 'client'` but a server executor also happens to get registered for the same name during a refactor. `createToolRegistry` makes that specific mistake impossible to ship: registering a server executor against a `location: 'client'` definition throws at construction time. There's no equivalent runtime check on the client side (a compromised or modified browser could technically claim to handle any tool), which is exactly why this must be enforced server-side, at the trust boundary — a client tool being `client`-routed is a guarantee about what our server will _accept as satisfying that tool_, not a claim about what the browser is capable of.

## Parallel tool calls and `response.trigger`

OpenAI's Realtime API can ask for multiple function calls in a single turn — e.g. one server tool and one client tool together. If every `tool.result` triggered the model to continue immediately (the way `input_text.send` already does for a plain user message), the first call to resolve would prompt a response before the others' results even exist.

Instead, `tool.result` never triggers continuation on its own. `RealtimeSession` tracks a per-session `outstandingToolCalls` counter, incremented the moment a call is routed and decremented when it resolves; only when it reaches zero does the orchestrator send an explicit `{ type: 'response.trigger' }` `ClientEvent`, which the OpenAI adapter maps to a bare `response.create`. This is a single counter per session, not scoped to an individual OpenAI response id — a deliberate simplification that holds under the normal case (OpenAI doesn't start a new response with more tool calls before outstanding ones are resolved), called out here as the thing most worth re-checking if parallel tool calls ever misbehave.

## What's unverified

The mapping from OpenAI's wire event for a completed function call (`response.output_item.done` with `item.type === 'function_call'`, in `providers/openai.ts`'s `fromOpenAiEvent`) is based on a docs fetch, not confirmed live traffic — this codebase has been burned by doc/reality drift on this exact API before (see the `output_` prefix gotcha in [REALTIME_SESSION.md](REALTIME_SESSION.md)), and there's no prior confirmed observation of `output_item.*` events in this codebase at all. If a tool call doesn't seem to reach `session.ts`, add a temporary `console.debug` in `fromOpenAiEvent`'s unmapped-event fallback, restart the dev server as needed (see the "Dev-mode reload behavior" table in [REALTIME_SESSION.md](REALTIME_SESSION.md) for exactly which changes require a restart vs. pick up on HMR), and read the real event name/shape from `preview_logs` rather than re-guessing from docs — same playbook as the previous gotcha.