Browse Source
Extends the realtime session protocol with tool.call/tool.result/response.trigger so the model can invoke functions that run either on our server or, relayed through the session, in the browser (e.g. DOM interaction) — a single outstanding-call counter lets a turn mix both without racing the model's continuation. createToolRegistry enforces at startup that a client-only tool can never accidentally be backed by server logic. Also introduces a project/ namespace alongside core/, server/core/, and client/core/, for code specific to whichever client project branches from this template. Core never imports project; the two composition roots (server.js, vite.config.ts) do the merge instead, so template updates can still be pulled into a branched project without touching core files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>main
25 changed files with 856 additions and 19 deletions
@ -0,0 +1,51 @@ |
|||
# Tool calling |
|||
|
|||
Lets the model call functions during a realtime session — e.g. a RAG lookup, or a client-side DOM interaction. Builds on the [realtime session layer](REALTIME_SESSION.md); read that first if you haven't. |
|||
|
|||
## Two execution locations, one mechanism |
|||
|
|||
A tool call can be satisfied in two different places: |
|||
|
|||
- **Server tools** run on our server (e.g. a RAG lookup hitting a vector DB, or anything touching real data/secrets). |
|||
- **Client tools** run in the browser (e.g. "highlight this element", "scroll to section") — the server can't do these itself. |
|||
|
|||
From the protocol's point of view both look the same: the provider emits a normalized `{ type: 'tool.call', callId, name, arguments }` `ServerEvent` ([`protocol.ts`](../src/lib/core/protocol.ts)). What differs is what `RealtimeSession` (`session.ts`) does with it: |
|||
|
|||
- A **server** tool is executed in-process — the browser never sees the call at all. |
|||
- A **client** tool is forwarded to the browser as-is; the browser runs its handler and replies with `{ type: 'tool.result', callId, output }`, which `RealtimeSession` relays up to the provider. If the browser never replies (disconnected tab, slow handler), a 10s timeout (`CLIENT_TOOL_TIMEOUT_MS` in `session.ts`) resolves the call with an error so the model's turn never hangs indefinitely. |
|||
|
|||
Either way, the result reaches the provider as the same `tool.result` `ClientEvent` — `session.ts` constructs it directly for server tools, or receives it from the browser for client tools. |
|||
|
|||
## The core/project ownership boundary |
|||
|
|||
This repo is a template meant to be branched per client project. Everything under `core`/`server/core`/`client/core` is template infrastructure; everything under `project`/`server/project`/`client/project` is where a branched project's own code goes: |
|||
|
|||
| Layer | Core (template, gets pulled forward) | Project (branched-client-specific) | |
|||
| ----------- | ------------------------------------------------------------- | ---------------------------------------------------------- | |
|||
| Isomorphic | `src/lib/core/tools.ts` — `ToolDefinition` shape | `src/lib/project/tools.ts` — this project's tool schemas | |
|||
| Server-only | `src/lib/server/core/tools.ts` — registry + validation | `src/lib/server/project/tools/` — this project's executors | |
|||
| Client-only | `src/lib/client/core/tool-dispatcher.ts` — dispatch mechanism | `src/lib/client/project/tools/` — this project's handlers | |
|||
|
|||
**Rule: core never imports from project, only the reverse.** This is what lets a branched project pull template updates without merge conflicts in core files. In practice this means `server/core/defaults.ts` cannot import the project's tool registry directly — the merge happens at the two composition roots that are allowed to see both (`server.js` for production, `vite.config.ts` for dev, which passes the merged defaults into `dev-plugin.ts` as a parameter rather than having that core file import project code itself). |
|||
|
|||
## Adding a tool |
|||
|
|||
1. Add a `ToolDefinition` to `src/lib/project/tools.ts` — `name`, `description`, a JSON-schema `parameters` object, and `location: 'server' | 'client'`. |
|||
2. If `location: 'server'`, implement a `ToolExecutor` (`(args) => Promise<unknown>`) under `src/lib/server/project/tools/` and register it in that directory's `index.ts`. |
|||
3. If `location: 'client'`, implement a `ClientToolHandler` (same signature) under `src/lib/client/project/tools/` and register it in that directory's `index.ts`. Wire it wherever the page constructs its `RealtimeClient`, via `createToolDispatcher(client, CLIENT_TOOL_HANDLERS)` (see `/demo/realtime/+page.svelte` for the reference wiring). |
|||
|
|||
`createToolRegistry` (`server/core/tools.ts`) validates the result at startup and throws if a server tool has no executor, a client tool _does_ have one registered (see next section), or an executor has no matching definition — a misconfigured tool fails loudly at boot, not silently the first time the model happens to call it. |
|||
|
|||
## Why the client/server split is enforced, not just documented |
|||
|
|||
A tool's `location` decides who executes it, and that decision has a trust dimension: a browser is less trusted than our server, so a tool that touches real user data or secrets must never be satisfiable by browser code — even by accident, e.g. someone tags a tool `location: 'client'` but a server executor also happens to get registered for the same name during a refactor. `createToolRegistry` makes that specific mistake impossible to ship: registering a server executor against a `location: 'client'` definition throws at construction time. There's no equivalent runtime check on the client side (a compromised or modified browser could technically claim to handle any tool), which is exactly why this must be enforced server-side, at the trust boundary — a client tool being `client`-routed is a guarantee about what our server will _accept as satisfying that tool_, not a claim about what the browser is capable of. |
|||
|
|||
## Parallel tool calls and `response.trigger` |
|||
|
|||
OpenAI's Realtime API can ask for multiple function calls in a single turn — e.g. one server tool and one client tool together. If every `tool.result` triggered the model to continue immediately (the way `input_text.send` already does for a plain user message), the first call to resolve would prompt a response before the others' results even exist. |
|||
|
|||
Instead, `tool.result` never triggers continuation on its own. `RealtimeSession` tracks a per-session `outstandingToolCalls` counter, incremented the moment a call is routed and decremented when it resolves; only when it reaches zero does the orchestrator send an explicit `{ type: 'response.trigger' }` `ClientEvent`, which the OpenAI adapter maps to a bare `response.create`. This is a single counter per session, not scoped to an individual OpenAI response id — a deliberate simplification that holds under the normal case (OpenAI doesn't start a new response with more tool calls before outstanding ones are resolved), called out here as the thing most worth re-checking if parallel tool calls ever misbehave. |
|||
|
|||
## What's unverified |
|||
|
|||
The mapping from OpenAI's wire event for a completed function call (`response.output_item.done` with `item.type === 'function_call'`, in `providers/openai.ts`'s `fromOpenAiEvent`) is based on a docs fetch, not confirmed live traffic — this codebase has been burned by doc/reality drift on this exact API before (see the `output_` prefix gotcha in [REALTIME_SESSION.md](REALTIME_SESSION.md)), and there's no prior confirmed observation of `output_item.*` events in this codebase at all. If a tool call doesn't seem to reach `session.ts`, add a temporary `console.debug` in `fromOpenAiEvent`'s unmapped-event fallback, restart the dev server (this module isn't part of Vite's HMR graph), and read the real event name/shape from `preview_logs` rather than re-guessing from docs — same playbook as the previous gotcha. |
|||
@ -0,0 +1,85 @@ |
|||
import { describe, expect, it, vi } from 'vitest'; |
|||
import type { ServerEvent } from '../../core/protocol.js'; |
|||
import type { RealtimeClient } from './client.js'; |
|||
import { createToolDispatcher } from './tool-dispatcher.js'; |
|||
|
|||
/** Minimal stand-in for RealtimeClient — only what createToolDispatcher touches. */ |
|||
class FakeClient { |
|||
private handler: ((event: ServerEvent) => void) | null = null; |
|||
sent: unknown[] = []; |
|||
|
|||
on(handler: (event: ServerEvent) => void): () => void { |
|||
this.handler = handler; |
|||
return () => { |
|||
this.handler = null; |
|||
}; |
|||
} |
|||
|
|||
send(event: unknown): void { |
|||
this.sent.push(event); |
|||
} |
|||
|
|||
emit(event: ServerEvent): void { |
|||
this.handler?.(event); |
|||
} |
|||
} |
|||
|
|||
function flush() { |
|||
return new Promise((resolve) => setTimeout(resolve, 0)); |
|||
} |
|||
|
|||
describe('createToolDispatcher', () => { |
|||
it('dispatches to the matching handler and sends back its result', async () => { |
|||
const client = new FakeClient(); |
|||
const handler = vi.fn().mockResolvedValue({ ok: true }); |
|||
createToolDispatcher(client as unknown as RealtimeClient, { my_tool: handler }); |
|||
|
|||
client.emit({ type: 'tool.call', callId: 'call-1', name: 'my_tool', arguments: '{"x":1}' }); |
|||
await flush(); |
|||
|
|||
expect(handler).toHaveBeenCalledWith({ x: 1 }); |
|||
expect(client.sent).toContainEqual({ |
|||
type: 'tool.result', |
|||
callId: 'call-1', |
|||
output: JSON.stringify({ ok: true }) |
|||
}); |
|||
}); |
|||
|
|||
it('sends an error result when no handler matches', async () => { |
|||
const client = new FakeClient(); |
|||
createToolDispatcher(client as unknown as RealtimeClient, {}); |
|||
|
|||
client.emit({ type: 'tool.call', callId: 'call-1', name: 'missing_tool', arguments: '{}' }); |
|||
await flush(); |
|||
|
|||
expect(client.sent).toContainEqual({ |
|||
type: 'tool.result', |
|||
callId: 'call-1', |
|||
output: JSON.stringify({ error: 'No client handler for tool "missing_tool".' }) |
|||
}); |
|||
}); |
|||
|
|||
it('sends an error result when the handler rejects', async () => { |
|||
const client = new FakeClient(); |
|||
const handler = vi.fn().mockRejectedValue(new Error('boom')); |
|||
createToolDispatcher(client as unknown as RealtimeClient, { my_tool: handler }); |
|||
|
|||
client.emit({ type: 'tool.call', callId: 'call-1', name: 'my_tool', arguments: '{}' }); |
|||
await flush(); |
|||
|
|||
expect(client.sent).toContainEqual({ |
|||
type: 'tool.result', |
|||
callId: 'call-1', |
|||
output: JSON.stringify({ error: 'boom' }) |
|||
}); |
|||
}); |
|||
|
|||
it('ignores non-tool.call server events', () => { |
|||
const client = new FakeClient(); |
|||
createToolDispatcher(client as unknown as RealtimeClient, {}); |
|||
|
|||
client.emit({ type: 'response.done' }); |
|||
|
|||
expect(client.sent).toEqual([]); |
|||
}); |
|||
}); |
|||
@ -0,0 +1,50 @@ |
|||
import { formatToolError } from '../../core/tools.js'; |
|||
import type { RealtimeClient } from './client.js'; |
|||
|
|||
export type ClientToolHandler = (args: Record<string, unknown>) => Promise<unknown>; |
|||
|
|||
/** |
|||
* Wires a `RealtimeClient` to a map of client-side tool handlers: listens |
|||
* for `tool.call`, runs the matching handler, and sends its result back — |
|||
* kept out of `RealtimeClient` itself, which stays transport-only. Handler |
|||
* implementations are project-owned (`$lib/client/project/tools`); this is |
|||
* just the generic dispatch mechanism. |
|||
* |
|||
* Returns an unsubscribe function, matching `RealtimeClient.on()`'s convention. |
|||
*/ |
|||
export function createToolDispatcher( |
|||
client: RealtimeClient, |
|||
handlers: Record<string, ClientToolHandler> |
|||
): () => void { |
|||
return client.on((event) => { |
|||
if (event.type !== 'tool.call') return; |
|||
void dispatch(client, handlers, event); |
|||
}); |
|||
} |
|||
|
|||
async function dispatch( |
|||
client: RealtimeClient, |
|||
handlers: Record<string, ClientToolHandler>, |
|||
event: { callId: string; name: string; arguments: string } |
|||
): Promise<void> { |
|||
const handler = handlers[event.name]; |
|||
if (!handler) { |
|||
client.send({ |
|||
type: 'tool.result', |
|||
callId: event.callId, |
|||
output: formatToolError(`No client handler for tool "${event.name}".`) |
|||
}); |
|||
return; |
|||
} |
|||
try { |
|||
const args = JSON.parse(event.arguments); |
|||
const result = await handler(args); |
|||
client.send({ type: 'tool.result', callId: event.callId, output: JSON.stringify(result) }); |
|||
} catch (err) { |
|||
client.send({ |
|||
type: 'tool.result', |
|||
callId: event.callId, |
|||
output: formatToolError(err instanceof Error ? err.message : 'Tool execution failed.') |
|||
}); |
|||
} |
|||
} |
|||
@ -0,0 +1,12 @@ |
|||
import type { ClientToolHandler } from '../../core/tool-dispatcher.js'; |
|||
|
|||
/** Briefly flashes the page background — proves the client tool relay round-trip visibly. */ |
|||
export const flashPage: ClientToolHandler = async () => { |
|||
const original = document.body.style.transition; |
|||
document.body.style.transition = 'background-color 150ms ease'; |
|||
document.body.style.backgroundColor = '#fef08a'; |
|||
await new Promise((resolve) => setTimeout(resolve, 300)); |
|||
document.body.style.backgroundColor = ''; |
|||
document.body.style.transition = original; |
|||
return { flashed: true }; |
|||
}; |
|||
@ -0,0 +1,7 @@ |
|||
import type { ClientToolHandler } from '../../core/tool-dispatcher.js'; |
|||
import { flashPage } from './flash-page.js'; |
|||
|
|||
/** This project's client-side tool handlers, keyed by tool name. */ |
|||
export const CLIENT_TOOL_HANDLERS: Record<string, ClientToolHandler> = { |
|||
flash_page: flashPage |
|||
}; |
|||
@ -0,0 +1,20 @@ |
|||
/** |
|||
* Isomorphic tool-calling types shared by server and client. A project |
|||
* defines its own tools under `$lib/project/tools.ts` conforming to |
|||
* `ToolDefinition` — see `$lib/server/core/tools.ts` for how `location` |
|||
* routes execution. |
|||
*/ |
|||
|
|||
export interface ToolDefinition { |
|||
name: string; |
|||
description: string; |
|||
/** JSON schema for the tool's arguments, passed through to the provider as-is. */ |
|||
parameters: Record<string, unknown>; |
|||
/** Where this tool executes — see `$lib/server/core/tools.ts`'s `createToolRegistry`. */ |
|||
location: 'server' | 'client'; |
|||
} |
|||
|
|||
/** Shared error-output shape so server executors and client handlers produce identical results. */ |
|||
export function formatToolError(message: string): string { |
|||
return JSON.stringify({ error: message }); |
|||
} |
|||
@ -0,0 +1,28 @@ |
|||
import type { ToolDefinition } from '../core/tools.js'; |
|||
|
|||
/** |
|||
* Project-owned tool definitions — isomorphic (needed by the server at |
|||
* session-start to declare tools to the provider, but the schema itself is |
|||
* neither server- nor client-only). A branched client project edits this |
|||
* file freely; core never imports from it (see docs/TOOLS.md). |
|||
* |
|||
* The two examples below exercise both execution paths end-to-end and are |
|||
* meant to be replaced by real tools (e.g. a RAG lookup) — see |
|||
* src/lib/server/project/tools/ for the `get_current_time` executor and |
|||
* src/lib/client/project/tools/ for the `flash_page` handler. |
|||
*/ |
|||
export const TOOL_DEFINITIONS: ToolDefinition[] = [ |
|||
{ |
|||
name: 'get_current_time', |
|||
description: 'Get the current server time. Use this if the user asks what time it is.', |
|||
parameters: { type: 'object', properties: {}, required: [] }, |
|||
location: 'server' |
|||
}, |
|||
{ |
|||
name: 'flash_page', |
|||
description: |
|||
"Briefly flash the page background to get the user's visual attention. Use this when the user asks you to demonstrate a UI action.", |
|||
parameters: { type: 'object', properties: {}, required: [] }, |
|||
location: 'client' |
|||
} |
|||
]; |
|||
@ -0,0 +1,53 @@ |
|||
import { describe, expect, it } from 'vitest'; |
|||
import type { ToolDefinition } from '../../core/tools.js'; |
|||
import { createToolRegistry } from './tools.js'; |
|||
|
|||
const SERVER_TOOL: ToolDefinition = { |
|||
name: 'server_tool', |
|||
description: 'A server-executed tool.', |
|||
parameters: { type: 'object', properties: {}, required: [] }, |
|||
location: 'server' |
|||
}; |
|||
|
|||
const CLIENT_TOOL: ToolDefinition = { |
|||
name: 'client_tool', |
|||
description: 'A client-executed tool.', |
|||
parameters: { type: 'object', properties: {}, required: [] }, |
|||
location: 'client' |
|||
}; |
|||
|
|||
describe('createToolRegistry', () => { |
|||
it('builds a registry when every server tool has an executor and no client tool does', () => { |
|||
const registry = createToolRegistry([SERVER_TOOL, CLIENT_TOOL], { |
|||
server_tool: async () => ({ ok: true }) |
|||
}); |
|||
expect(registry.definitions).toEqual([SERVER_TOOL, CLIENT_TOOL]); |
|||
expect(registry.executors.has('server_tool')).toBe(true); |
|||
expect(registry.executors.has('client_tool')).toBe(false); |
|||
}); |
|||
|
|||
it('is fine with no tools at all', () => { |
|||
const registry = createToolRegistry([], {}); |
|||
expect(registry.definitions).toEqual([]); |
|||
expect(registry.executors.size).toBe(0); |
|||
}); |
|||
|
|||
it('throws when a server tool has no registered executor', () => { |
|||
expect(() => createToolRegistry([SERVER_TOOL], {})).toThrow(/server_tool/); |
|||
}); |
|||
|
|||
it('throws when a client-only tool has a server executor registered', () => { |
|||
expect(() => createToolRegistry([CLIENT_TOOL], { client_tool: async () => ({}) })).toThrow( |
|||
/client_tool/ |
|||
); |
|||
}); |
|||
|
|||
it('throws when an executor is registered with no matching tool definition', () => { |
|||
expect(() => |
|||
createToolRegistry([SERVER_TOOL], { |
|||
server_tool: async () => ({}), |
|||
orphan_tool: async () => ({}) |
|||
}) |
|||
).toThrow(/orphan_tool/); |
|||
}); |
|||
}); |
|||
@ -0,0 +1,51 @@ |
|||
import type { ToolDefinition } from '../../core/tools.js'; |
|||
|
|||
export type ToolExecutor = (args: Record<string, unknown>) => Promise<unknown>; |
|||
|
|||
export interface ToolRegistry { |
|||
definitions: ToolDefinition[]; |
|||
executors: Map<string, ToolExecutor>; |
|||
} |
|||
|
|||
/** |
|||
* The policy guardrail for tool ownership: throws at construction (startup, |
|||
* fail loud) rather than letting a misconfigured tool ship silently. |
|||
* |
|||
* - Every `location: 'server'` definition must have a matching executor — |
|||
* otherwise a server tool call would have nothing to run. |
|||
* - No `location: 'client'` definition may have a matching executor |
|||
* registered — a tool tagged client-only can never accidentally also be |
|||
* backed by server business logic. If it needs server logic, it isn't a |
|||
* client tool. |
|||
* - Every executor must have a matching definition — no orphaned executors. |
|||
*/ |
|||
export function createToolRegistry( |
|||
definitions: ToolDefinition[], |
|||
executors: Record<string, ToolExecutor> |
|||
): ToolRegistry { |
|||
const executorMap = new Map(Object.entries(executors)); |
|||
const definitionNames = new Set(definitions.map((d) => d.name)); |
|||
|
|||
for (const definition of definitions) { |
|||
const hasExecutor = executorMap.has(definition.name); |
|||
if (definition.location === 'server' && !hasExecutor) { |
|||
throw new Error( |
|||
`Tool "${definition.name}" is location: 'server' but has no registered executor.` |
|||
); |
|||
} |
|||
if (definition.location === 'client' && hasExecutor) { |
|||
throw new Error( |
|||
`Tool "${definition.name}" is location: 'client' but has a server executor registered. ` + |
|||
`A client-only tool must never be backed by server logic — if it needs server logic, ` + |
|||
`it isn't a client tool.` |
|||
); |
|||
} |
|||
} |
|||
for (const name of executorMap.keys()) { |
|||
if (!definitionNames.has(name)) { |
|||
throw new Error(`Executor registered for "${name}" has no matching tool definition.`); |
|||
} |
|||
} |
|||
|
|||
return { definitions, executors: executorMap }; |
|||
} |
|||
@ -0,0 +1,5 @@ |
|||
import type { ToolExecutor } from '../../core/tools.js'; |
|||
|
|||
export const getCurrentTime: ToolExecutor = async () => { |
|||
return { iso: new Date().toISOString() }; |
|||
}; |
|||
@ -0,0 +1,13 @@ |
|||
import { createToolRegistry } from '../../core/tools.js'; |
|||
import { TOOL_DEFINITIONS } from '../../../project/tools.js'; |
|||
import { getCurrentTime } from './get-current-time.js'; |
|||
|
|||
/** |
|||
* Assembles this project's server-side tool executors against the shared |
|||
* definitions and validates the result — see `createToolRegistry` for the |
|||
* policy this enforces (every server tool has an executor, no client tool |
|||
* has one). Add a new server executor here as it's implemented. |
|||
*/ |
|||
export const toolRegistry = createToolRegistry(TOOL_DEFINITIONS, { |
|||
get_current_time: getCurrentTime |
|||
}); |
|||
Loading…
Reference in new issue