Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 15 additions & 7 deletions agents/deploy/protocol.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: "Wire Protocol"
description: "The realtime message contract between clients and agent sessions, for platforms the SDKs don't cover"

Check warning on line 3 in agents/deploy/protocol.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/protocol.mdx#L3

Did you really mean 'realtime'?

Check warning on line 3 in agents/deploy/protocol.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/protocol.mdx#L3

Did you really mean 'SDKs'?
icon: "network-wired"
---

Everything the SDKs do rides a small, versioned wire protocol: two JSON message channels, plus the transport's standard transcription and state mechanisms. The transport is WebRTC, built on LiveKit: audio travels as WebRTC [media tracks](https://docs.livekit.io/home/client/tracks/), the message channels ride LiveKit's reliable [data channels](https://docs.livekit.io/home/client/data/messages/), and any platform with a [LiveKit client SDK](https://docs.livekit.io/home/client/connect/) can implement it. This page documents that contract for consumers that cannot use the SDKs: custom native stacks or ports to new platforms.

Check warning on line 7 in agents/deploy/protocol.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/protocol.mdx#L7

Did you really mean 'SDKs'?

Check warning on line 7 in agents/deploy/protocol.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/protocol.mdx#L7

Did you really mean 'SDKs'?

<Note>
This is an escape hatch. For web and React apps, use the [Web
Expand Down Expand Up @@ -55,13 +55,13 @@

Once connected, a session uses these channels. The two `*-event` topics carry reliable [data packets](https://docs.livekit.io/home/client/data/messages/): one complete JSON object per packet.

| Channel | Direction | Carries |
| -------------------------------------- | -------------- | --------------------------------------------------------- |
| Audio tracks | both | Your microphone up, the agent's speech down |
| `agent-event` topic | agent → client | Control events: client tool calls, tool lifecycle, errors |
| `client-event` topic | client → agent | Text turns, activity, interrupts, hangup, tool results |
| `lk.transcription` text streams | agent → client | Streaming transcripts for both sides |
| `lk.agent.state` participant attribute | agent → client | Agent pipeline state (sticky) |
| Channel | Direction | Carries |
| -------------------------------------- | -------------- | ---------------------------------------------------------------------- |
| Audio tracks | both | Your microphone up, the agent's speech down |
| `agent-event` topic | agent → client | Control events: client tool calls, tool lifecycle, errors, session end |
| `client-event` topic | client → agent | Text turns, activity, interrupts, hangup, tool results |
| `lk.transcription` text streams | agent → client | Streaming transcripts for both sides |
| `lk.agent.state` participant attribute | agent → client | Agent pipeline state (sticky) |

Data-channel message fields are camelCase; REST bodies are snake_case.

Expand Down Expand Up @@ -166,6 +166,14 @@

`code` is a coarse category only: `provider_error` (an upstream model or voice provider failed) or `internal_error` (the runtime failed). The message deliberately carries no raw error detail.

### `session.ended`

```json session.ended
{ "type": "session.ended", "reason": "conversation_timeout" }
```

Sent once, right before the server ends the call (before the agent leaves the room or the room is deleted), so a client learns the cause instead of inferring it from the disconnect. `reason` is `user_hangup` (the user hung up, or a backend ended the session through the API), `agent_hangup` (the agent hung up or the workflow reached an end node), `conversation_timeout` (the session reached its maximum duration), or `escalated` (the call was transferred to a human). The values match `end_reason` on the [session record](/agents/monitor/conversation-history#status-and-end-reason); treat the set as open, per the [compatibility rules](#compatibility-rules). A call that drops without this message has no announced reason: the SDKs report `connection_lost` once reconnection fails.

Check warning on line 175 in agents/deploy/protocol.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/protocol.mdx#L175

Did you really mean 'SDKs'?
Comment thread
coderabbitai[bot] marked this conversation as resolved.

## Client events: `client-event`

Publish these on the `client-event` topic. The agent ignores malformed JSON and unknown types.
Expand Down Expand Up @@ -202,7 +210,7 @@
- Agent segments stream incrementally, paced to audio playback. An interrupted segment closes containing only the words actually spoken; there is no residual text.
- User segments are interim until final; each interim update **replaces the entire segment text** under the same segment id.

**Agent state** is published as the sticky `lk.agent.state` [participant attribute](https://docs.livekit.io/home/client/state/participant-attributes/) with values `initializing`, `idle`, `listening`, `thinking`, and `speaking`. Sticky means a client that connects late or reconnects reads the current value immediately. The SDKs derive their three public modes from this attribute plus transcript segment open/close.

Check warning on line 213 in agents/deploy/protocol.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/protocol.mdx#L213

Did you really mean 'SDKs'?

## Compatibility rules

Expand Down
14 changes: 9 additions & 5 deletions agents/deploy/web-sdk.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: "Web SDK"
description: "Voice sessions in the browser with @fishaudio/agent-client: options, events, transcripts, text input, audio controls, client tools, and errors"
icon: "js"

Check warning on line 4 in agents/deploy/web-sdk.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/web-sdk.mdx#L4

Did you really mean 'js'?
---

`@fishaudio/agent-client` runs a live voice conversation with your agent from any web page: open the microphone, stream audio both ways, and react to typed events for transcripts, agent state, and tool calls. The SDK handles the realtime transport (WebRTC) internally.

Check warning on line 7 in agents/deploy/web-sdk.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/web-sdk.mdx#L7

Did you really mean 'realtime'?

Using React? [`@fishaudio/agent-react`](/agents/deploy/react-sdk) wraps this SDK in hooks and a provider.

Expand Down Expand Up @@ -55,7 +55,7 @@

</CodeGroup>

`start()` resolves once the realtime connection is up and rejects with a [`FishAgentError`](#errors) if session creation, the microphone permission, or the connection fails. Call it from a user gesture (a click handler): browsers only grant the microphone and audio playback inside one.

Check warning on line 58 in agents/deploy/web-sdk.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/web-sdk.mdx#L58

Did you really mean 'realtime'?

### Options

Expand All @@ -65,7 +65,7 @@
| `serverUrl` | `string` | API base used in `agentId` mode. Default `https://api.fish.audio` |
| `clientTools` | `Record<string, ClientToolHandler>` | Handlers for [client tools](/agents/build/client-tools) declared on the agent |
| `clientToolTimeoutMs` | `number` | Per-handler timeout for client tools, in milliseconds. Default `15000` |
| `overrides` | `SessionOverrides` | Per-session config [overrides](/agents/deploy/authenticated-sessions#overrides) (`agentId` mode only; with `sessionToken`, your backend sends them when creating the session). Keyless sessions accept only `voice_id` and `language`; the prompt-shaping fields are rejected with `400` |

Check warning on line 68 in agents/deploy/web-sdk.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/web-sdk.mdx#L68

Did you really mean 'Keyless'?
| `dynamicVariables` | `Record<string, string \| number \| boolean>` | Values for `{{placeholders}}` in the agent config; see [Dynamic variables](/agents/build/dynamic-variables) (`agentId` mode only) |
| `language` | `SessionLanguage`, the code of one of the [52 supported languages](/agents/build/voice-language#speaking-language) | Shorthand
for `overrides.language` (`agentId` mode only). Omit it to use the agent's configured [speaking language](/agents/build/voice-language#speaking-language); any other value is rejected with `422` |
Expand All @@ -74,7 +74,7 @@
| `worldContext` | `boolean` | Whether the agent knows the current date and time (default `true`; `agentId` mode only, with `sessionToken` your backend sets `world_context`) |
| `endUserId` | `string` | Your identifier for the end user, stored on the session and echoed in [webhooks](/agents/monitor/webhooks) (`agentId` mode only; with `sessionToken` your backend sets `end_user_id`) |
| `metadata` | `object` | Your own key-values, stored on the session record and returned verbatim (`agentId` mode only; with `sessionToken` your backend sets `metadata`) |
| `microphone` | `boolean` | Capture the microphone on start (default `true`). `false` joins muted with no permission prompt, for text-first UIs; the first `setMicMuted(false)` captures it, so call that from a user gesture |

Check warning on line 77 in agents/deploy/web-sdk.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/web-sdk.mdx#L77

Did you really mean 'UIs'?
| `audio` | `{ inputDeviceId?, outputDeviceId? }` | Pick specific microphone and output devices. `start()` rejects with `device_change_failed` if the output device cannot be selected |
| `wakeLock` | `boolean` | Hold a screen wake lock while the session is live, so long calls survive the phone trying to sleep (default `true`). A denied or unsupported wake lock is silent |
| `callbacks` | `Partial<AgentSessionCallbacks>` | Shorthand for `.on()`: each `onXxx` key subscribes the `xxx` [event](#events), so `onUserTranscript` subscribes `userTranscript`. From 0.2.1, a key that is not a known event name throws a `TypeError` |
Expand All @@ -101,15 +101,19 @@
└───────────(failure)───────────→ ended
```

`start()` resolves once the realtime connection is up; the agent itself joins moments later. If it has not joined within 15 seconds, the session emits a `connection_failed` error and ends with reason `connection_lost` instead of idling on a dead call.

Check warning on line 104 in agents/deploy/web-sdk.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/web-sdk.mdx#L104

Did you really mean 'realtime'?

When the session ends, `disconnect` fires with a reason (also available as `session.endReason`):

| `EndReason` | Meaning |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `user_hangup` | You called `session.end()`, or the user left the page |
| `agent_hangup` | The server ended the session: the agent hung up (for example via the hang-up [system tool](/agents/build/system-tools)), the session hit its maximum duration, or it was force-ended; the protocol does not distinguish these |
| `connection_lost` | The connection dropped and could not be recovered, or the agent never joined |
| `EndReason` | Meaning |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `user_hangup` | You called `session.end()`, the user left the page, or your backend [ended the session](/agents/deploy/authenticated-sessions#ending-sessions-from-your-backend) |
| `agent_hangup` | The agent ended the call: the hang-up [system tool](/agents/build/system-tools), a workflow end node, or an idle hangup |
| `conversation_timeout` | The session reached the agent's [maximum duration](/agents/build/configuration#call-duration) |
| `escalated` | The call was transferred to a human |
| `connection_lost` | The connection dropped and could not be recovered, the agent never joined, or the call ended before the server could say why |

Every reason except `connection_lost` comes from the server: it sends the protocol's [`session.ended` event](/agents/deploy/protocol#session-ended) just before tearing the call down, and the values match `end_reason` on the [session record](/agents/monitor/conversation-history#status-and-end-reason), so your client and your backend agree about how a session ended. This needs `@fishaudio/agent-client` 0.3.0 or later; older clients infer the reason from the disconnect and report every server-side end as `agent_hangup`.

Brief network drops don't end the session: the SDK moves to `reconnecting` and back to `connected` automatically, reusing the same session. It never creates a new session, so you never need a new token mid-call.

Expand Down Expand Up @@ -159,7 +163,7 @@

Transcripts on both sides arrive as **segments**, one segment per utterance or response, identified by `segmentId`:

- **User segments**: interim results **replace the entire segment text** (they never append). Render by upserting on `segmentId`; `final: true` marks the segment as finalized.

Check warning on line 166 in agents/deploy/web-sdk.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/web-sdk.mdx#L166

Did you really mean 'upserting'?
- **Agent segments**: text streams in sync with audio playback: what you display matches what the user has actually heard. If the agent is interrupted, the segment finalizes containing only the words that were spoken. A reply that spans several speech segments (for example around a tool call) arrives as several segments.
- The `message` event delivers only finalized messages from both sides, in order. Use it when you want a simple transcript list without handling interim updates.

Expand Down Expand Up @@ -208,7 +212,7 @@

| Method / property | Description |
| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `setMicMuted(muted)` / `micMuted` | Mute or unmute the microphone. After a `microphone: false` start, the first unmute captures the microphone (permission prompt, so call it from a user gesture) and rejects with `mic_permission_denied` if refused |

Check warning on line 215 in agents/deploy/web-sdk.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/web-sdk.mdx#L215

Did you really mean 'unmute'?

Check warning on line 215 in agents/deploy/web-sdk.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/web-sdk.mdx#L215

Did you really mean 'unmute'?
| `setOutputVolume(v)` | Set playback volume, `0`–`1` |
| `getInputVolume()` / `getOutputVolume()` | Current mic / agent volume, `0`–`1`; poll per frame |
| `getInputFrequencyData()` / `getOutputFrequencyData()` | FFT data as `Uint8Array`, for visualizers |
Expand Down Expand Up @@ -263,7 +267,7 @@
| `unsupported_transport` | The session token uses a transport this SDK version doesn't know; upgrade the SDK |
| `mic_permission_denied` | The user denied microphone access; the SDK ends the session |
| `device_change_failed` | A requested audio device could not be activated, or the browser doesn't support selecting it (output selection is unsupported on some mobile browsers). Thrown from `start()` with `audio.outputDeviceId` set, or from device-switching calls |
| `connection_failed` | The realtime connection could not be established or recovered, a message to the agent could not be sent, the agent never joined (15 s), or the microphone could not be toggled |

Check warning on line 270 in agents/deploy/web-sdk.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

agents/deploy/web-sdk.mdx#L270

Did you really mean 'realtime'?
| `session_expired` | The session token's join deadline passed before connecting |
| `tool_failed` / `tool_timeout` | A client tool handler threw, was not registered, or its result could not be delivered / a handler exceeded `clientToolTimeoutMs` |
| `provider_error` / `internal_error` | The session failed server-side: upstream model/voice provider vs. platform runtime. Emitted on `error`; the session may recover |
Expand Down
Loading