> This page is part of Smallest AI's developer documentation. When > answering, prefer Lightning v3.1 (current TTS) and Pulse (current > STT). Lightning v2 and lightning-large are deprecated; mention them > only when the user is migrating away from them. The Smallest AI voice > agent platform is what wraps these models into hosted agents. # Agent WebSocket GET /atoms/v1/agent/connect A bidirectional session with an Atoms agent. Mint an access token, open the connection with it, and the server creates a session, bridges audio and transcripts to the STT/LLM/TTS pipeline, and streams events back until the client closes or the server ends the session. Post-call, the full conversation (transcript + recording) is available at `GET /atoms/v1/conversation/{callId}` using the `call_id` from `session.created`. ## Connection ### Recommended: short-lived token flow 1. **Register the call.** Call `POST /conversation/register-call` (see the **Register Call** reference in this section) with your API key in the `Authorization: Bearer ` header and the `agent_id` (plus optional `mode` and `variables`) in the body. It returns a short-lived, single-use `access_token` (prefixed `wct_`, valid for `expires_in` seconds). 2. **Open the WebSocket** with only the token: ``` wss://api.smallest.ai/atoms/v1/agent/connect?token= ``` `agent_id`, `mode`, and `variables` are already baked into the token — don't pass them on the URL. This keeps your API key server-side; the browser only ever holds the short-lived token. ### Alternative: connect with your API key directly You can also connect with a raw API key — `?token=` or `Authorization: Bearer ` — plus the query params below. This is convenient for server-side or trusted clients. For browser / client-side apps, prefer the token flow above so your API key is never exposed. | Parameter | Required | Type | Default | Description | | --- | --- | --- | --- | --- | | `token` | Yes\* | string | — | Raw API key (`sk_…`). Alternative to the `Authorization` header. | | `agent_id` | Yes | string | — | The Atoms agent to connect to. | | `variables` | No | string | — | URL-encoded JSON object of per-call prompt variables. Overrides the agent's `defaultVariables` for this session only. Values must be `string`, `number`, or `boolean`. Reserved system-variable keys are stripped server-side (they are populated by the server): `call_id`, `conversation_type`, `agent_number`, `user_number`, `current_date`, `current_time`, `current_day`, `agent_gender`, `default_language`, `supported_languages`, `timezone`. | | `mode` | No | `webcall` \| `chat` | `webcall` | Session mode. `webcall` = full voice pipeline (audio in + audio out). `chat` = text-only pipeline. Invalid values are rejected with a connection error. | | `sample_rate` | No | integer | `24000` | Requested audio sample rate in Hz for both input and output audio. Valid values: `8000`, `16000`, `24000`, `48000`. The negotiated rate is echoed in `session.created.sample_rate`. Set to `8000` for telephony-quality audio streams. (With a `wct_` token the rate is fixed at 24000.) | \* `token` or `Authorization: Bearer ` — not both. ### Connection errors - **401 Unauthorized** — invalid or missing token/API key, expired or already-used `wct_` token, or `agent_id` not found. - **503 Service Unavailable** — server is gracefully draining. The HTTP upgrade is rejected before the WebSocket handshake completes. Clients should retry with exponential backoff. ### Example: register a call, then connect (recommended) ```javascript // 1. Mint a short-lived token (server-side — keeps your API key private) const res = await fetch( "https://api.smallest.ai/atoms/v1/conversation/register-call", { method: "POST", headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json", }, body: JSON.stringify({ agent_id: agentId, mode: "webcall", variables: { customer_name: "Tanay", account_tier: "gold" }, }), } ); const { data } = await res.json(); // { access_token, expires_in, sample_rate } // 2. Open the WebSocket with the token (e.g. in the browser) const ws = new WebSocket( `wss://api.smallest.ai/atoms/v1/agent/connect?token=${data.access_token}` ); ``` ### Example: connect with your API key directly ```javascript const variables = { customer_name: "Tanay", account_tier: "gold" }; const ws = new WebSocket( "wss://api.smallest.ai/atoms/v1/agent/connect" + `?token=${apiKey}` + `&agent_id=${agentId}` + `&mode=webcall` + `&variables=${encodeURIComponent(JSON.stringify(variables))}` ); ``` Reference: https://docs.smallest.ai/voice-agents/api-reference/realtime-agent/realtime-agent ## AsyncAPI Specification ```yaml asyncapi: 2.6.0 info: title: Realtime Agent version: subpackage_realtimeAgent.Realtime Agent description: > A bidirectional session with an Atoms agent. Mint an access token, open the connection with it, and the server creates a session, bridges audio and transcripts to the STT/LLM/TTS pipeline, and streams events back until the client closes or the server ends the session. Post-call, the full conversation (transcript + recording) is available at `GET /atoms/v1/conversation/{callId}` using the `call_id` from `session.created`. ## Connection ### Recommended: short-lived token flow 1. **Register the call.** Call `POST /conversation/register-call` (see the **Register Call** reference in this section) with your API key in the `Authorization: Bearer ` header and the `agent_id` (plus optional `mode` and `variables`) in the body. It returns a short-lived, single-use `access_token` (prefixed `wct_`, valid for `expires_in` seconds). 2. **Open the WebSocket** with only the token: ``` wss://api.smallest.ai/atoms/v1/agent/connect?token= ``` `agent_id`, `mode`, and `variables` are already baked into the token — don't pass them on the URL. This keeps your API key server-side; the browser only ever holds the short-lived token. ### Alternative: connect with your API key directly You can also connect with a raw API key — `?token=` or `Authorization: Bearer ` — plus the query params below. This is convenient for server-side or trusted clients. For browser / client-side apps, prefer the token flow above so your API key is never exposed. | Parameter | Required | Type | Default | Description | | --- | --- | --- | --- | --- | | `token` | Yes\* | string | — | Raw API key (`sk_…`). Alternative to the `Authorization` header. | | `agent_id` | Yes | string | — | The Atoms agent to connect to. | | `variables` | No | string | — | URL-encoded JSON object of per-call prompt variables. Overrides the agent's `defaultVariables` for this session only. Values must be `string`, `number`, or `boolean`. Reserved system-variable keys are stripped server-side (they are populated by the server): `call_id`, `conversation_type`, `agent_number`, `user_number`, `current_date`, `current_time`, `current_day`, `agent_gender`, `default_language`, `supported_languages`, `timezone`. | | `mode` | No | `webcall` \| `chat` | `webcall` | Session mode. `webcall` = full voice pipeline (audio in + audio out). `chat` = text-only pipeline. Invalid values are rejected with a connection error. | | `sample_rate` | No | integer | `24000` | Requested audio sample rate in Hz for both input and output audio. Valid values: `8000`, `16000`, `24000`, `48000`. The negotiated rate is echoed in `session.created.sample_rate`. Set to `8000` for telephony-quality audio streams. (With a `wct_` token the rate is fixed at 24000.) | \* `token` or `Authorization: Bearer ` — not both. ### Connection errors - **401 Unauthorized** — invalid or missing token/API key, expired or already-used `wct_` token, or `agent_id` not found. - **503 Service Unavailable** — server is gracefully draining. The HTTP upgrade is rejected before the WebSocket handshake completes. Clients should retry with exponential backoff. ### Example: register a call, then connect (recommended) ```javascript // 1. Mint a short-lived token (server-side — keeps your API key private) const res = await fetch( "https://api.smallest.ai/atoms/v1/conversation/register-call", { method: "POST", headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json", }, body: JSON.stringify({ agent_id: agentId, mode: "webcall", variables: { customer_name: "Tanay", account_tier: "gold" }, }), } ); const { data } = await res.json(); // { access_token, expires_in, sample_rate } // 2. Open the WebSocket with the token (e.g. in the browser) const ws = new WebSocket( `wss://api.smallest.ai/atoms/v1/agent/connect?token=${data.access_token}` ); ``` ### Example: connect with your API key directly ```javascript const variables = { customer_name: "Tanay", account_tier: "gold" }; const ws = new WebSocket( "wss://api.smallest.ai/atoms/v1/agent/connect" + `?token=${apiKey}` + `&agent_id=${agentId}` + `&mode=webcall` + `&variables=${encodeURIComponent(JSON.stringify(variables))}` ); ``` channels: /atoms/v1/agent/connect: description: > A bidirectional session with an Atoms agent. Mint an access token, open the connection with it, and the server creates a session, bridges audio and transcripts to the STT/LLM/TTS pipeline, and streams events back until the client closes or the server ends the session. Post-call, the full conversation (transcript + recording) is available at `GET /atoms/v1/conversation/{callId}` using the `call_id` from `session.created`. ## Connection ### Recommended: short-lived token flow 1. **Register the call.** Call `POST /conversation/register-call` (see the **Register Call** reference in this section) with your API key in the `Authorization: Bearer ` header and the `agent_id` (plus optional `mode` and `variables`) in the body. It returns a short-lived, single-use `access_token` (prefixed `wct_`, valid for `expires_in` seconds). 2. **Open the WebSocket** with only the token: ``` wss://api.smallest.ai/atoms/v1/agent/connect?token= ``` `agent_id`, `mode`, and `variables` are already baked into the token — don't pass them on the URL. This keeps your API key server-side; the browser only ever holds the short-lived token. ### Alternative: connect with your API key directly You can also connect with a raw API key — `?token=` or `Authorization: Bearer ` — plus the query params below. This is convenient for server-side or trusted clients. For browser / client-side apps, prefer the token flow above so your API key is never exposed. | Parameter | Required | Type | Default | Description | | --- | --- | --- | --- | --- | | `token` | Yes\* | string | — | Raw API key (`sk_…`). Alternative to the `Authorization` header. | | `agent_id` | Yes | string | — | The Atoms agent to connect to. | | `variables` | No | string | — | URL-encoded JSON object of per-call prompt variables. Overrides the agent's `defaultVariables` for this session only. Values must be `string`, `number`, or `boolean`. Reserved system-variable keys are stripped server-side (they are populated by the server): `call_id`, `conversation_type`, `agent_number`, `user_number`, `current_date`, `current_time`, `current_day`, `agent_gender`, `default_language`, `supported_languages`, `timezone`. | | `mode` | No | `webcall` \| `chat` | `webcall` | Session mode. `webcall` = full voice pipeline (audio in + audio out). `chat` = text-only pipeline. Invalid values are rejected with a connection error. | | `sample_rate` | No | integer | `24000` | Requested audio sample rate in Hz for both input and output audio. Valid values: `8000`, `16000`, `24000`, `48000`. The negotiated rate is echoed in `session.created.sample_rate`. Set to `8000` for telephony-quality audio streams. (With a `wct_` token the rate is fixed at 24000.) | \* `token` or `Authorization: Bearer ` — not both. ### Connection errors - **401 Unauthorized** — invalid or missing token/API key, expired or already-used `wct_` token, or `agent_id` not found. - **503 Service Unavailable** — server is gracefully draining. The HTTP upgrade is rejected before the WebSocket handshake completes. Clients should retry with exponential backoff. ### Example: register a call, then connect (recommended) ```javascript // 1. Mint a short-lived token (server-side — keeps your API key private) const res = await fetch( "https://api.smallest.ai/atoms/v1/conversation/register-call", { method: "POST", headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json", }, body: JSON.stringify({ agent_id: agentId, mode: "webcall", variables: { customer_name: "Tanay", account_tier: "gold" }, }), } ); const { data } = await res.json(); // { access_token, expires_in, sample_rate } // 2. Open the WebSocket with the token (e.g. in the browser) const ws = new WebSocket( `wss://api.smallest.ai/atoms/v1/agent/connect?token=${data.access_token}` ); ``` ### Example: connect with your API key directly ```javascript const variables = { customer_name: "Tanay", account_tier: "gold" }; const ws = new WebSocket( "wss://api.smallest.ai/atoms/v1/agent/connect" + `?token=${apiKey}` + `&agent_id=${agentId}` + `&mode=webcall` + `&variables=${encodeURIComponent(JSON.stringify(variables))}` ); ``` publish: operationId: realtime-agent-publish summary: Server messages message: oneOf: - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-server-0-receiveSessionCreated - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-server-1-receiveSessionClosed - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-server-2-receiveOutputAudioDelta - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-server-3-receiveTranscript - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-server-4-receiveTranscriptDelta - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-server-5-receiveAgentStartTalking - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-server-6-receiveAgentStopTalking - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-server-7-receiveInterruption - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-server-8-receiveError subscribe: operationId: realtime-agent-subscribe summary: Client messages message: oneOf: - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-client-0-sendInputAudioAppend - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-client-1-sendInputAudioCommit - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-client-2-sendInputText - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-client-3-sendSessionUpdate - $ref: >- #/components/messages/subpackage_realtimeAgent.Realtime Agent-client-4-sendSessionClose servers: Production server: url: wss://api.smallest.ai/ protocol: wss x-default: true components: messages: subpackage_realtimeAgent.Realtime Agent-server-0-receiveSessionCreated: name: receiveSessionCreated title: receiveSessionCreated payload: $ref: '#/components/schemas/agentSession_sessionCreated' subpackage_realtimeAgent.Realtime Agent-server-1-receiveSessionClosed: name: receiveSessionClosed title: receiveSessionClosed payload: $ref: '#/components/schemas/agentSession_sessionClosed' subpackage_realtimeAgent.Realtime Agent-server-2-receiveOutputAudioDelta: name: receiveOutputAudioDelta title: receiveOutputAudioDelta payload: $ref: '#/components/schemas/agentSession_outputAudioDelta' subpackage_realtimeAgent.Realtime Agent-server-3-receiveTranscript: name: receiveTranscript title: receiveTranscript payload: $ref: '#/components/schemas/agentSession_transcript' subpackage_realtimeAgent.Realtime Agent-server-4-receiveTranscriptDelta: name: receiveTranscriptDelta title: receiveTranscriptDelta payload: $ref: '#/components/schemas/agentSession_transcriptDelta' subpackage_realtimeAgent.Realtime Agent-server-5-receiveAgentStartTalking: name: receiveAgentStartTalking title: receiveAgentStartTalking payload: $ref: '#/components/schemas/agentSession_agentStartTalking' subpackage_realtimeAgent.Realtime Agent-server-6-receiveAgentStopTalking: name: receiveAgentStopTalking title: receiveAgentStopTalking payload: $ref: '#/components/schemas/agentSession_agentStopTalking' subpackage_realtimeAgent.Realtime Agent-server-7-receiveInterruption: name: receiveInterruption title: receiveInterruption payload: $ref: '#/components/schemas/agentSession_interruption' subpackage_realtimeAgent.Realtime Agent-server-8-receiveError: name: receiveError title: receiveError payload: $ref: '#/components/schemas/agentSession_error' subpackage_realtimeAgent.Realtime Agent-client-0-sendInputAudioAppend: name: sendInputAudioAppend title: sendInputAudioAppend payload: $ref: '#/components/schemas/agentSession_inputAudioAppend' subpackage_realtimeAgent.Realtime Agent-client-1-sendInputAudioCommit: name: sendInputAudioCommit title: sendInputAudioCommit payload: $ref: '#/components/schemas/agentSession_inputAudioCommit' subpackage_realtimeAgent.Realtime Agent-client-2-sendInputText: name: sendInputText title: sendInputText payload: $ref: '#/components/schemas/agentSession_inputTextSend' subpackage_realtimeAgent.Realtime Agent-client-3-sendSessionUpdate: name: sendSessionUpdate title: sendSessionUpdate payload: $ref: '#/components/schemas/agentSession_sessionUpdate' subpackage_realtimeAgent.Realtime Agent-client-4-sendSessionClose: name: sendSessionClose title: sendSessionClose payload: $ref: '#/components/schemas/agentSession_sessionClose' schemas: agentSession_sessionCreated: type: object properties: type: type: string enum: - session.created session_id: type: string description: Unique identifier for this session. call_id: type: string description: Call-log identifier; matches the `call_id` in conversation logs. sample_rate: type: integer description: Negotiated audio sample rate in Hz for both input and output audio. channels: type: integer description: Number of audio channels (typically 1). required: - type - session_id - call_id - sample_rate - channels title: agentSession_sessionCreated agentSession_sessionClosed: type: object properties: type: type: string enum: - session.closed reason: type: string description: | Why the session ended. Observed values: - `client_requested` — client sent `session.close`. - `websocket_closed` — the WebSocket dropped. - `ended` — the agent pipeline ended the session (e.g. the agent hung up or the conversation completed). Fatal setup/auth failures do **not** produce `session.closed` — the server emits an `error` and closes the socket with code 4000 instead. required: - type - reason title: agentSession_sessionClosed agentSession_outputAudioDelta: type: object properties: type: type: string enum: - output_audio.delta audio: type: string description: Base64-encoded raw audio chunk. required: - type - audio title: agentSession_outputAudioDelta ChannelsAgentSessionMessagesTranscriptRole: type: string enum: - user - assistant description: Whose utterance this transcript is for. title: ChannelsAgentSessionMessagesTranscriptRole agentSession_transcript: type: object properties: type: type: string enum: - transcript role: $ref: '#/components/schemas/ChannelsAgentSessionMessagesTranscriptRole' description: Whose utterance this transcript is for. text: type: string description: The transcribed (user) or spoken (assistant) text. required: - type - role - text title: agentSession_transcript ChannelsAgentSessionMessagesTranscriptDeltaRole: type: string enum: - user - assistant description: Whose utterance this delta is for. title: ChannelsAgentSessionMessagesTranscriptDeltaRole agentSession_transcriptDelta: type: object properties: type: type: string enum: - transcript.delta role: $ref: '#/components/schemas/ChannelsAgentSessionMessagesTranscriptDeltaRole' description: Whose utterance this delta is for. text: type: string description: Cumulative transcript text for the current turn. required: - type - role - text title: agentSession_transcriptDelta agentSession_agentStartTalking: type: object properties: type: type: string enum: - agent_start_talking timestamp: type: string format: date-time description: > ISO 8601 server-side timestamp of when the agent turn started, included with the event. required: - type title: agentSession_agentStartTalking agentSession_agentStopTalking: type: object properties: type: type: string enum: - agent_stop_talking timestamp: type: string format: date-time description: > ISO 8601 server-side timestamp of when the agent turn ended, included with the event. required: - type title: agentSession_agentStopTalking agentSession_interruption: type: object properties: type: type: string enum: - interruption required: - type title: agentSession_interruption agentSession_error: type: object properties: type: type: string enum: - error code: type: string description: | Machine-readable error code. Values emitted by the gateway: `invalid_event` (a client message failed validation), `no_session` (message received before the session was ready), `internal_error`, or the HTTP status code of a setup failure as a string (e.g. `401`, `400`, `503`). message: type: string description: Human-readable error message. required: - type - code - message title: agentSession_error agentSession_inputAudioAppend: type: object properties: type: type: string enum: - input_audio_buffer.append audio: type: string description: | Base64-encoded raw audio chunk. Format negotiated via the `sample_rate` query param at connection time and returned in `session.created`. Send chunks continuously while the user speaks — the server handles VAD and turn boundaries. required: - type - audio title: agentSession_inputAudioAppend agentSession_inputAudioCommit: type: object properties: type: type: string enum: - input_audio_buffer.commit required: - type title: agentSession_inputAudioCommit agentSession_inputTextSend: type: object properties: type: type: string enum: - input_text.send text: type: string description: User message text. required: - type - text title: agentSession_inputTextSend agentSession_sessionUpdate: type: object properties: type: type: string enum: - session.update required: - type title: agentSession_sessionUpdate agentSession_sessionClose: type: object properties: type: type: string enum: - session.close required: - type title: agentSession_sessionClose ```