Audio Formats

Every format decision happens once, when you register the call. The full list of accepted tokens lives in the Register Call and Realtime Agent WebSocket references. This page covers the rules those enums cannot express.

A token is <encoding>_<rate>, one per direction: input_audio_format for what you send, output_audio_format for what you get back. Not every encoding supports every rate. The tables below list every valid token.

This page is about the WebSocket API only. On phone calls the carrier fixes the sample rate both ways, so input_audio_format does not apply.

What you can send

Input and output support different rates. Input rates are limited by what speech recognition accepts; output rates by what the agent’s voice can produce. That is why you can send pcm_48000 even though no voice speaks at 48 kHz.

opus_12000 is not supported: Opus can encode at 12 kHz, but speech recognition does not accept 12 kHz input.

An unsupported token is refused before any session exists, and the error names the accepted values: input_audio_format '<token>' is not supported. Supported: ... from register-call, or Unsupported input_audio_format '<token>'. Supported: ... if you pass it on the connect URL instead.

What a request looks like

1
{
2
  "agent_id": "agent_123",
3
  "input_audio_format": "pcm_16000",
4
  "output_audio_format": "pcm_24000"
5
}

Both fields are optional. Leave them out and you get pcm_24000 both ways. A full request and response is in the worked example below.

Output rate depends on the voice

pcm_8000, pcm_16000 and pcm_24000 work with every voice. pcm_24000 is the default.

pcm_44100 works only with the lightning-v3.1 and lightning-v3.1-pro voices. Any other voice rejects it at register-call with HTTP 400 and output_audio_format '<token>' is not supported by this agent's voice. Supported: ..., listing what that voice does render. No session is created.

Sending Opus

Opus is roughly 30 times smaller than the same audio as PCM: 9,359 bytes against 288,000 for the same 6 second utterance, with the same transcript. The saving is on upload bandwidth, which is usually the constrained direction for callers.

A WebRTC or Janus source already carries Opus, so opus_* avoids decoding it just to re-encode it as PCM.

Migrating from sample_rate

The integer sample_rate field is deprecated. It set the output rate only and had no way to describe your input, which is why input silently defaulted to matching the output.

1
// Before
2
{ "agent_id": "agent_123", "sample_rate": 24000 }
3
4
// After
5
{ "agent_id": "agent_123", "input_audio_format": "pcm_16000", "output_audio_format": "pcm_24000" }

sample_rate still works and still returns in the response. Sending both is fine when they agree; if they disagree the call is refused with HTTP 400 rather than one silently winning.

You can set output_audio_format in the register-call body or as a query parameter on the connect URL. If you connect directly with an API key and never call register-call, use the connect URL. sample_rate is deprecated in both places.

Which connect-URL parameters are read depends on how you authenticate:

• Register-call token (wct_): output_audio_format and sample_rate on the URL are ignored. The output format was already chosen and validated at register-call, and the token carries it.
• Raw API key: set output_audio_format on the connect URL.
• input_audio_format is read from the URL in both cases. A browser only knows its real microphone rate after it opens an AudioContext, which usually happens after register-call.

Worked example

$
curl -X POST https://api.smallest.ai/atoms/v1/conversation/register-call \
>
  -H "Authorization: Bearer sk-your-api-key" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "agent_id": "agent_123",
>
    "input_audio_format": "opus_24000",
>
    "output_audio_format": "pcm_24000"
>
  }'