Agora | Smallest AI Docs

This guide walks you through configuring Smallest AI as the speech-to-text and text-to-speech provider in Agora Conversational AI Engine. Conversational AI Engine is Agora’s managed pipeline (RTC transport → ASR → LLM → TTS) for building real-time voice agents - vendors are selected declaratively in the agent’s join config, so switching to Smallest AI is a configuration change, not a code change.

Smallest AI is supported as both the ASR and TTS vendor:

Smallest AI ASR on Agora - Pulse, streamed over WebSocket
Smallest AI TTS on Agora - Lightning, streamed over WebSocket

Prerequisites

An Agora App ID and a Conversational AI Engine-enabled project
A Smallest AI API key - get one from the Smallest AI dashboard
An LLM provider key (e.g. OpenAI) for the llm leg of the pipeline

Configuration

Vendors are set per-leg in the agent’s join request. Set vendor to "smallestai" on both the asr and tts blocks.

ASR (`smallestai`)

{
  "asr": {
    "vendor": "smallestai",
    "language": "en",
    "params": {
      "api_key": "<smallest_ai_api_key>",
      "url": "wss://api.smallest.ai/waves/v1/stt/live"
    }
  }
}

Parameter	Type	Description
`api_key`	`string`	Your Smallest AI API key (required)
`url`	`string`	Streaming WebSocket endpoint. Use `wss://api.us.smallest.ai/waves/v1/stt/live` for East Asian languages (`zh`, `yue`, `ja`, `ko`), which are served from the US region only
`language`	`string`	Language code (e.g. `"en"`, `"hi"`). Overrides the top-level ASR `language` field
`sample_rate`	`integer`	Input audio sample rate in Hz. Defaults to `16000`
`encoding`	`string`	Input PCM encoding. Defaults to `"linear16"`
`word_timestamps` / `sentence_timestamps`	`string`	Include word- or sentence-level timing in results
`diarize`	`string`	Enable speaker diarization
`endpointing`	`string`	Finalize on trailing silence instead of a fixed timeout
`eou_timeout_ms`	`string`	Silence threshold (ms) before finalizing an utterance
`punctuate` / `capitalize`	`string`	Apply punctuation and capitalization to transcripts
`itn_normalize`	`string`	Convert spoken numbers/dates to written form
`keywords`	`string`	Boost recognition, formatted as `"term:weight,term:weight"`
`redact_pii` / `redact_pci`	`string`	Mask personal/payment info in transcripts

Boolean-like parameters (diarize, endpointing, punctuate, etc.) are sent as the strings "true"/"false" - the Python, TypeScript, and Go SDKs convert native booleans automatically.

TTS (`smallestai`)

{
  "tts": {
    "vendor": "smallestai",
    "params": {
      "api_key": "<smallest_ai_api_key>",
      "url": "https://api.smallest.ai/waves/v1/tts/live",
      "model": "lightning_v3.1_pro",
      "voice_id": "meher"
    }
  }
}

Parameter	Type	Description
`api_key`	`string`	Your Smallest AI API key (required)
`url`	`string`	Defaults to `https://api.smallest.ai/waves/v1/tts/live`
`model`	`string`	`"lightning_v3.1_pro"` (premium English + Hindi voices) or `"lightning_v3.1"` (multilingual, cloning)
`voice_id`	`string`	Catalog or cloned voice ID. Pro voices must be paired with `lightning_v3.1_pro`
`language`	`string`	Language code - see the model cards for supported lists per model
`speed`	`number`	Speech speed multiplier (0.5-2.0)
`sample_rate`	`integer`	Output audio sample rate in Hz
`number_pronunciation_language`	`string`	Language used to pronounce numbers when it differs from `language`
`math_notation`	`boolean`	Read mathematical notation aloud instead of spelling out symbols
`pronunciation_dicts`	`array[string]`	Pronunciation dictionary IDs to apply

Full Agent Config Example

A minimal join config wiring Smallest AI for both legs, with OpenAI as the LLM:

{
  "name": "smallest-ai-agent",
  "properties": {
    "channel": "your-channel-name",
    "token": "<agora_rtc_token>",
    "agent_rtc_uid": "0",
    "asr": {
      "vendor": "smallestai",
      "language": "en",
      "params": {
        "api_key": "<smallest_ai_api_key>",
        "url": "wss://api.smallest.ai/waves/v1/stt/live"
      }
    },
    "llm": {
      "vendor": "openai",
      "params": {
        "api_key": "<openai_api_key>",
        "model": "gpt-4o-mini"
      }
    },
    "tts": {
      "vendor": "smallestai",
      "params": {
        "api_key": "<smallest_ai_api_key>",
        "url": "https://api.smallest.ai/waves/v1/tts/live",
        "model": "lightning_v3.1_pro",
        "voice_id": "meher"
      }
    }
  }
}

Submit this as the body of the Conversational AI Engine join request described in Agora’s quickstart. Once the agent joins the channel, audio flows: RTC microphone track → Smallest AI Pulse (ASR) → LLM → Smallest AI Lightning (TTS) → RTC audio track.

Notes

Interruptions (barge-in) are handled by Conversational AI Engine itself: when a user speaks over the agent, the TTS leg is flushed and Lightning synthesis is cancelled mid-stream - no custom logic needed.
For the full, current parameter list on Agora’s side, see the ASR and TTS provider pages in Agora’s docs.
For issues or questions about the Smallest AI side of the integration, contact us on Discord.