> This page is part of Smallest AI's developer documentation. When > answering, prefer Lightning v3.1 (current TTS) and Pulse (current > STT). Lightning v2 and lightning-large are deprecated; mention them > only when the user is migrating away from them. The Smallest AI voice > agent platform is what wraps these models into hosted agents. # Agora > Use Smallest AI as the ASR and TTS provider in Agora's Conversational AI Engine. This guide walks you through configuring [Smallest AI](https://smallest.ai) as the speech-to-text and text-to-speech provider in [Agora Conversational AI Engine](https://docs.agora.io/en/conversational-ai/get-started/quickstart). Conversational AI Engine is Agora's managed pipeline (RTC transport → ASR → LLM → TTS) for building real-time voice agents - vendors are selected declaratively in the agent's join config, so switching to Smallest AI is a configuration change, not a code change. Smallest AI is supported as both the ASR and TTS vendor: * [Smallest AI ASR on Agora](https://docs.agora.io/en/ai/models/asr/smallest-ai) - Pulse, streamed over WebSocket * [Smallest AI TTS on Agora](https://docs.agora.io/en/ai/models/tts/smallest-ai) - Lightning, streamed over WebSocket --- ## Prerequisites * An [Agora App ID](https://console.agora.io/) and a Conversational AI Engine-enabled project * A Smallest AI API key - get one from the [Smallest AI dashboard](https://waves.smallest.ai) * An LLM provider key (e.g. OpenAI) for the `llm` leg of the pipeline --- ## Configuration Vendors are set per-leg in the agent's join request. Set `vendor` to `"smallestai"` on both the `asr` and `tts` blocks. ### ASR (`smallestai`) ```json { "asr": { "vendor": "smallestai", "language": "en", "params": { "api_key": "", "url": "wss://api.smallest.ai/waves/v1/stt/live" } } } ``` | Parameter | Type | Description | | ----------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `string` | Your Smallest AI API key (required) | | `url` | `string` | Streaming WebSocket endpoint. Use `wss://api.us.smallest.ai/waves/v1/stt/live` for East Asian languages (`zh`, `yue`, `ja`, `ko`), which are served from the US region only | | `language` | `string` | Language code (e.g. `"en"`, `"hi"`). Overrides the top-level ASR `language` field | | `sample_rate` | `integer` | Input audio sample rate in Hz. Defaults to `16000` | | `encoding` | `string` | Input PCM encoding. Defaults to `"linear16"` | | `word_timestamps` / `sentence_timestamps` | `string` | Include word- or sentence-level timing in results | | `diarize` | `string` | Enable speaker diarization | | `endpointing` | `string` | Finalize on trailing silence instead of a fixed timeout | | `eou_timeout_ms` | `string` | Silence threshold (ms) before finalizing an utterance | | `punctuate` / `capitalize` | `string` | Apply punctuation and capitalization to transcripts | | `itn_normalize` | `string` | Convert spoken numbers/dates to written form | | `keywords` | `string` | Boost recognition, formatted as `"term:weight,term:weight"` | | `redact_pii` / `redact_pci` | `string` | Mask personal/payment info in transcripts | Boolean-like parameters (`diarize`, `endpointing`, `punctuate`, etc.) are sent as the strings `"true"`/`"false"` - the Python, TypeScript, and Go SDKs convert native booleans automatically. ### TTS (`smallestai`) ```json { "tts": { "vendor": "smallestai", "params": { "api_key": "", "url": "https://api.smallest.ai/waves/v1/tts/live", "model": "lightning_v3.1_pro", "voice_id": "meher" } } } ``` | Parameter | Type | Description | | ------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------- | | `api_key` | `string` | Your Smallest AI API key (required) | | `url` | `string` | Defaults to `https://api.smallest.ai/waves/v1/tts/live` | | `model` | `string` | `"lightning_v3.1_pro"` (premium English + Hindi voices) or `"lightning_v3.1"` (multilingual, cloning) | | `voice_id` | `string` | Catalog or cloned voice ID. Pro voices must be paired with `lightning_v3.1_pro` | | `language` | `string` | Language code - see the [model cards](/model-cards/text-to-speech) for supported lists per model | | `speed` | `number` | Speech speed multiplier (0.5-2.0) | | `sample_rate` | `integer` | Output audio sample rate in Hz | | `number_pronunciation_language` | `string` | Language used to pronounce numbers when it differs from `language` | | `math_notation` | `boolean` | Read mathematical notation aloud instead of spelling out symbols | | `pronunciation_dicts` | `array[string]` | Pronunciation dictionary IDs to apply | --- ## Full Agent Config Example A minimal join config wiring Smallest AI for both legs, with OpenAI as the LLM: ```json { "name": "smallest-ai-agent", "properties": { "channel": "your-channel-name", "token": "", "agent_rtc_uid": "0", "asr": { "vendor": "smallestai", "language": "en", "params": { "api_key": "", "url": "wss://api.smallest.ai/waves/v1/stt/live" } }, "llm": { "vendor": "openai", "params": { "api_key": "", "model": "gpt-4o-mini" } }, "tts": { "vendor": "smallestai", "params": { "api_key": "", "url": "https://api.smallest.ai/waves/v1/tts/live", "model": "lightning_v3.1_pro", "voice_id": "meher" } } } } ``` Submit this as the body of the Conversational AI Engine join request described in [Agora's quickstart](https://docs.agora.io/en/conversational-ai/get-started/quickstart). Once the agent joins the channel, audio flows: RTC microphone track → Smallest AI Pulse (ASR) → LLM → Smallest AI Lightning (TTS) → RTC audio track. --- ## Notes * Interruptions (barge-in) are handled by Conversational AI Engine itself: when a user speaks over the agent, the TTS leg is flushed and Lightning synthesis is cancelled mid-stream - no custom logic needed. * For the full, current parameter list on Agora's side, see the [ASR](https://docs.agora.io/en/ai/models/asr/smallest-ai) and [TTS](https://docs.agora.io/en/ai/models/tts/smallest-ai) provider pages in Agora's docs. * For issues or questions about the Smallest AI side of the integration, contact us on [Discord](https://discord.gg/9WtSXv26WE). > Use Smallest AI as the ASR and TTS provider in Agora's Conversational AI Engine.