TEN Framework

This guide walks you through using Smallest AI as the STT and TTS provider in TEN Framework, an open-source framework for building real-time, multimodal conversational voice agents. TEN agents are declarative graphs — a voice agent is a pipeline of nodes (RTC transport → STT → LLM → TTS) defined in a property.json file, so switching to Smallest AI is a configuration change, not a code change.

Extensions

Two extensions are available, one per capability:

Prerequisites

• Docker with Docker Compose
• An Agora App ID (TEN’s default RTC transport)
• An LLM provider key (e.g. OpenAI)
• A Smallest AI API key — get one from the Smallest AI dashboard

Setup

$
git clone https://github.com/TEN-framework/ten-framework.git
$
cd ten-framework/ai_agents

$
AGORA_APP_ID=...
$
OPENAI_API_KEY=...
$
SMALLEST_API_KEY=...

1
{
2
  "type": "extension",
3
  "name": "stt",
4
  "addon": "smallest_asr_python",
5
  "property": {
6
    "params": {
7
      "language": "en",
8
      "sample_rate": 16000
9
    }
10
  }
11
},
12
{
13
  "type": "extension",
14
  "name": "tts",
15
  "addon": "smallest_tts_python",
16
  "property": {
17
    "params": {
18
      "model": "lightning_v3.1",
19
      "voice_id": "magnus",
20
      "sample_rate": 24000
21
    }
22
  }
23
}

$
docker compose up -d
$
docker exec ten_agent_dev bash -c "cd /app/agents/examples/voice-assistant && task install"
$
docker exec -d ten_agent_dev bash -c "cd /app/agents/examples/voice-assistant && task run"

Configuration Reference

smallest_asr_python (STT)

Any additional key in params (e.g. word_timestamps, eou_timeout, punctuate) is forwarded verbatim as a query parameter to the live endpoint — see the Pulse feature docs for the full list.

smallest_tts_python (TTS)

Any additional key in params (e.g. language, pronunciation_dicts) is forwarded verbatim in the request body — see the TTS API reference.

Notes

• Graph-level property.params override the extension defaults, so different graphs in the same app can use different voices or languages.
• Interruptions (barge-in) are handled by the framework: when the user speaks over the agent, TEN flushes the TTS extension and Lightning synthesis is cancelled mid-stream — no custom logic needed.
• Interim transcripts stream with final=false for live captions; final results carry word-level timing.
• For issues with the extensions, open an issue in the TEN Framework repository or contact us on Discord.