TEN Framework

This guide walks you through using Smallest AI as the STT and TTS provider in TEN Framework, an open-source framework for building real-time, multimodal conversational voice agents. TEN agents are declarative graphs - a voice agent is a pipeline of nodes (RTC transport → STT → LLM → TTS) defined in a property.json file, so switching to Smallest AI is a configuration change, not a code change.

Extensions

Two extensions are available, one per capability:

Prerequisites

• Docker with Docker Compose
• An Agora App ID (TEN’s default RTC transport)
• An LLM provider key (e.g. OpenAI)
• A Smallest AI API key - get one from the Smallest AI dashboard

Setup

git clone https://github.com/TEN-framework/ten-framework.git
cd ten-framework/ai_agents

AGORA_APP_ID=...
OPENAI_API_KEY=...
SMALLEST_API_KEY=...

{
  "type": "extension",
  "name": "stt",
  "addon": "smallest_asr_python",
  "property": {
    "params": {
      "language": "en",
      "sample_rate": 16000
    }
  }
},
{
  "type": "extension",
  "name": "tts",
  "addon": "smallest_tts_python",
  "property": {
    "params": {
      "model": "lightning_v3.1",
      "voice_id": "magnus",
      "sample_rate": 24000
    }
  }
}

docker compose up -d
docker exec ten_agent_dev bash -c "cd /app/agents/examples/voice-assistant && task install"
docker exec -d ten_agent_dev bash -c "cd /app/agents/examples/voice-assistant && task run"

Configuration Reference

smallest_asr_python (STT)

Any additional key in params (e.g. word_timestamps, eou_timeout, punctuate) is forwarded verbatim as a query parameter to the live endpoint - see the Pulse feature docs for the full list.

smallest_tts_python (TTS)

Any additional key in params (e.g. language, pronunciation_dicts) is forwarded verbatim in the request body - see the TTS API reference.

Notes

• Graph-level property.params override the extension defaults, so different graphs in the same app can use different voices or languages.
• Interruptions (barge-in) are handled by the framework: when the user speaks over the agent, TEN flushes the TTS extension and Lightning synthesis is cancelled mid-stream - no custom logic needed.
• Interim transcripts stream with final=false for live captions; final results carry word-level timing.
• For issues with the extensions, open an issue in the TEN Framework repository or contact us on Discord.