Skip to navigation
Text to Speech (Lightning)

Word-level timestamps

View as Markdown
Real-Time

Pass word_timestamps: true on a Lightning TTS WebSocket request to receive per-word timing events interleaved with the audio stream. Each event tells you exactly when a word starts and ends in the generated audio.

When to use this

  • Live captions - render words to the screen as they’re spoken.
  • Karaoke-style highlighting - sync the active-word UI to playback.
  • Avatar lip-sync - drive viseme transitions off word boundaries.
  • Word-level analytics - log per-word latency or downstream events.

Endpoint support

SurfaceWord timestamps
wss://api.smallest.ai/waves/v1/tts/live (WebSocket)✅
POST /waves/v1/tts (sync HTTP)❌ - flag accepted, silently ignored
POST /waves/v1/tts/live (HTTP SSE)❌ - same

If you need word timing, use the WebSocket path.

Voice support

Word events are emitted only when the chosen voice family has the aligner checkpoint baked in.

LanguageVoice familyWord events
English (en)Base-queue voices - meher, devansh, kartik, maithili, liam, avery✅
Hindi (hi)Base-queue voices (same list)✅
Marathi / Bengali / Gujarati / Punjabi / Odianorth-Indic family❌
Tamil / Telugu / Kannada / Malayalamsouth-Indic family❌

For unsupported voice families the flag is accepted - audio works normally, but no word_timestamp frames are emitted.

Request

{
  "voice_id": "meher",
  "text": "I bought 3 cats for $100 on Dec 25th",
  "model": "lightning_v3.1_pro",
  "sample_rate": 44100,
  "word_timestamps": true
}

word_timestamps defaults to false. Clients that don’t set the flag see no behavior change.

Response frames

The server interleaves word_timestamp frames with chunk frames in audio-time order, followed by a single complete:

chunk → chunk → word_timestamp{id:0} → chunk → word_timestamp{id:1} → … → complete

Each word_timestamp frame:

{
  "session_id": "ws_550e8400-e29b",
  "request_id": "task_6ba7b810-9dad",
  "status": "word_timestamp",
  "data": {
    "id": 5,
    "word": "$100",
    "start": 1.12,
    "end": 1.92
  }
}
FieldTypeDescription
data.idinteger0-indexed position of the word within the input text.
data.wordstringExact substring from the input - un-normalized. "$100" stays "$100", "25th" stays "25th", "3" stays "3". Non-Latin scripts are preserved verbatim (e.g. Devanagari for Hindi).
data.startfloat (seconds)Start of the word in the audio stream.
data.endfloat (seconds)End of the word.

Example

# Requires: pip install websocket-client
# Opt into word_timestamps on the v3.1 WebSocket, save the audio to WAV,
# and dump every received word event with its timing.
import base64
import json
import os
import wave
from websocket import WebSocketApp
API_KEY = os.environ["SMALLEST_API_KEY"]
WS_URL = "wss://api.smallest.ai/waves/v1/tts/live"
SAMPLE_RATE = 44100
TEXT = "I bought 3 cats for $100 on Dec 25th"
VOICE_ID = "meher"
audio_chunks: list[str] = []
word_events: list[dict] = []
def on_open(ws):
    ws.send(json.dumps({
        "voice_id": VOICE_ID,
        "text": TEXT,
        "sample_rate": SAMPLE_RATE,
        "model": "lightning_v3.1_pro",
        "word_timestamps": True,
    }))
def on_message(ws, raw):
    msg = json.loads(raw)
    status = msg.get("status")
    if status == "chunk":
        audio_chunks.append(msg["data"]["audio"])
    elif status == "word_timestamp":
        w = msg["data"]
        word_events.append(w)
        print(f"  id={w['id']:<2}  {w['word']!r:<15}  {w['start']:.3f}s → {w['end']:.3f}s")
    elif status == "complete":
        ws.close()
def on_close(_ws, *_):
    pcm = b"".join(base64.b64decode(c) for c in audio_chunks)
    with wave.open("out.wav", "wb") as f:
        f.setnchannels(1); f.setsampwidth(2); f.setframerate(SAMPLE_RATE)
        f.writeframes(pcm)
    print(f"\nsaved out.wav  ({len(audio_chunks)} chunks, {len(word_events)} word events)")
WebSocketApp(
    WS_URL,
    header=[f"Authorization: Bearer {API_KEY}"],
    on_open=on_open,
    on_message=on_message,
    on_close=on_close,
).run_forever()

Backward compatibility

Pure addition. Existing integrations that don’t set word_timestamps see the same wire shape as before - same chunk frames, same complete terminator, no new event types to handle.

See also