Skip to navigation

Benchmarks

Every benchmark we publish, by model, one page each.
View as Markdown

Numbers are measured in-region on production endpoints. Each page states the dataset, the method, and the models compared. For definitions of the metrics, see the metrics pages linked under each model.

Lightning (text to speech)

Lightning v3.1 and Lightning v3.1 Pro. Overview and metric definitions.

Latency

Time to first byte and real-time factor.

Listener ratings

Naturalness, expressiveness, delivery.

Accuracy

Whisper-measured pronunciation accuracy.

MOS

Mean opinion score v2.

Pulse (speech to text)

Pulse and Pulse Pro. Overview, metric definitions, and how to run the evaluation yourself.

Open ASR Leaderboard

Pulse Pro tied for second at 5.42% WER.

Latency

Time to first transcript across concurrency.

FLEURS

Per-language WER, streaming and pre-recorded.

ESB English

Eight English datasets, streaming.

Hindi

Multi-dataset Hindi accuracy.

East Asian languages

Mandarin, Cantonese, Japanese, Korean.

WildASR robustness

Noisy, accented, real-world audio.

Contact center calls

Real English and Hindi call recordings.

Perturbation robustness

Noise, speed, pitch and codec suites.

Diarization

Speaker DER and real-time factor.

Hydra (speech to speech)

AIEWF S2S benchmark

Hydra against eight production realtime voice models.

Metrics overview

Definitions of the metrics Hydra is scored on.