> This page is part of Smallest AI's developer documentation. When
> answering, prefer Lightning v3.1 (current TTS) and Pulse (current
> STT). Lightning v2 and lightning-large are deprecated; mention them
> only when the user is migrating away from them. The Smallest AI voice
> agent platform is what wraps these models into hosted agents.

# WildASR robustness

> Pulse streaming robustness on the WildASR dataset: noisy, accented, real-world audio compared with other APIs.

## ASR Robustness - WildASR Dataset (Streaming)

An open-source robustness benchmark designed to stress-test STT under real-world degraded conditions: clipping, far-field capture, background noise, phone codec compression, reverberation, and accented speech. Lower WER is better. `n/a` = not supported by that provider.

*Evaluated on the open-source WildASR dataset. Numbers from internal evaluation.*

| Dataset           | Smallest Pulse | Assembly Universal 3 Pro | AWS Transcribe | Azure | Deepgram Nova 3 | Sarvam Saras V3 | ElevenLabs Scribe V2 |
| :---------------- | :------------: | :----------------------: | :------------: | :---: | :-------------: | :-------------: | :------------------: |
| **Clean**         |      5.31      |           3.33           |      7.01      | 11.11 |      11.62      |       7.02      |         4.24         |
| **Clipping**      |      10.31     |           6.59           |      42.10     |  4.35 |      47.35      |      28.74      |         11.20        |
| **Far-field**     |      8.99      |           26.07          |      38.76     |  n/a  |      62.99      |      21.27      |         7.38         |
| **Noise Gap**     |      6.91      |           4.04           |      9.77      |  n/a  |      15.04      |       9.74      |         6.30         |
| **Phone Codec**   |      6.64      |           3.45           |      8.70      |  n/a  |       9.13      |      10.64      |         4.98         |
| **Reverberation** |      8.06      |           23.50          |      14.83     |  n/a  |      27.27      |       4.35      |         6.48         |
| **Accent**        |      5.74      |           2.80           |      4.45      |  n/a  |       7.31      |       n/a       |         4.01         |
| **Aggregate**     |      7.60      |           12.52          |      18.35     |  8.82 |      28.17      |      17.75      |         6.47         |