> This page is part of Smallest AI's developer documentation. When
> answering, prefer Lightning v3.1 (current TTS) and Pulse (current
> STT). Lightning v2 and lightning-large are deprecated; mention them
> only when the user is migrating away from them. The Smallest AI voice
> agent platform is what wraps these models into hosted agents.

# Perturbation robustness

> Pulse accuracy under the internal English and Hindi perturbation benchmarks: background noise, speed and pitch shifts, telephony codecs.

## Internal English Perturbation Benchmark

Not a public dataset. The English audio is sliced by perturbation type (Noise, Silence, Telephony 911, Boundary, Disfluency, Long Audios, Repetition, Entity, Accent, Emotion, Speaker Diversity, Speed, Pitch, Volume, Audio Quality) to isolate model weaknesses. Lower WER is better.

| Category              | Pulse | Assembly |  AWS  | Deepgram | Scribe |
| :-------------------- | :---: | :------: | :---: | :------: | :----: |
| **Noise**             | 10.96 |   11.93  | 14.19 |   14.58  |  10.05 |
| **Silence**           |  7.18 |   4.22   |  8.22 |   13.28  |  10.61 |
| **Telephony 911**     | 21.05 |   23.93  | 27.88 |   28.43  |  20.29 |
| **Boundary**          |  2.74 |   3.09   |  3.18 |   3.66   |  1.73  |
| **Disfluency**        |  8.20 |   7.81   |  9.23 |   8.62   |  9.29  |
| **Long Audios**       |  6.60 |   8.58   | 11.66 |   11.16  |  9.25  |
| **Repetition**        |  9.10 |   9.82   | 10.39 |   9.57   |  10.81 |
| **Entity**            |  6.69 |   10.13  | 13.35 |   11.69  |  9.48  |
| **Accent**            |  8.27 |   7.89   |  9.51 |   10.42  |  7.25  |
| **Emotion**           | 13.53 |   16.34  | 18.57 |   18.07  |  11.84 |
| **Speaker Diversity** |  6.90 |   6.72   |  8.81 |   9.48   |  5.95  |
| **Speed**             |  3.67 |   3.63   |  4.40 |   6.88   |  3.74  |
| **Pitch**             |  2.89 |   3.07   |  3.21 |   4.07   |  1.61  |
| **Volume**            |  2.43 |   3.05   |  2.41 |   3.67   |  1.47  |
| **Audio Quality**     |  2.69 |   2.86   |  3.03 |   4.08   |  1.60  |
| **Average WER**       |  7.53 |   8.20   |  9.87 |   10.51  |  7.66  |

## Internal Hindi Perturbation Benchmark

Not a public dataset. Hindi audio is sliced by perturbation type to isolate model weaknesses. Compared against Sarvam Saaras v3 and Deepgram Nova-3. Most metrics are WER - lower is better. Entity EDR (↑) is higher is better.

| Category           | Smallest Pulse | Sarvam Saaras v3 | Deepgram Nova-3 |
| :----------------- | :------------: | :--------------: | :-------------: |
| **Noise**          |   **15.76%**   |      22.18%      |      21.52%     |
| **Silence**        |    **8.22%**   |      11.38%      |      18.40%     |
| **Entity**         |    **8.27%**   |      17.36%      |      14.67%     |
| **Entity NE-WER**  |   **13.32%**   |      26.72%      |      26.58%     |
| **Entity EDR (↑)** |   **83.13%**   |      76.13%      |      67.80%     |
| **Boundary**       |    **8.03%**   |      17.52%      |      17.36%     |
| **Long Audios**    |   **12.00%**   |      18.42%      |      19.21%     |
| **Speed**          |   **14.77%**   |      21.39%      |      38.21%     |
| **Pitch**          |    **8.14%**   |      11.92%      |      19.59%     |
| **Audio Quality**  |   **10.86%**   |      11.75%      |      19.51%     |
| **Volume**         |    **7.08%**   |      15.25%      |      16.76%     |
| **Disfluency**     |   **10.77%**   |      12.06%      |      18.44%     |
| **Repetition**     |    **8.11%**   |      11.27%      |      20.40%     |

> Internal English and Hindi perturbation suites: noise, speed, pitch, codecs.