Perturbation robustness | Smallest AI Docs

Internal English Perturbation Benchmark

Not a public dataset. The English audio is sliced by perturbation type (Noise, Silence, Telephony 911, Boundary, Disfluency, Long Audios, Repetition, Entity, Accent, Emotion, Speaker Diversity, Speed, Pitch, Volume, Audio Quality) to isolate model weaknesses. Lower WER is better.

Category	Pulse	Assembly	AWS	Deepgram	Scribe
Noise	10.96	11.93	14.19	14.58	10.05
Silence	7.18	4.22	8.22	13.28	10.61
Telephony 911	21.05	23.93	27.88	28.43	20.29
Boundary	2.74	3.09	3.18	3.66	1.73
Disfluency	8.20	7.81	9.23	8.62	9.29
Long Audios	6.60	8.58	11.66	11.16	9.25
Repetition	9.10	9.82	10.39	9.57	10.81
Entity	6.69	10.13	13.35	11.69	9.48
Accent	8.27	7.89	9.51	10.42	7.25
Emotion	13.53	16.34	18.57	18.07	11.84
Speaker Diversity	6.90	6.72	8.81	9.48	5.95
Speed	3.67	3.63	4.40	6.88	3.74
Pitch	2.89	3.07	3.21	4.07	1.61
Volume	2.43	3.05	2.41	3.67	1.47
Audio Quality	2.69	2.86	3.03	4.08	1.60
Average WER	7.53	8.20	9.87	10.51	7.66

Internal Hindi Perturbation Benchmark

Not a public dataset. Hindi audio is sliced by perturbation type to isolate model weaknesses. Compared against Sarvam Saaras v3 and Deepgram Nova-3. Most metrics are WER - lower is better. Entity EDR (↑) is higher is better.

Category	Smallest Pulse	Sarvam Saaras v3	Deepgram Nova-3
Noise	15.76%	22.18%	21.52%
Silence	8.22%	11.38%	18.40%
Entity	8.27%	17.36%	14.67%
Entity NE-WER	13.32%	26.72%	26.58%
Entity EDR (↑)	83.13%	76.13%	67.80%
Boundary	8.03%	17.52%	17.36%
Long Audios	12.00%	18.42%	19.21%
Speed	14.77%	21.39%	38.21%
Pitch	8.14%	11.92%	19.59%
Audio Quality	10.86%	11.75%	19.51%
Volume	7.08%	15.25%	16.76%
Disfluency	10.77%	12.06%	18.44%
Repetition	8.11%	11.27%	20.40%