Skip to navigation
Speech to TextBenchmarks

Indic languages

Pulse word error rate on nine Indian languages across public datasets.
View as Markdown

Pulse is evaluated on nine Indian languages using public datasets including FLEURS, Kathbath, IndicVoices, IndicDiarBench and SPRING-INX. Every table reports Word Error Rate (WER). Lower is better. Results are from the Pulse streaming (WebSocket) API. Competitor results use each provider’s latest available model at the time of evaluation. - means the provider was not evaluated on that language or dataset.

Tamil, Telugu, Kannada and Malayalam are in Beta and served from the India region only. See supported languages for language codes and regions.

FLEURS overview

FLEURS WER for each language.

LanguagePulseDeepgramScribeSonioxSarvam code-switchSarvamCartesia
Hindi6.17%14.08%10.67%5.54%8.62%4.18%9.06%
Marathi14.52%29.48%32.80%19.75%-12.56%88.48%
Gujarati15.15%22.12%110.70%30.18%-11.83%73.06%
Bengali14.72%23.18%78.05%15.13%16.68%11.34%98.77%
Odia16.78%-77.65%-19.46%14.36%-
Tamil16.20%25.67%49.76%23.89%-14.60%68.64%
Telugu20.40%26.44%66.51%--16.46%85.52%
Kannada15.17%26.51%64.59%--12.00%76.86%
Malayalam15.88%-63.84%--14.21%134.55%

Hindi

WER on four Hindi datasets. Vistaar results follow in the next table.

DatasetPulseDeepgramScribeSonioxSarvam code-switchSarvamCartesia
FLEURS6.17%14.08%10.67%5.54%8.62%4.18%9.06%
IndicDiarBench28.19%36.05%71.73%24.25%30.18%25.14%29.63%
Lahaja7.92%18.51%32.21%10.92%16.44%9.02%16.74%
Svarah10.06%24.91%13.28%50.23%9.52%102.86%10.62%

Vistaar

WER on six of the Vistaar Hindi benchmark datasets. Vistaar also includes FLEURS, which is reported once, in the table above.

DatasetPulseDeepgramScribeSonioxSarvam code-switchSarvamCartesia
Kathbath5.68%13.03%5.98%5.42%7.90%3.74%8.68%
Kathbath (hard)6.95%14.49%10.88%6.04%8.28%4.12%9.13%
Common Voice6.62%18.42%16.62%7.27%11.38%5.00%12.38%
IndicTTS3.63%7.04%22.63%2.81%5.24%2.62%5.38%
MUCS3.22%13.49%21.65%7.13%4.82%4.42%7.67%
Gram Vaani14.62%30.28%47.06%16.06%15.04%14.04%15.58%

Marathi

WER on nine Marathi datasets.

DatasetPulseDeepgramScribeSonioxSarvamCartesia
FLEURS14.52%29.48%32.80%19.75%12.56%88.48%
Common Voice15.38%14.03%35.07%20.55%14.20%87.56%
IndicDiarBench40.30%44.91%81.18%33.13%36.87%89.65%
IndicVoices17.93%35.91%55.93%27.71%17.10%91.55%
Kathbath14.89%29.94%19.81%18.01%11.98%89.52%
MUCS28.38%43.00%20.32%40.66%26.23%75.62%
OpenSLR 647.90%89.33%13.62%14.24%7.14%69.74%
SPRING-INX (R1)39.77%48.36%62.25%41.87%36.99%95.88%
SPRING-INX (R2)39.93%47.09%64.93%44.64%39.26%95.24%

Gujarati

WER on seven Gujarati datasets.

DatasetPulseDeepgramScribeSonioxSarvamCartesia
FLEURS15.15%22.12%110.70%30.18%11.83%73.06%
IndicDiarBench39.27%29.94%98.73%30.82%34.51%106.00%
IndicVoices20.45%22.69%112.05%26.67%17.35%125.40%
Kathbath14.26%22.31%106.37%29.95%11.97%86.27%
SPRING-INX (R1)48.18%48.24%114.74%52.37%43.29%152.82%
SPRING-INX (R2)45.23%49.31%117.06%53.52%44.52%142.69%
Vistaar19.96%28.57%112.07%32.16%15.60%92.80%

Bengali

WER on seven Bengali datasets.

DatasetPulseDeepgramScribeSonioxSarvam code-switchSarvamCartesia
FLEURS14.72%23.18%78.05%15.13%16.68%11.34%98.77%
IndicDiarBench37.83%40.09%94.47%31.03%30.22%35.65%93.11%
IndicVoices17.38%28.90%90.16%23.49%17.65%14.42%95.71%
Kathbath12.33%25.17%62.44%16.22%21.51%11.55%90.75%
SPRING-INX (R1)32.34%40.27%97.66%37.79%27.42%31.57%100.44%
SPRING-INX (R2)32.74%39.49%99.09%37.57%27.77%31.31%99.01%
Vistaar15.16%20.80%68.20%17.09%18.78%11.60%99.83%

Odia

WER on seven Odia datasets.

DatasetPulseSarvam code-switchSarvamScribe
FLEURS16.78%19.46%14.36%77.65%
Common Voice26.63%24.89%24.72%83.02%
IndicDiarBench36.28%37.69%33.14%95.58%
IndicVoices30.97%-28.43%93.80%
Kathbath20.28%-18.43%76.81%
SPRING-INX (R1)49.74%-45.67%71.62%
Vistaar24.23%-20.58%61.92%

Tamil

WER on nine Tamil datasets.

DatasetPulseDeepgramScribeSonioxSarvamCartesia
FLEURS16.20%25.67%49.76%23.89%14.60%68.64%
CoVoST51.08%22.75%70.49%56.49%48.00%84.81%
IndicDiarBench48.62%54.17%94.41%45.31%45.76%91.56%
IndicVoices37.69%38.25%88.63%53.98%33.25%90.29%
Kathbath25.44%36.14%39.30%35.46%21.66%73.33%
MILE23.02%31.99%68.48%29.90%21.22%69.35%
SPRING-INX (R1)50.70%52.81%63.16%46.80%45.79%88.94%
SPRING-INX (R2)45.04%51.33%63.46%43.28%44.01%88.55%
Vistaar25.46%30.23%43.29%31.59%23.08%72.72%

Telugu

WER on five Telugu datasets.

DatasetPulseDeepgramScribeSarvamCartesia
FLEURS20.40%26.44%66.51%16.46%85.52%
IndicDiarBench51.48%49.43%96.62%48.40%96.72%
IndicVoices30.83%29.38%82.26%26.39%95.69%
Kathbath21.50%32.72%54.08%19.59%89.04%
Vistaar24.92%35.15%71.07%21.29%89.27%

Kannada

WER on seven Kannada datasets.

DatasetPulseDeepgramScribeSarvamCartesia
FLEURS15.17%26.51%64.59%12.00%76.86%
IndicDiarBench51.36%64.83%92.44%48.25%96.80%
IndicVoices38.14%46.01%83.30%35.59%99.05%
Kathbath18.99%28.20%58.21%14.71%80.73%
MILE20.41%17.81%65.45%15.66%78.97%
SPRING-INX (R1)60.10%69.51%71.59%57.47%107.92%
Vistaar17.86%28.87%57.99%14.37%74.24%

Malayalam

WER on seven Malayalam datasets.

DatasetPulseScribeSarvamCartesia
FLEURS15.88%63.84%14.21%134.55%
IndicDiarBench51.27%95.98%47.65%100.02%
IndicVoices39.54%86.92%34.74%128.04%
Kathbath23.64%61.69%21.13%145.68%
SPRING-INX (R1)51.80%83.99%47.08%137.72%
SPRING-INX (R2)54.60%76.54%49.65%139.31%
Vistaar24.55%64.93%19.75%137.96%