Diarization

Speaker diarization error rate and speed against other STT APIs.

Diarization

Pulse ships speaker diarization on both batch and streaming through the same API. See the Diarization feature page for the response shape and the enable flag.

Metric. DER (Diarization Error Rate) is the percent of speech time handled wrong (wrong speaker + missed speech + falsely detected speech). Scored with no collar, overlapping speech counted, permutation-optimal speaker matching, duration-pooled via pyannote.metrics. Lower is better.

Streaming DER vs other STT APIs

Per-dataset DER. Lower is better. - means we have not scored ElevenLabs on that dataset yet.

Dataset	Pulse	ElevenLabs	Deepgram
ami_ihm (headset meeting)	15.28	33.6	29.03
ami_sdm (far-field meeting)	23.13	37.6	37.86
dihard3_farfield (clean)	33.18	-	47.69
sbcsae (clean)	31.97	-	36.77
callhome_eng (English phone)	14.94	23.0	27.71
callhome_deu (German phone)	14.9	23.9	25.8
callhome_spa (Spanish phone)	31.0	39.6	41.4
simsamu (French emergency 911)	14.2	16.3	38.1
macro	23.70	-	35.81

Streaming DER on Indic languages

Streaming diarization measured end-to-end through the WebSocket API at concurrency 1. Lower is better. Pulse macro-average across all 22 Indic language codes is 23.2 DER; Pyannote Live-1 is 37.1; Deepgram Live is 77.6 (Deepgram Live does not have a first-class Indic pipeline).

Speed (real-time factor)

RTF = processing time / audio length. Lower is faster. 1/RTF is “times faster than real-time.” Measured through each vendor’s API at concurrency 1 with diarization enabled, the round-trip a customer actually sees, not local compute.

Vendor	RTF (median)	Times real-time
Pulse	~0.004	~240×
Deepgram	~0.007	~140×
pyannoteAI	~0.022	~45×
AssemblyAI	~0.028	~36×
ElevenLabs	~0.059	~17×

Streaming diarization emits speaker labels on is_final: true transcription events with roughly the same latency as transcription itself (~1 s tail from utterance end to final event on typical connectivity).

Third-party verification: VoiceArena Diarization Bench

Independent confirmation from VoiceArena’s Diarization Bench, a third-party leaderboard that scores every system on the same 139 sessions (~22 hours) of human-verified English audio, 2-8 speakers per session, using DIHARD scoring (md-eval) at a 0 ms collar - the strictest setting, no slack allowed at speaker-change boundaries.

On the Diarization + ASR track - the “with ASR” system-type filter, for systems that return speaker labels through an integrated transcription pipeline, same as Pulse - Pulse ranks #1:

Rank	Provider	Model	DER %	JER %
1	Smallest AI	Pulse	24.36	32.91
2	Mistral AI	Voxtral Mini Transcribe 3	40.41	50.48
3	ElevenLabs	Scribe v2	40.71	50.81
4	Mistral AI	Voxtral Small Transcribe 3	41.14	50.80
5	Meta	Muse Voice Transcribe	42.06	49.15
6	AssemblyAI	Universal-3.5 Pro	44.64	54.85
7	Soniox	Soniox v5 Async	49.78	60.36
8	Speechmatics	Enhanced	53.30	66.43
9	Deepgram	Nova-3	67.13	73.84

Pulse’s error is driven mostly by false alarm (14.91 of its 24.36 DER), the opposite profile from most of the field, which mostly misses speech outright (Deepgram’s missed-speech component alone is 57.56 of its 67.13 DER).

By condition, 0 ms collar:

Condition	Pulse DER %
In-person	24.7
Online	22.4
2 speakers	13.3
3 speakers	20.9
4 speakers	25.6
5-8 speakers	39.2
Overlap < 15%	16.7
Overlap 15-30%	19.9
Overlap 30%+	30.8

View full methodology here: VoiceArena Diarization Bench methodology.