Major difference with whisper.cpp? about whisperkit HOT 4 OPEN

Thanks for the report @bjnortier :) I just tested:

1-) whisper.cpp + --beam-size 1: Produces the first sentence from the result you shared and early stops
2-) whisper.cpp + --beam-size 5: Almost the same result as you reported above
3-) WhisperKit (always) --beam-size 1: Halts early

We will investigate whether an aggressive hallucination guardrail or some input audio processing is causing this discrepancy for this particular sample and report back.

In general, we do not anticipate any quality drop when moving from whisper.cpp to WhisperKit because the evaluation results on librispeech are almost identical and whisper.cpp has higher error rates on earnings22 (Eval results published here, whisper.cpp is not yet tabulated but eval files are in this repo)

from whisperkit.

Yes, definitely a beam search thing, thanks for this great example. The model wants to predict no speech in the beginning because the audio is quiet, and stops early as @atiorh mentioned. You can get around this somewhat by forcing the first tokens i.e. telling the model there is definitely speech here, by using the decoding option --use-prefill-prompt, but there is still some missing speech.

(crowd shouting) And our walls and shattered shields when the Egypt man comes crashing down. But it is not this day. This day we fight. By all that you hold dear, this good earth, I bid you stand, hand over west. (shouting)

I managed to improve it a bit by scaling the loudness and a simple band pass filter for human speech frequency ranges using this file and this command
swift run whisperkit-cli transcribe --model small --audio-path ~/Downloads/aragorn_loudest_filtered.wav --verbose --word-timestamps --use-prefill-prompt --temperature=0.2

Hold your breath! Hold your breath! Back to Gondor! Abroja! My brothers! I see in your eyes the same fear that would take the heart of me. But they may come when the courage of men fails, when we forsake our friends and break all bonds of fellowship. But it is not this day. An hour of wolves and shattered shields when the age of men comes crashing down. But it is not this day. This day we fight. By all that you hold dear, this good earth I bid you stand, head for the west. (drill whirring)

Will look into what we can clean up on the audio side by default, but beam search will require some reworking with the model, CoreML has some limitations that prevented us from implementing this originally.

from whisperkit.

If you notice more discrepancies, please keep flagging

from whisperkit.

Ok interesting, thanks for the info!

from whisperkit.