Comments (16)
WilliamTambellini
commented on June 24, 2024
1
I confirm @magnacartatron
With 1.6.1:
$ ./main -m ggml-large-v3.bin samples/SonyGroupCorpEarningsQ32024_137031005111_English.wav -l english
...
main: processing 'samples/SonyGroupCorpEarningsQ32024_137031005111_English.wav' (67585536 samples, 4224.1 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = english, task = transcribe, timestamps = 1 ...
[00:00:00.000 --> 00:00:15.520] We will now begin FI 2023 Q3, Consolidated Financial Results Announcement for Sony Group Corporation, IAM Okada, Corporate Communications Service Master of Ceremonies.
[00:00:15.520 --> 00:00:21.700] The people on the stage are Mr. Hiroki Totoki, President, COO, and CFO.
[00:00:22.700 --> 00:00:33.300] Ms. Naomi Matsuoka, Senior Vice President in Charge of Corporate Planning and Control, Lead of Group Diversity Equipment and Inclusion, Support for Financial Services and Entertainment Area.
[00:00:33.300 --> 00:00:39.160] Mr. Sadahiko Hayakawa, Senior Vice President in Charge of Finance and IR.
[00:00:39.160 --> 00:00:48.480] These three people will be explaining the FI 2023 Q3 results and fully forecast, followed by Q&A.
[00:00:48.480 --> 00:00:52.420] A total of 70.
[00:00:52.420 --> 00:00:53.980] 30 minutes is allocated.
[00:00:53.980 --> 00:00:55.940] Mr. Totoki, the floor is yours.
[00:00:55.940 --> 00:01:05.140] Today, after Mr. Matsuoka and Mr. Hayakawa explain the contents shown here, I will summarize the entire earnings briefly.
[00:01:05.140 --> 00:01:07.340] Mr. Hayakawa, please go ahead.
[00:01:07.340 --> 00:01:11.820] Matsuoka and Hayakawa will explain.
[00:01:11.820 --> 00:01:21.860] Consolidated sales for the quarter were 3,747.5 billion yen, a significant increase of 22%,
[00:01:21.860 --> 00:01:22.400] compared to the previous quarter.
[00:01:22.400 --> 00:01:22.400] This is a significant increase of 22% compared to the previous quarter.
[00:01:22.400 --> 00:01:22.400] This is a significant increase of 22% compared to the previous quarter.
[00:01:22.400 --> 00:01:22.400] This is a significant increase of 22% compared to the previous quarter.
but whisper 1.5.2 outputs correctly.
@ggerganov that s a critical bug. Any way for someone from your team to work on it asap? Best.
from whisper.cpp.
ggerganov
commented on June 24, 2024
1
There is mounting evidence that the large v3 model is broken and repeats more than v2, so I recommend not using it. It's not a whisper.cpp problem - it's the model data itself
from whisper.cpp.
WilliamTambellini
commented on June 24, 2024
1.5.5 also hallucinates
from whisper.cpp.
WilliamTambellini
commented on June 24, 2024
1.5.4 does NOT hallucinate.
from whisper.cpp.
magnacartatron
commented on June 24, 2024
@ggerganov I agree with you, but I find it strange that this is happening with CUDA only and not Metal. On same exact sample for same exact version running side by side, Metal performs with no hallucinations while CUDA build has them. This is for version 1.6.2. I've personally given up on large-v3, (seems to run fine through python). But I'm wondering if there was a bug introduced in new versions. Additional details CUDA build made with cuda tools 12.5, current drivers 550. Metal build don't on well.... macOS 14.5 with latest Xcode updates.
from whisper.cpp.
MathiasSchindler
commented on June 24, 2024
This is overlapping with the behaviour described in issue 1949 (also using CUDA, btw)
#1949
from whisper.cpp.
ggerganov
commented on June 24, 2024
It's not impossible that there is a bug, but atm we don't have a good way to verify this. Ideally, we should have implemented a procedure where we measure the WER on a large set of audio and we would make sure after each change that the WER does not increase. But this is still not implemented
The only difference IIRC in v3 is that the vocab size is slightly larger. So I guess it could be possible that some of the CUDA kernels does not handle that specific kernel size well
But I'm more leaning into the possibility that v3 is simply broken and no conclusions can be made based on the results with this model - even if it appeared to work better in the past
from whisper.cpp.
WilliamTambellini
commented on June 24, 2024
Neither
https://github.com/openai/whisper
nor HugginFace whisper hallucinate using the same model (v3 large) and same input.
So likely a bug in the library?
Whatever
c7dc37f
also hallucinates.
Backtracking ...
from whisper.cpp.
WilliamTambellini
commented on June 24, 2024
8e409d1
does not hallucinates.
from whisper.cpp.
WilliamTambellini
commented on June 24, 2024
ok hallucinations started from
2852e1a
from @josharian
from whisper.cpp.
josharian
commented on June 24, 2024
I find it hard to see how those could be related. Not saying you're wrong, just that it is prima facie pretty implausible. If you revert that change at head, does the behavior you're seeing change?
from whisper.cpp.
ggerganov
commented on June 24, 2024
@WilliamTambellini Please also check the examples from the original PR when we added v3 support: #1444
Back then, I tested the reference implementation provided by OpenAI with the samples that we have been using in whisper.cpp since the beginning of the project and the results are suspicious: repetitions + invalid characters. If these are still present, then I don't think we can do anything to fix this on whisper.cpp side
from whisper.cpp.
WilliamTambellini
commented on June 24, 2024
Tks @ggerganov
no hallucination with
#1444
(basically 1.4.3): the output is the same than the original openAI whisper impl, and the transcription is almost perfect.
No hallucination in whispercpp 1.5.4.
Hallucinations started from 1.5.5 and later, meaning unusable for production purpose.
Here it the file:
https://file.io/7UWYV9njmAn3
hallucinations start from after about 2mn of audio.
from whisper.cpp.
magnacartatron
commented on June 24, 2024
@WilliamTambellini if one uses even the built in samples, file hp0.wav its very clear immediately how the hallucinations and repetitions start. It's a small 4 minute audio file and quick to transcribe.
from whisper.cpp.
ggerganov
commented on June 24, 2024
Neither openai/whisper nor HugginFace whisper hallucinate using the same model (v3 large) and same input.
Not sure what you tested since you didn't provide any details, so I just spend the time and checked again and the conclusion is exactly the same as I explained in #1444 and in this thread - the reference OpenAI implementation leads to repetitions using v3
$ ▶ whisper ../whisper.cpp/samples/hp0.wav --model large
/opt/homebrew/lib/python3.11/site-packages/whisper/transcribe.py:115: UserWarning: FP16 is not supported on CPU; using FP32 instead
warnings.warn("FP16 is not supported on CPU; using FP32 instead")
Detecting language using up to the first 30 seconds. Use `--language` to specify the language
Detected language: English
[00:00.000 --> 00:11.840] Henry F. Phillips, from Wikipedia, the free encyclopedia, at en.wikipedia.org
[00:11.840 --> 00:26.140] Henry F. Phillips, 1890-1958
[00:26.140 --> 00:38.160] A U.S. businessman from Portland, Oregon, has the honor of having the Phillips head screw and screwdriver named after him.
[00:39.280 --> 00:52.120] The importance of the cross-head screw design lies in its self-centering property, useful on automated production lines that use powered screwdrivers.
[00:53.760 --> 00:56.120] Phillips' major contribution was in...
[00:56.140 --> 01:04.640] driving the cross-head concept forward, to the point where it was adopted by screwmakers and automobile companies.
[01:05.580 --> 01:10.380] Although he received patents for the design in 1936,
[01:10.380 --> 01:17.720] U.S. Patent No. 2,046,343
[01:17.720 --> 01:24.100] U.S. Patents 2,046,837
[01:24.100 --> 01:25.380] to 2,046,837
[01:26.140 --> 01:29.140] to 2,046,847
[01:30.540 --> 01:37.120] It was so widely copied, that by 1949 Phillips lost his patent.
[01:38.280 --> 01:43.680] The American Screw Company was responsible for devising a means of manufacturing the screw,
[01:43.680 --> 01:49.280] and successfully patented and licensed their method.
[01:51.040 --> 01:55.600] Other screw makers of the 1930s dismissed the Phillips concept,
[01:56.140 --> 02:03.640] since it calls for a relatively complex recessed socket shape in the head of the screw, as
[02:03.640 --> 02:09.600] distinct from the simple milled slot of a slotted-type screw.
[02:09.600 --> 02:17.020] The Phillips Screw Company and the American Screw Company went on to devise the posidrive
[02:17.020 --> 02:23.740] screw, which differs from the Phillips in that it is designed to accommodate greater
[02:23.740 --> 02:27.300] torque than the Phillips.
[02:27.300 --> 02:35.540] An image accompanied this article, captioned, Phillips Screw Head.
[02:35.540 --> 02:43.800] The following is an infobox which accompanies this article, infobox, part of the series
[02:43.800 --> 02:52.740] on screw drive types, slotted, commonly, erroneously, flathead.
[02:52.740 --> 02:53.740] Phillips.
[02:53.740 --> 02:54.740] Crosshead.
[02:54.740 --> 02:55.740] Posidrive.
[02:55.740 --> 02:56.740] Superdrive.
[02:56.740 --> 02:57.740] Torx.
[02:57.740 --> 02:58.740] Hex.
[02:58.740 --> 02:59.740] Allen.
[02:59.740 --> 03:00.740] Robertson.
[03:00.740 --> 03:01.740] Tri-wing.
[03:01.740 --> 03:02.740] Torxset.
[03:02.740 --> 03:03.740] Spannerhead.
[03:03.740 --> 03:04.740] Triple square.
[03:04.740 --> 03:05.740] XZN.
[03:05.740 --> 03:06.740] Others are crosshead.
[03:06.740 --> 03:07.740] Crosshead.
[03:07.740 --> 03:08.740] Spannerhead.
[03:08.740 --> 03:09.740] Crosshead.
[03:09.740 --> 03:10.740] Crosshead.
[03:10.740 --> 03:11.740] Torxset.
[03:11.740 --> 03:12.740] Spannerhead.
[03:12.740 --> 03:13.740] Torxset.
[03:13.740 --> 03:14.740] Torxset.
[03:14.740 --> 03:15.740] Spannerhead.
[03:15.740 --> 03:16.740] Torxset.
[03:16.740 --> 03:17.740] Torxset.
[03:17.740 --> 03:18.740] Torxset.
[03:18.740 --> 03:19.740] Torxset.
[03:19.740 --> 03:20.740] Torxset.
[03:20.740 --> 03:21.740] Torxset.
[03:21.740 --> 03:22.740] Torxset.
[03:22.740 --> 03:23.740] Torxset.
[03:23.740 --> 03:33.940] polydrive, spline drive, double hex. Many images accompanied this info box. This
[03:33.940 --> 03:44.980] page was last modified on the 9th of April 2008 at 1704. All text is available
[03:44.980 --> 03:51.240] under the terms of the GNU free documentation license. See copyrights for
[03:51.240 --> 03:57.920] details. Wikipedia is a registered trademark of the Wikimedia Foundation
[03:57.920 --> 04:08.200] Incorporated, a US registered 501c3 tax-deductible nonprofit charity. This
[04:08.200 --> 04:14.100] sound file and all text in the article are licensed under the GNU free
[04:14.100 --> 04:19.320] documentation license available at
[04:19.320 --> 04:20.320] www.gnu.org.
[04:21.240 --> 04:31.180] www.gnu.org slash copyleft slash fdl dot htmlHere is v2 for reference:
ggerganov ▶ gg-studio ▶ SSH ▶ ~/development/github/whisper ▶
11:58:03 ▶ main ▶ 5? ▶ $ ▶ whisper ../whisper.cpp/samples/hp0.wav --model large-v2
/opt/homebrew/lib/python3.11/site-packages/whisper/transcribe.py:115: UserWarning: FP16 is not supported on CPU; using FP32 instead
warnings.warn("FP16 is not supported on CPU; using FP32 instead")
Detecting language using up to the first 30 seconds. Use `--language` to specify the language
Detected language: English
[00:00.000 --> 00:13.520] Henry F. Phillips, from Wikipedia, The Free Encyclopedia, at en.wikipedia.org.
[00:13.520 --> 00:21.360] Henry F. Phillips, from Wikipedia, The Free Encyclopedia.
[00:21.360 --> 00:33.640] Henry F. Phillips, 1890-1958, a U.S. businessman from Portland, Oregon, has the honor of having
[00:33.640 --> 00:40.280] the Phillips head screw and screwdriver named after him.
[00:40.280 --> 00:47.640] The importance of the crosshead screw design lies in its self-centering property, useful
[00:47.640 --> 00:54.040] on automated production lines that use powered screwdrivers.
[00:54.040 --> 01:00.280] Phillips' major contribution was in driving the crosshead concept forward to the point
[01:00.280 --> 01:06.240] where it was adopted by screwmakers and automobile companies.
[01:06.240 --> 01:19.520] Although he received patents for the design in 1936, U.S. Patent No. 2,046,343, U.S.
[01:19.520 --> 01:35.800] Patents 2,046,837 to 2,046,840, it was so widely copied that by 1949 Phillips lost his
[01:35.800 --> 01:38.280] patent.
[01:38.280 --> 01:44.360] The American Screw Company was responsible for devising a means of manufacturing the
[01:44.360 --> 01:51.200] screw, and successfully patented and licensed their method.
[01:51.200 --> 01:57.720] Other screwmakers of the 1930s dismissed the Phillips concept since it calls for a relatively
[01:57.720 --> 02:05.040] complex recessed socket shape in the head of the screw, as distinct from the simple
[02:05.040 --> 02:09.600] milled slot of a slotted-type screw.
[02:09.600 --> 02:17.040] The Phillips Screw Company and the American Screw Company went on to devise the posidrive
[02:17.040 --> 02:23.840] screw, which differs from the Phillips in that it is designed to accommodate greater
[02:23.840 --> 02:27.280] torque than the Phillips.
[02:27.280 --> 02:35.520] An image accompanied this article, captioned, Phillips Screw Head.
[02:35.520 --> 02:40.440] The following is an infobox which accompanies this article.
[02:40.440 --> 02:46.880] Infobox, part of the series on screw drive types.
[02:46.880 --> 02:53.800] Slotted, commonly, erroneously, flathead.
[02:53.800 --> 02:58.000] Phillips, crosshead.
[02:58.000 --> 03:02.000] Posidrive, supadrive.
[03:02.000 --> 03:04.000] Torx.
[03:04.480 --> 03:06.480] Hex, Allen.
[03:06.480 --> 03:08.480] Robertson.
[03:08.480 --> 03:10.480] Tri-wing.
[03:10.480 --> 03:12.480] Torxset.
[03:12.480 --> 03:14.480] Spannerhead.
[03:14.480 --> 03:16.480] Triple-square, XZN.
[03:16.480 --> 03:28.480] Others, polydrive, splinedrive, double-hex.
[03:28.480 --> 03:32.480] Many images accompanied this infobox.
[03:32.960 --> 03:42.960] This page was last modified on the 9th of April, 2008, at 1704.
[03:42.960 --> 03:48.960] All text is available under the terms of the GNU Free Documentation License.
[03:48.960 --> 03:52.960] See copyrights for details.
[03:52.960 --> 04:00.960] Wikipedia is a registered trademark of the Wikimedia Foundation, Inc., a U.S. registered
[04:01.440 --> 04:07.440] 501c3 tax-deductible non-profit charity.
[04:07.440 --> 04:15.440] This sound file and all text in the article are licensed under the GNU Free Documentation License,
[04:15.920 --> 04:23.920] available at www.gnu.org
[04:23.920 --> 04:31.920] slash copyleft slash fdl dot html.So using the same argument as you do, I can say that the OpenAI implementation has a bug. Or what is more likely - the v3 model is simply broken
from whisper.cpp.
WilliamTambellini
commented on June 24, 2024
Tks @ggerganov
Good catch: indeed the v2 model does nt seem to hallucinate on my audio file.
That being said the quality of v2 is much lower than v3:
so using v2 is not a solution neither.
The facts are still that:
- the original whisper project does not hallucinate on these examples using the v3 model
- whispercpp 1.5.4 (and previous) dont hallucinate on these audio with the v3 model
Would it be a bug in the ggml export of v3?
Best
from whisper.cpp.
Related Issues (20)
- Nonsense Output At times in spite of silence
- CUBLAS CMAKE issue
- WHISPER_CUDA compile error HOT 1
- [Question] Does Whisper.cpp support Flash Attention V2?
- Loading Whisperkit model via dropdown (top right) does not give warning about initial model activation HOT 1
- Error when initializing context from Java
- Transcription issue with audio files longer than 10 minutes when CUDA is enabled - Linux Ubuntu 20.04
- Cannot compile for HIPBLAS (whisper_mel_calc_create_cuda is undefined) HOT 2
- Segmentation fault with CUDA on commit (#2206) HOT 9
- take more time if using quantized model on Apple Metal HOT 2
- App crashes after live transcribing for awhile
- error with strean HOT 1
- No rule to make target 'stream'. Stop. HOT 1
- Cannot compile for HIPBLAS ( undefined reference due to --no-allow-shlib-undefined: void ggml_cuda_flash_attn_ext_wmma_f16_case )
- running example use too long time in linux
- Stream: Error during binary build
- CUDA compile falure, error: user-defined literal operator not found, type name not allowed, and undefines.
- run1
- What backend to use?
- Quantizing (my way).
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from whisper.cpp.