v1.6.0 #2155

ggerganov · 2024-05-15T07:13:56Z

ggerganov
May 15, 2024
Maintainer

Overview

Can optionally enable Flash Attention for faster processing on CUDA and Metal devices (whisper : use flash attention #2152)
Faster ppc64 performance (40aeeee) (not tested)
Fix main slowdown bug (main : pass nullptr when regex is empty #2070)

Shoutout to @JohannesGaessler for contributing efficient FA CUDA kernels

Some performance numbers for this release:

M1 Pro

CPU	Config	Model	Th	Enc.	Dec.	Bch5	PP	Commit
M1 Pro	METAL	tiny	1	39.21	1.74	0.61	0.04	`22c96b4`
M1 Pro	METAL	base	1	70.76	2.60	0.93	0.06	`22c96b4`
M1 Pro	METAL	small	1	217.28	6.42	2.14	0.17	`22c96b4`
M1 Pro	METAL	medium	1	596.74	14.43	4.75	0.45	`22c96b4`

CPU	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
M1 Pro	METAL	tiny	1	1	30.77	1.59	0.54	0.03	`22c96b4`
M1 Pro	METAL	base	1	1	60.42	2.29	0.81	0.05	`22c96b4`
M1 Pro	METAL	small	1	1	183.82	5.12	1.81	0.14	`22c96b4`
M1 Pro	METAL	medium	1	1	517.92	11.60	4.01	0.38	`22c96b4`

M2 Ultra

CPU	Config	Model	Th	Enc.	Dec.	Bch5	PP	Commit
M2 ULTRA	METAL	tiny	1	12.32	1.35	0.49	0.01	`22c96b4`
M2 ULTRA	METAL	tiny-q5_0	1	11.65	1.30	0.51	0.01	`22c96b4`
M2 ULTRA	METAL	tiny-q5_1	1	12.08	1.30	0.51	0.01	`22c96b4`
M2 ULTRA	METAL	base	1	17.58	1.90	0.76	0.02	`22c96b4`
M2 ULTRA	METAL	base-q5_0	1	18.89	1.86	0.79	0.02	`22c96b4`
M2 ULTRA	METAL	base-q5_1	1	20.69	1.88	0.79	0.02	`22c96b4`
M2 ULTRA	METAL	small	1	49.32	3.85	1.71	0.05	`22c96b4`
M2 ULTRA	METAL	small-q5_0	1	54.91	3.81	1.82	0.06	`22c96b4`
M2 ULTRA	METAL	small-q5_1	1	54.92	3.81	1.79	0.06	`22c96b4`
M2 ULTRA	METAL	medium	1	134.34	8.04	3.82	0.13	`22c96b4`
M2 ULTRA	METAL	medium-q5_0	1	151.68	7.59	4.07	0.14	`22c96b4`
M2 ULTRA	METAL	medium-q5_1	1	151.58	7.67	4.07	0.14	`22c96b4`
M2 ULTRA	METAL	medium-dis	1	120.82	1.07	0.41	0.02	`22c96b4`
M2 ULTRA	METAL	large-v2	1	235.63	12.27	5.85	0.22	`22c96b4`
M2 ULTRA	METAL	large-v2-q5_0	1	273.38	11.17	6.40	0.26	`22c96b4`
M2 ULTRA	METAL	large-v2-q5_1	1	272.44	11.32	6.29	0.26	`22c96b4`
M2 ULTRA	METAL	large-v2-dis	1	212.51	1.20	0.47	0.02	`22c96b4`

CPU	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
M2 ULTRA	METAL	tiny	1	1	9.07	1.33	0.45	0.01	`22c96b4`
M2 ULTRA	METAL	tiny-q5_0	1	1	9.74	1.33	0.47	0.01	`22c96b4`
M2 ULTRA	METAL	tiny-q5_1	1	1	8.93	1.31	0.46	0.01	`22c96b4`
M2 ULTRA	METAL	base	1	1	15.75	1.87	0.71	0.02	`22c96b4`
M2 ULTRA	METAL	base-q5_0	1	1	17.04	1.83	0.74	0.02	`22c96b4`
M2 ULTRA	METAL	base-q5_1	1	1	17.17	1.83	0.74	0.02	`22c96b4`
M2 ULTRA	METAL	small	1	1	42.33	3.64	1.60	0.05	`22c96b4`
M2 ULTRA	METAL	small-q5_0	1	1	47.61	3.63	1.70	0.05	`22c96b4`
M2 ULTRA	METAL	small-q5_1	1	1	47.70	3.66	1.68	0.05	`22c96b4`
M2 ULTRA	METAL	medium	1	1	114.42	7.53	3.55	0.11	`22c96b4`
M2 ULTRA	METAL	medium-q5_0	1	1	132.63	7.02	3.77	0.13	`22c96b4`
M2 ULTRA	METAL	medium-q5_1	1	1	132.28	7.10	3.76	0.13	`22c96b4`
M2 ULTRA	METAL	medium-dis	1	1	102.34	1.01	0.42	0.01	`22c96b4`
M2 ULTRA	METAL	large-v2	1	1	203.01	11.03	5.45	0.20	`22c96b4`
M2 ULTRA	METAL	large-v2-q5_0	1	1	240.05	10.18	5.98	0.23	`22c96b4`
M2 ULTRA	METAL	large-v2-q5_1	1	1	239.22	10.23	5.87	0.23	`22c96b4`
M2 ULTRA	METAL	large-v2-dis	1	1	181.14	1.14	0.48	0.02	`22c96b4`

Ryzen 9 5950X + RTX 2060

CPU	Config	Model	Th	Enc.	Dec.	Bch5	PP	Commit
Ryzen 9 5950X	AVX2	tiny	8	195.29	1.57	0.51	0.26	`22c96b4`
Ryzen 9 5950X	AVX2	tiny-q5_0	8	213.33	1.10	0.50	0.30	`22c96b4`
Ryzen 9 5950X	AVX2	tiny-q5_1	8	219.38	1.18	0.53	0.32	`22c96b4`
Ryzen 9 5950X	AVX2	base	8	424.85	3.71	1.03	0.46	`22c96b4`
Ryzen 9 5950X	AVX2	base-q5_0	8	473.61	1.81	0.82	0.52	`22c96b4`
Ryzen 9 5950X	AVX2	base-q5_1	8	484.14	1.92	0.85	0.56	`22c96b4`
Ryzen 9 5950X	AVX2	small	8	1458.32	12.66	3.09	1.26	`22c96b4`
Ryzen 9 5950X	AVX2	small-q5_0	8	1673.22	6.42	2.18	1.45	`22c96b4`
Ryzen 9 5950X	AVX2	small-q5_1	8	1724.78	6.72	2.32	1.52	`22c96b4`
Ryzen 9 5950X	AVX2	medium	8	4333.87	36.80	8.56	3.37	`22c96b4`
Ryzen 9 5950X	AVX2	medium-q5_0	8	5194.09	19.21	5.71	3.97	`22c96b4`
Ryzen 9 5950X	AVX2	medium-q5_1	8	5450.39	20.01	5.99	4.17	`22c96b4`
Ryzen 9 5950X	AVX2	medium-dis	8	3995.19	5.08	1.21	0.55	`22c96b4`
Ryzen 9 5950X	AVX2	large-v2	8	8056.16	69.74	16.11	6.13	`22c96b4`
Ryzen 9 5950X	AVX2	large-v2-q5_0	8	9799.58	35.16	10.49	7.28	`22c96b4`
Ryzen 9 5950X	AVX2	large-v2-q5_1	8	ms	36.74	11.02	7.65	`22c96b4`
Ryzen 9 5950X	AVX2	large-v2-dis	8	7490.03	7.40	1.70	0.72	`22c96b4`

GPU	Config	Model	Th	Enc.	Dec.	Bch5	PP	Commit
RTX 2060	AVX2 CUDA	tiny	8	12.54	0.93	0.29	0.02	`22c96b4`
RTX 2060	AVX2 CUDA	tiny-q5_0	8	12.73	0.98	0.24	0.02	`22c96b4`
RTX 2060	AVX2 CUDA	tiny-q5_1	8	12.72	0.99	0.24	0.02	`22c96b4`
RTX 2060	AVX2 CUDA	base	8	24.14	1.28	0.41	0.03	`22c96b4`
RTX 2060	AVX2 CUDA	base-q5_0	8	24.58	1.38	0.35	0.03	`22c96b4`
RTX 2060	AVX2 CUDA	base-q5_1	8	24.58	1.37	0.35	0.03	`22c96b4`
RTX 2060	AVX2 CUDA	small	8	74.70	2.91	0.84	0.07	`22c96b4`
RTX 2060	AVX2 CUDA	small-q5_0	8	76.12	2.84	0.77	0.08	`22c96b4`
RTX 2060	AVX2 CUDA	small-q5_1	8	76.14	2.84	0.76	0.08	`22c96b4`
RTX 2060	AVX2 CUDA	medium	8	200.69	6.46	1.83	0.17	`22c96b4`
RTX 2060	AVX2 CUDA	medium-q5_0	8	204.80	5.90	1.65	0.19	`22c96b4`
RTX 2060	AVX2 CUDA	medium-q5_1	8	205.61	5.85	1.61	0.19	`22c96b4`
RTX 2060	AVX2 CUDA	medium-dis	8	186.17	0.86	0.24	0.02	`22c96b4`
RTX 2060	AVX2 CUDA	large-v2	8	347.22	10.36	2.82	0.29	`22c96b4`
RTX 2060	AVX2 CUDA	large-v2-q5_0	8	357.06	8.81	2.58	0.34	`22c96b4`
RTX 2060	AVX2 CUDA	large-v2-q5_1	8	356.97	8.62	2.49	0.33	`22c96b4`
RTX 2060	AVX2 CUDA	large-v2-dis	8	318.05	1.03	0.34	0.04	`22c96b4`

GPU	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
RTX 2060	AVX2 CUDA	tiny	8	1	7.21	0.76	0.29	0.02	`22c96b4`
RTX 2060	AVX2 CUDA	tiny-q5_0	8	1	7.42	0.82	0.18	0.02	`22c96b4`
RTX 2060	AVX2 CUDA	tiny-q5_1	8	1	7.38	0.82	0.18	0.02	`22c96b4`
RTX 2060	AVX2 CUDA	base	8	1	13.49	1.04	0.36	0.02	`22c96b4`
RTX 2060	AVX2 CUDA	base-q5_0	8	1	13.94	1.13	0.26	0.03	`22c96b4`
RTX 2060	AVX2 CUDA	base-q5_1	8	1	13.94	1.14	0.26	0.03	`22c96b4`
RTX 2060	AVX2 CUDA	small	8	1	42.81	2.33	0.69	0.05	`22c96b4`
RTX 2060	AVX2 CUDA	small-q5_0	8	1	44.43	2.25	0.59	0.06	`22c96b4`
RTX 2060	AVX2 CUDA	small-q5_1	8	1	44.11	2.24	0.58	0.06	`22c96b4`
RTX 2060	AVX2 CUDA	medium	8	1	115.47	5.17	1.45	0.11	`22c96b4`
RTX 2060	AVX2 CUDA	medium-q5_0	8	1	120.37	4.63	1.25	0.13	`22c96b4`
RTX 2060	AVX2 CUDA	medium-q5_1	8	1	120.28	4.55	1.21	0.13	`22c96b4`
RTX 2060	AVX2 CUDA	medium-dis	8	1	101.69	0.75	0.20	0.02	`22c96b4`
RTX 2060	AVX2 CUDA	large-v2	8	1	205.67	8.49	2.19	0.18	`22c96b4`
RTX 2060	AVX2 CUDA	large-v2-q5_0	8	1	214.07	6.88	1.94	0.22	`22c96b4`
RTX 2060	AVX2 CUDA	large-v2-q5_1	8	1	213.98	6.70	1.86	0.22	`22c96b4`
RTX 2060	AVX2 CUDA	large-v2-dis	8	1	176.71	0.91	0.31	0.03	`22c96b4`

V100

GPU	Config	Model	Th	Enc.	Dec.	Bch5	PP	Commit
V100	AVX2 CUDA	tiny	1	6.21	1.11	0.30	0.02	`22c96b4`
V100	AVX2 CUDA	tiny-q5_1	1	5.97	1.10	0.26	0.02	`22c96b4`
V100	AVX2 CUDA	base	1	10.95	1.47	0.42	0.03	`22c96b4`
V100	AVX2 CUDA	base-q5_1	1	11.13	1.53	0.36	0.03	`22c96b4`
V100	AVX2 CUDA	small	1	31.57	2.96	0.84	0.05	`22c96b4`
V100	AVX2 CUDA	small-q5_1	1	32.19	3.14	0.75	0.05	`22c96b4`
V100	AVX2 CUDA	medium	1	85.88	6.49	1.80	0.10	`22c96b4`
V100	AVX2 CUDA	medium-q5_0	1	87.53	5.82	1.37	0.10	`22c96b4`
V100	AVX2 CUDA	large-v2	1	142.23	8.92	2.62	0.15	`22c96b4`

GPU	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
V100	AVX2 CUDA	tiny	1	1	3.96	0.82	0.24	0.02	`22c96b4`
V100	AVX2 CUDA	tiny-q5_1	1	1	4.05	0.85	0.18	0.02	`22c96b4`
V100	AVX2 CUDA	base	1	1	7.21	1.16	0.36	0.02	`22c96b4`
V100	AVX2 CUDA	base-q5_1	1	1	7.39	1.21	0.26	0.02	`22c96b4`
V100	AVX2 CUDA	small	1	1	19.81	2.41	0.71	0.04	`22c96b4`
V100	AVX2 CUDA	small-q5_1	1	1	20.50	2.31	0.51	0.04	`22c96b4`
V100	AVX2 CUDA	medium	1	1	56.02	4.89	1.44	0.07	`22c96b4`
V100	AVX2 CUDA	medium-q5_0	1	1	57.85	4.73	1.09	0.08	`22c96b4`
V100	AVX2 CUDA	large-v2	1	1	92.73	7.18	2.14	0.10	`22c96b4`

For reference, here is the performance for v1.5.0

What's Changed

Added up-to-date repository for Python bindings to README by @AIWintermuteAI in Added up-to-date repository for Python bindings to README #2063
main : pass nullptr when regex is empty by @ggerganov in main : pass nullptr when regex is empty #2070
Fix for fft sizes in whisper.cpp by @goldwaving in Fix for fft sizes in whisper.cpp #2080
make : change GNU make default CXX from g++ to c++ by @przemoc in make : change GNU make default CXX from g++ to c++ #2100
node : fix compilation by @pprobst in node : fix compilation #2115
minor: add CMakeSettings.json to gitignore by @iboB in minor: add CMakeSettings.json to gitignore #2094
build : improve disabling AVX-512 by @przemoc in build : improve disabling AVX-512 #2129
ggml : expose SSE3 and SSSE3 for MSVC when AVX is available by @przemoc in ggml : expose SSE3 and SSSE3 for MSVC when AVX is available #2128
sync : ggml by @ggerganov in sync : ggml #2142
main : fix double quote escaping in csv output by @mashizora in main : fix double quote escaping in csv output #2090
fix(compile): fix josn_cpp, INTERFACE library requires no source arguments by @xingchensong in fix(compile): fix josn_cpp, INTERFACE library requires no source arguments #2069
whisper.android: update example, add field to print timestamp by @codezjx in whisper.android: update example, add field to print timestamp #2072
main: add options for temperature control by @ziegenberg in main: add options for temperature control #2088
main: dont print timings with --no-prints by @ziegenberg in main: dont print timings with --no-prints #2108
Remove un-needed request header from fetchRemote by @MagicalTux in Remove un-needed request header from fetchRemote #2119
Add additional params to addon.node by @valVk in Add additional params to addon.node #2000
Added CMake changes for building HIP/ROCm CUDA for AMD GPUs. by @aldorof in Added CMake changes for building HIP/ROCm CUDA for AMD GPUs. #2102
node : add audio_ctx and audio buffer params by @pprobst in node : add audio_ctx and audio buffer params #2123
fix: model path encoding in windows by @thewh1teagle in fix: model path encoding in windows #2086
Extended talk-llama example to reject runs without required arguments. by @petterreinholdtsen in Extended talk-llama example to reject runs without required arguments. #2153
whisper : use flash attention by @ggerganov in whisper : use flash attention #2152

New Contributors

@AIWintermuteAI made their first contribution in Added up-to-date repository for Python bindings to README #2063
@goldwaving made their first contribution in Fix for fft sizes in whisper.cpp #2080
@mashizora made their first contribution in main : fix double quote escaping in csv output #2090
@xingchensong made their first contribution in fix(compile): fix josn_cpp, INTERFACE library requires no source arguments #2069
@codezjx made their first contribution in whisper.android: update example, add field to print timestamp #2072
@ziegenberg made their first contribution in main: add options for temperature control #2088
@MagicalTux made their first contribution in Remove un-needed request header from fetchRemote #2119
@valVk made their first contribution in Add additional params to addon.node #2000
@aldorof made their first contribution in Added CMake changes for building HIP/ROCm CUDA for AMD GPUs. #2102
@thewh1teagle made their first contribution in fix: model path encoding in windows #2086
@petterreinholdtsen made their first contribution in Extended talk-llama example to reject runs without required arguments. #2153

Full Changelog: v1.5.5...v1.6.0

This discussion was created from the release v1.6.0.

Binaries

https://github.com/ggerganov/whisper.cpp/actions/runs/9091347125

ggerganov · 2024-05-15T07:15:02Z

ggerganov
May 15, 2024
Maintainer Author

Windows binaries will be attached when the CI is ready

2 replies

niksedk May 17, 2024

Could the binary files be attached as Assets in the release?
The CI artifacts are harder to download and are not available as a permanent url.

ggerganov May 19, 2024
Maintainer Author

Done

Sing303 · 2024-05-15T10:11:12Z

Sing303
May 15, 2024

At 4090, the acceleration is ~+23%. Nice work

0 replies

thewh1teagle · 2024-05-20T08:36:47Z

thewh1teagle
May 20, 2024

Does macos m2 requires cuda to run fast?
It works very fast on m1 but slow on m2 (I compile with metal enabled)

Also, does cuda have function like is_available() so I can enable on Windows both opencl and cuda and choose cuda only if available? (If opencl is sufficient and effective enough I prefer using it only)

In other words, I'm looking for getting the best results cross platform in single app installer

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v1.6.0 #2155

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

v1.6.0 #2155

ggerganov May 15, 2024 Maintainer

Overview

M1 Pro

M2 Ultra

Ryzen 9 5950X + RTX 2060

V100

What's Changed

New Contributors

Binaries

Replies: 3 comments · 2 replies

ggerganov May 15, 2024 Maintainer Author

niksedk May 17, 2024

ggerganov May 19, 2024 Maintainer Author

Sing303 May 15, 2024

thewh1teagle May 20, 2024

ggerganov
May 15, 2024
Maintainer

Replies: 3 comments 2 replies

ggerganov
May 15, 2024
Maintainer Author

ggerganov May 19, 2024
Maintainer Author

Sing303
May 15, 2024

thewh1teagle
May 20, 2024