Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add gzip HTTP handler/client wrapper #383

Merged
merged 14 commits into from
Jun 2, 2021
Merged

Add gzip HTTP handler/client wrapper #383

merged 14 commits into from
Jun 2, 2021

Conversation

klauspost
Copy link
Owner

@klauspost klauspost commented May 31, 2021 •
edited

Fork and clean up+extend the dead nytimes/gziphandler project.

Default settings comparison:

λ benchcmp before.txt after.txt                                        
benchmark                         old ns/op     new ns/op     delta    
BenchmarkGzipHandler_S2k-32       51302         25554         -50.19%  
BenchmarkGzipHandler_S20k-32      301426        174900        -41.98%  
BenchmarkGzipHandler_S100k-32     1546203       912349        -40.99%  
BenchmarkGzipHandler_P2k-32       3973          2116          -46.74%  
BenchmarkGzipHandler_P20k-32      20319         12237         -39.78%  
BenchmarkGzipHandler_P100k-32     96079         57348         -40.31%  
                                                                       
benchmark                         old MB/s     new MB/s     speedup    
BenchmarkGzipHandler_S2k-32       39.92        80.14        2.01x      
BenchmarkGzipHandler_S20k-32      67.94        117.10       1.72x      
BenchmarkGzipHandler_S100k-32     66.23        112.24       1.69x      
BenchmarkGzipHandler_P2k-32       515.44       967.76       1.88x      
BenchmarkGzipHandler_P20k-32      1007.92      1673.55      1.66x      
BenchmarkGzipHandler_P100k-32     1065.79      1785.58      1.68x      
                                                                       
benchmark                         old allocs     new allocs     delta  
BenchmarkGzipHandler_S2k-32       22             19             -13.64%
BenchmarkGzipHandler_S20k-32      25             22             -12.00%
BenchmarkGzipHandler_S100k-32     28             25             -10.71%
BenchmarkGzipHandler_P2k-32       22             19             -13.64%
BenchmarkGzipHandler_P20k-32      25             22             -12.00%
BenchmarkGzipHandler_P100k-32     27             24             -11.11%

Client Transport:

Speed compared to standard library for an approximate 127KB payload:

BenchmarkTransport

Single core:
BenchmarkTransport/gzhttp-32         	    1995	    609791 ns/op	 214.14 MB/s	   10129 B/op	      73 allocs/op
BenchmarkTransport/stdlib-32         	    1567	    772161 ns/op	 169.11 MB/s	   53950 B/op	      99 allocs/op

Multi Core:
BenchmarkTransport/gzhttp-par-32     	   29113	     36802 ns/op	3548.27 MB/s	   11061 B/op	      73 allocs/op
BenchmarkTransport/stdlib-par-32     	   16114	     66442 ns/op	1965.38 MB/s	   54971 B/op	      99 allocs/op

This includes both serving the http request, parsing requests and decompressing.

@klauspost klauspost changed the title Add gzip HTTP wrapper Add gzip HTTP handler/client wrapper Jun 1, 2021
@klauspost klauspost merged commit b4546d4 into master Jun 2, 2021
@klauspost klauspost deleted the add-gzip-wrapper branch June 2, 2021 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Incorrect check for short write Range-Requests aren't properly handled
1 participant
@klauspost