Integrate with fasthttp #500

tsenart · 2020-03-07T15:40:46Z

Fixes #486

AndreiPashkin · 2020-03-07T18:06:52Z

How can we solve this without depending on two http libraries?

What about HTTP/2? FastHTTP currently doesn't support it.

tsenart · 2020-03-07T18:14:04Z

Depending on two HTTP libraries means we’d still support HTTP2, since one of them, net/http, provides it.

AndreiPashkin · 2020-03-08T00:18:57Z

So we have to depend to two libraries if we want to support HTTP/2.

AndreiPashkin · 2020-03-08T16:01:33Z

@tsenart, I don't understand - are we pursuing not using 2 libraries and backwards compatibility of modules APIs or not?

tsenart · 2020-03-08T20:23:24Z

I’m experimenting with using two libraries and breaking backwards compatibility, maybe it’s worth it.

blakewatters · 2020-03-14T12:23:06Z

Honestly the loss of HTTP/2 support issue is all that held me back from going full throttle to push my original experiment forward to something that could be merged (and having no idea if @tsenart would merge it).

It's a bigger lift, but an interface that lets you switch engines feels like the ideal setup -- plus if you get the abstractions right you in theory support arbitrary underlying transports with HTTP semantics (where's that Golang QUIC library at?).

FastHTTP really needs to get HTTP/2 support landed.

But the performance gains are just astronomical compared to net/http. It's easily the difference between running a single node versus going distributed

blakewatters · 2020-03-14T12:31:50Z

attack.go

+	}
+
+	// TODO(tsenart): Add fasthttp support for:
+	//  - opts.redirects


The Do function lets you manually handle redirects https://godoc.org/github.com/valyala/fasthttp#Do

Ideally, I'd like the same behaviour as net/http. I guess I need to do it manually.

attack.go

blakewatters · 2020-03-14T12:55:13Z

attack.go

+	//  - opts.keepalive
+	//  - opts.proxyHeaders
+	//  - HTTP_PROXY
+	//  - opts.localAddr


Proxy and local addr through Dial function? Hitting the edge of my Golang-fu but trying to help: valyala/fasthttp#363

blakewatters · 2020-03-14T13:22:14Z

I was seeing 4x speedups on my hack-job implementation (60k rps -> 240k). Your changes are tighter but it should be faster. FastHTTP's main trick seems to be avoiding memory copies at all costs, maybe there's implicit copies going on?

tsenart · 2020-03-14T20:48:01Z

@blakewatters

It's a bigger lift, but an interface that lets you switch engines feels like the ideal setup

What do you think of the Hitter interface in this PR?

I was seeing 4x speedups on my hack-job implementation (60k rps -> 240k)

Could you check-out this branch and run the benchmarks I wrote today?

go test -run '^$' -bench=Hit ./...

FastHTTP's main trick seems to be avoiding memory copies at all costs, maybe there's implicit copies going on?

Main allocations come from allocating Result to return and copying headers to http.Header.
Did you do anything substantially different there?

tsenart · 2020-03-15T13:20:13Z

@blakewatters: I'm getting better results :-)

$ go test -run='^$' -bench Hit
goos: linux
goarch: amd64
pkg: github.com/tsenart/vegeta/lib
BenchmarkFastHTTPHitter_Hit-16            500396              2391 ns/op            418246 req/s            1300 B/op         21 allocs/op
BenchmarkNetHTTPHitter_Hit-16             116127             18996 ns/op             52643 req/s           10326 B/op         81 allocs/op
PASS
ok      github.com/tsenart/vegeta/lib   3.893s

tsenart · 2020-03-15T13:30:08Z

Check this out!

name           old time/op    new time/op    delta
Hitter_Hit-16   12.5µs ±111%     2.2µs ± 1%   -82.27%  (p=0.000 n=26+27)

name           old req/s      new req/s      delta
Hitter_Hit-16     79.0k ±97%    450.5k ± 1%  +470.33%  (p=0.000 n=30+27)

name           old alloc/op   new alloc/op   delta
Hitter_Hit-16    10.3kB ±13%     1.3kB ± 0%   -87.62%  (p=0.000 n=30+24)

name           old allocs/op  new allocs/op  delta
Hitter_Hit-16      82.4 ± 2%      21.0 ± 0%   -74.53%  (p=0.000 n=27+30)

peterbourgon · 2020-03-15T19:08:54Z

lib/hitters.go

+// TODO: Write CLI integration tests
+
+type Hitter interface {
+	Hit(*Target) *Result


Why pointers and not values? I bet you might see some even substantial perf improvements on long test runs if this were

type Hitter interface { Hit(Target) Result }

Benchmarks seems inconclusive, but slightly worse with values. These aren't very small structs. It follows the same pattern as the Doer interface, which takes in an *http.Request and returns an *http.Response.

Old = pointers
New = values

name old time/op new time/op delta FastHTTPHitter_Hit-12 10.3µs ± 6% 10.7µs ±11% ~ (p=0.133 n=9+10) NetHTTPHitter_Hit-12 1.79ms ± 4% 1.79ms ±15% ~ (p=0.167 n=8+9) name old req/s new req/s delta FastHTTPHitter_Hit-12 95.8k ±12% 93.8k ±10% ~ (p=0.315 n=10+10) NetHTTPHitter_Hit-12 3.46 ±700% 117.33 ±372% ~ (p=0.529 n=8+9) name old alloc/op new alloc/op delta FastHTTPHitter_Hit-12 1.28kB ± 1% 1.28kB ± 1% ~ (p=0.955 n=10+10) NetHTTPHitter_Hit-12 5.45kB ±28% 6.21kB ±50% ~ (p=0.459 n=9+10) name old allocs/op new allocs/op delta FastHTTPHitter_Hit-12 21.0 ± 0% 21.0 ± 0% ~ (all equal) NetHTTPHitter_Hit-12 79.0 ± 0% 80.4 ±11% ~ (p=0.776 n=8+9)

I think you'd have to run the test for long enough to trigger a bunch of GC in order to see meaningful differences. But no big deal in any case.

AndreiPashkin · 2020-03-15T21:05:47Z

@tsenart, If I want to add a commit (with docstrings) to this branch - how can I do that (I'm new to Golang but at least I want to contribute with whatever I can)? 🤔

leosunmo · 2020-12-16T01:28:37Z

lib/hitters.go

+// TODO: Fix attacker tests
+// TODO: Write CLI integration tests
+
+type Hitter interface {


Unrelated to the (awesome) fasthttp addition, this abstraction would make it a lot easier to "re-purpose" Vegeta for SQL/general TCP load testing. A Postgres Hitter could (somewhat) easily be implemented on top of this.

I realise I'd still have to create an interface for the Target struct, but it's at least one step closer.

Some very old prior art that is basically what I'd need: https://github.com/masahide/vegeta-mysql

tsenart mentioned this pull request Mar 7, 2020

Integrate with FastHTTP (massive performance gains) #486

Open

blakewatters reviewed Mar 14, 2020

View reviewed changes

attack.go Outdated Show resolved Hide resolved

blakewatters reviewed Mar 14, 2020

View reviewed changes

tsenart mentioned this pull request Mar 15, 2020

client: Implement DoRedirects valyala/fasthttp#765

Merged

peterbourgon reviewed Mar 15, 2020

View reviewed changes

tsenart and others added 12 commits March 26, 2020 10:43

Use fasthttp library

fc06a06

fixup! Timeout

2bf368d

fixup! Support TLSClientConfig

23a2a0d

Hitters

5a24ecb

wip

d345463

Benchmarks

578eff1

Use DoTimeout

ad5b790

Remove dead code test

41865bc

Optimisations

e3b197b

Tackle a few more TODOs

f84ce97

benchmark with attacker

5a0bd3d

Support redirects

8c9abb5

tsenart force-pushed the fasthttp branch from d342ec1 to 8c9abb5 Compare March 26, 2020 09:47

tsenart added 4 commits March 26, 2020 11:22

Use NetHTTPHitter for proxy support

1e46c0f

Add -skip-body flag

646a273

Remove -connections flag

c3a24f7

New Attack API

9eba1f8

leosunmo reviewed Dec 16, 2020

View reviewed changes

bbros-dev mentioned this pull request Apr 22, 2021

Attempt to address Coordinated Omission: wrk2...? bbros-dev/rust-web-frameworks-benchmark#4

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate with fasthttp #500

Integrate with fasthttp #500

tsenart commented Mar 7, 2020 •

edited

AndreiPashkin commented Mar 7, 2020

tsenart commented Mar 7, 2020

AndreiPashkin commented Mar 8, 2020

AndreiPashkin commented Mar 8, 2020 •

edited

tsenart commented Mar 8, 2020

blakewatters commented Mar 14, 2020

blakewatters Mar 14, 2020

tsenart Mar 14, 2020

blakewatters Mar 14, 2020

blakewatters commented Mar 14, 2020

tsenart commented Mar 14, 2020 •

edited

tsenart commented Mar 15, 2020

tsenart commented Mar 15, 2020

peterbourgon Mar 15, 2020

tsenart Mar 15, 2020

tsenart Mar 15, 2020

peterbourgon Mar 15, 2020

AndreiPashkin commented Mar 15, 2020 •

edited

leosunmo Dec 16, 2020

Integrate with fasthttp #500

Are you sure you want to change the base?

Integrate with fasthttp #500

Conversation

tsenart commented Mar 7, 2020 • edited

AndreiPashkin commented Mar 7, 2020

tsenart commented Mar 7, 2020

AndreiPashkin commented Mar 8, 2020

AndreiPashkin commented Mar 8, 2020 • edited

tsenart commented Mar 8, 2020

blakewatters commented Mar 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blakewatters commented Mar 14, 2020

tsenart commented Mar 14, 2020 • edited

tsenart commented Mar 15, 2020

tsenart commented Mar 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndreiPashkin commented Mar 15, 2020 • edited

Choose a reason for hiding this comment

tsenart commented Mar 7, 2020 •

edited

AndreiPashkin commented Mar 8, 2020 •

edited

tsenart commented Mar 14, 2020 •

edited

AndreiPashkin commented Mar 15, 2020 •

edited