Unsatisfying Proxy Benchmark Results

[yushaolong7]

hi,
i have done benchmark between janus and krakend (https://github.com/devopsfaith/krakend).
the result show me that janus worked not very satisfied. i saw janus source code. the elegant code! but i can't find why it is like that. And how to improve janus performance? it still puzzles me now.

below is the way i worked:

1. command:
   i used ab and wrk tools done this job, detail commad list:

#wrk command
wrk -t 10 -c 200 -d 30s --latency  *resoure-url*  
#ab command
ab -n 10000 -c 200  *resoure-url*

2. mechine hardware configuration:

• gateway-machine : 4cpu/8G
• server-machine: 4cpu/8G

3.benchmark result:

	without-proxy	janus-gateway	krakend-gateway
ab	timeused:25ms, qps:7800	timeused:79ms, qps:2900	timeused:37ms, qps:5300
wrk	timeused:15ms, qps:13400	timeused:71ms, qps:2800	timeused:38ms, qps:5300

note: timeused mean per request timeused.

[s4nji]

Aggregating issue to #362.

[swanandt]

Hi S4nji

I saw the issue of bench marking comparison with Krakend issue is closed which is asking for pointers as why Janus is not as optimal as Krakend.

So is this now addressed using some feature or some optimization are already done in some release as part of this comparison in Janus source code.

[s4nji]

Hi @swanandt!

Looks like I misunderstood the issue as request for more benchmarks and aggregated it to #362.
I just ran a quick benchmark on my local machine with the api gateways acting as simple proxy and I'm still getting similar results:

Setup	Requests per Second
No Gateway	19997.7514 rps
Janus	4411.9717 rps
KrakenD	8263.6511 rps

I'm re-opening the issue as question to why janus has lower performance numbers and what improvements can be done.

[swanandt]

Thanks for re-opening the issue.

Out of curiosity I was doing quick code check and krakend is not doing something very drastically different in terms of logging ( also using logrus ) and other part but as you pointed out could be just metrics.

One quick code check option can be to "comment metrics specific code "init" of it and see if the numbers are comparable that will prove the point of metrics enablement is adding to it and further one config can be added to disable it and then also adding hint that enabling / disabling of metrics get you how much % optimization.

Just while writing this comment , I got one more questions , is there audit logging support in Janus ??? I know during performance discussion not a valid question when metrics is planned to be configurable :-)

Br
Swanand

[swanandt]

Hi

BTW not able to open
https://gist.github.com/s4nji/06464d394f8e4f2ddb4a7bdd88abd71a#file-result-md

you have shared earlier ...!!!

[swanandt]

Hi

Any update as how I can get the result file you shared and if you can share the way you test we can also try to test it

Thanks
Swanand

[s4nji]

Hey @swanandt,

I ran the benchmark again, and you can find the setup details here:
https://gist.github.com/s4nji/66ea4748e53544b9f3363c49621fba9a#file-result-md

Summary:

Setup	Requests per Second
No Gateway	21103.2602 rps
Janus	5451.2701 rps
Janus with statsd	5563.9192 rps
Janus with Prometheus	4781.4349 rps
KrakenD	7476.4710 rps
KrakenD with Prometheus	6326.9421 rps

[Leonardo-Ferreira]

interesting numbers, I have some credits laying around and I could use them to setup a more "controlled" test environment on Azure... would you be willing to work together so we can setup a baseline for reference? I would you need you to setup a delivery pipeline, pref based on IaaS cause containerized apps (AKS and etc) might mask some issues... The idea is not to benchmark Janus against others, but against its previous/stable version to verify if how our changes are impacting overall performance