Benchmarking and scaling up

[bitsapien]

I have one instance of glot-run running. I did a benchmarking using 'Apache ab' with the following parameters:

• Number of Request - 200
• Concurrency - 10
• Print "Hello World!" program in python

These are the results -

This is ApacheBench, Version 2.3 <$Revision: 1748469 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software: Cowboy
Server Hostname: 127.0.0.1
Server Port: 8090

Document Path: /languages/python/latest
Document Length: 42 bytes

Concurrency Level: 10
Time taken for tests: 165.044 seconds
Complete requests: 200
Failed requests: 170
(Connect: 0, Receive: 0, Length: 170, Exceptions: 0)
Non-2xx responses: 170
Total transferred: 29120 bytes
Total body sent: 55800
HTML transferred: 1260 bytes
Requests per second: 1.21 [#/sec] (mean)
Time per request: 8252.200 [ms] (mean)
Time per request: 825.220 [ms] (mean, across all concurrent requests)
Transfer rate: 0.17 [Kbytes/sec] received
0.33 kb/s sent
0.50 kb/s total

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.6 0 7
Processing: 4140 8146 4262.4 5039 18715
Waiting: 4139 8145 4262.3 5038 18715
Total: 4141 8146 4262.4 5039 18716

Percentage of the requests served within a certain time (ms)
50% 5039
66% 9413
75% 11697
80% 12977
90% 15369
95% 16340
98% 17946
99% 18164
100% 18716 (longest request)

That shows a 15% success rate.
Planning to have a coding challenge for my college, and requests will be close to the above parameters. Been thinking about various approaches, one of them being where code run requests are pushed to Redis, while multiple glot-run nodes listen for requests, and process when available.
Confused about how to go about scaling this setup. Looking for suggestions. Thanks.

[prasmussen]

I'm guessing the bottleneck is on the machine where the docker daemon is running and not in glot-run itself. I can think of 3 ways to scale the docker api:

1. Vertical.
   Add more/faster cpu's and faster disks on the machine where the docker daemon is running.

2. Horizontal.
   Set up a load balancer (haproxy/nginx/etc) and multiple machines running the docker daemon and configure DOCKER_API_URL to go to the load balancer.

3. Queue.
   Add a queue in front of the docker daemon, I don't have much experience with this, but it looks like nginx plus and haproxy has support for queueing request when the max connection limit is reached. You must also have to a high DOCKER_RUN_TIMEOUT configured in glot-run in this case. The following haproxy configuration options seems relevant:
   https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-maxconn
   https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.2-maxqueue
   https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4-timeout%20queue

[javierprovecho]

swarm should be the best way to go, it is an official proxy for docker daemon with extended capabilities of orchestrators. You should point DOCKER_API_URL to swarm endpoint and register all your docker daemons with swarm.

more info here:
https://www.docker.com/products/docker-swarm

[prasmussen]

Swarm definitely looks like the way to go 👍

[bitsapien]

Thanks, I'll try doing that.

[rushi216]

@bitsapien any success with docker swarm? I am trying to do same

[bitsapien]

@rushi216 have not tried it.

[sanjayme97]

@bitsapien how did you set up this locally? guide me on this