Benchmarking Getting Started

This is the getting started guide for benchmarking.

If you're interested in development please visit the getting started guide for development.

You will need three computers (or virtual machines) networked together to fill the three benchmark roles of database server, load generation server, and web framework server.

Network Security

Make sure to secure the 3 computers / virtual machines to ensure the accuracy of the benchmarking results and your own network security.

• Disable password-based SSH and only use key-based authentication.
• Set up a private network amongst the machines and have them communicate with each other only via the private networking interfaces.
• Set up firewalls on all your machines (on the cloud service or on the machines themselves) to restrict any public access and only open necessary ports (e.g. only SSH port).

Note: If testing a pull request or doing development, it is usually adequate to only use one computer. In that case, your server, client, and database IPs will be 127.0.0.1.

Benchmark Suite Deployment

Required Host Roles

• App (aka web framework) Server
  • Runs the web framework containers
  • Operating system: Ubuntu Server 22.04 LTS 64-bit
• Database Server
  • Runs database containers (e.g. MySQL, PostgreSQL, MongoDB, etc).
  • Operating system: Ubuntu Server 22.04 LTS 64-bit
• Client (aka load generation) Server
  • Runs the benchmark client container
  • Operating system: Ubuntu Server 22.04 LTS 64-bit

Manual Deployment

Prerequisites

Before you get started, the following are the steps you must take on each machine (App, Database, Client) to run the benchmarking suite:

1. Docker installed (you can verify this via docker run hello-world)
2. /lib/systemd/system/docker.service needs ExecStart to have an additional flag: -H [machine's ip]:2375
3. sudo systemctl daemon-reload
4. sudo service docker restart (you can verify this via sudo service docker status and see the newly added -H [machine's ip]:2375 flag)

Since the tests can run for several hours, it helps to set everything up so that once the tests are running, you can leave the machines unattended and don't need to be around to enter ssh or sudo passwords.

Running the Benchmarking Suite

Let us assume make some assumptions for the example:

1. You cloned the repository into the home directory of user techempower
2. Your App machine has the IP 10.0.0.1
3. Your Database machine has the IP 10.0.0.2
4. Your Client machine has the IP 10.0.0.3

You can build the techempower/tfb by running:

docker build --build-arg GROUP_ID=$(id -g) --build-arg USER_ID=$(id -u) -t techempower/tfb .

The following command, given the prerequisites here, would run a benchmark for all the tests in the codebase.

docker run \
  -e USER_ID=$(id -u) \
  --network=host \
  --mount type=bind,source=/home/techempower/FrameworkBenchmarks,target=/FrameworkBenchmarks \
  techempower/tfb \
  --server-host 10.0.0.1 \
  --database-host 10.0.0.2 \
  --client-host 10.0.0.3 \
  --network-mode host \
  --quiet