Watermill Benchmark

This is an early set of tools for benchmarking watermill.

Warning: This tool is meant to provide a rough estimate on how fast each Pub/Sub can process messages. It uses very simplified infrastructure to set things up and default configurations.

Keep in mind that final performance depends on multiple factors.

How it works

• All tests are run on a single 16 CPU GCloud compute instance (n1-highcpu-16).
• Docker Compose is used to run Pub/Sub infrastructure and benchmark code (except for Google Cloud Pub/Sub).
• The tool will first produce a big number of messages on a generated topic.
• Then it will subscribe to the topic and consume all of the messages.
• Multiple message sizes can be chosen (by default: 16, 64 and 256 bytes).

Results

This is an early version of benchmark results. Expect it to be updated and grow.

Kafka (one node)

Message size (bytes)	Publish (messages / s)	Subscribe (messages / s)
16	44,090	108.285
64	41,371	108,848
256	41,497	111,756

NATS Jetstream (16 Subscribers)

Message size (bytes)	Publish (messages / s)	Subscribe (messages / s)	Subscribe (messages / s - async ack)
16	49,255	33,009	63,065
64	49,296	33,016	62,667
256	48.488	32,573	62,745

NATS Jetstream (48 Subscribers)

Message size (bytes)	Publish (messages / s)	Subscribe (messages / s)	Subscribe (messages / s - async ack)
16	48,882	45,275	90,917
64	48,681	44,746	89,527
256	48,097	44,487	90,510

Redis

Message size (bytes)	Publish (messages / s)	Subscribe (messages / s)
16	61,642	11,213
64	58,554	11,252
256	58,906	11,408

SQL (MySQL)

Message size (bytes)	Publish (messages / s)	Subscribe (messages / s)
16	5,599	167
64	5,625	168
256	5,381	164

SQL (PostgreSQL)

Message size (bytes)	Publish (messages / s)	Subscribe (messages / s)
16	3,834	455
64	3,923	468
256	3,855	472

Google Cloud Pub/Sub (16 subscribers)

Message size (bytes)	Publish (messages / s)	Subscribe (messages / s)
16	3,689	30,229
64	3,408	26,448
256	6,967	30,123

AMQP (RabbitMQ, 16 subscribers)

Message size (bytes)	Publish (messages / s)	Subscribe (messages / s)
16	2,702	13,192
64	2,712	12,980
256	2,692	8,027

GoChannel

Message size (bytes)	Publish (messages / s)	Subscribe (messages / s)
16	331,882	118,943
64	298,847	123,499
256	373,053	130,940

VM Setup

The project includes Terraform definition for setting up an instance on Google Cloud Platform.

It will spin up a fresh Ubuntu 19.04 instance, install docker with dependencies and clone this repository.

Set environment variables:

# path to GCP credentials file
TF_VAR_credentials_path=
# project name on GCP
TF_VAR_project=
# public part of the key that you will use to access SSH
TF_VAR_pub_key_path=

Create the VM:

cd setup
terraform apply

The command will output the public IP address of the server. Use ssh with user benchmark to access it.

After running all benchmarks, destroy the VM:

terraform destroy

Configuration

Google Pub/Sub

Set environment variables in compose/.env:

# path to json file within the project with GCP credentials
GOOGLE_APPLICATION_CREDENTIALS=compose/key.json
# project name on GCP
GOOGLE_CLOUD_PROJECT=

Running

Run benchmarks with:

./run.sh <pubsub>