Alternator results

Goals

Evaluate the performance of Alternator, comparing to previous results obtained with https://www.scylladb.com/2018/12/13/scylla-vs-amazon-dynamodb/

The original test compared Scylla Open Source 2.1/Scylla Enterprise 2018.1 vs. Amazon DynamoDB. The test below used a more recent Scylla Open Source 3.0.x release.

Server Setup

Alternator cluster is designed to match the ScyllaDB cluster previously seen in the blog: 3 nodes are used, of i3.8xlarge on AWS. The table is created with replication factor of 3.

Client Setup

Table is created with:

#!/usr/bin/python
import boto3
import sys

dynamodb = boto3.resource('dynamodb',region_name='None',endpoint_url='http://%s:8000'%(sys.argv[1]))
dynamodb.create_table(
    TableName='usertable',
    BillingMode='PAY_PER_REQUEST',
    KeySchema=[{  'AttributeName': 'key', 'KeyType': 'HASH'   } ],
    AttributeDefinitions=[{  'AttributeName': 'key',  'AttributeType': 'S'  } ],
)

YCSB is compiled from source from git@github.com:brianfrankcooper/YCSB.git@e075edcb2d1ececbe9197a4ddee719db1e52da80

Workload results

WorkloadA (50% reads, 50% updates)

Data: 1000000000 records, 1TB
Client VMs: 15
Threads per client: 50
total throughput fixed at: 120,000 operations/second

Uniform distribution, result from one client

[OVERALL], RunTime(ms), 5400027
[OVERALL], Throughput(ops/sec), 7988.961536673798
[TOTAL_GCS_PS_Scavenge], Count, 35326
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 47906
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8871437124295859
[TOTAL_GCS_PS_MarkSweep], Count, 297
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 4068
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.07533295666854999
[TOTAL_GCs], Count, 35623
[TOTAL_GC_TIME], Time(ms), 51974
[TOTAL_GC_TIME_%], Time(%), 0.962476669098136
[READ], Operations, 21575824
[READ], AverageLatency(us), 5100.720363078602
[READ], MinLatency(us), 1685
[READ], MaxLatency(us), 216063
[READ], 95thPercentileLatency(us), 8735
[READ], 99thPercentileLatency(us), 10391
[READ], Return=OK, 21575824
[CLEANUP], Operations, 50
[CLEANUP], AverageLatency(us), 0.04
[CLEANUP], MinLatency(us), 0
[CLEANUP], MaxLatency(us), 2
[CLEANUP], 95thPercentileLatency(us), 0
[CLEANUP], 99thPercentileLatency(us), 2
[UPDATE], Operations, 21564784
[UPDATE], AverageLatency(us), 3629.162547234417
[UPDATE], MinLatency(us), 1552
[UPDATE], MaxLatency(us), 212607
[UPDATE], 95thPercentileLatency(us), 6103
[UPDATE], 99thPercentileLatency(us), 7327
[UPDATE], Return=OK,

Cluster-wide throughput, putItem + getItem: throughput uniform run

Average Latencies, putItem + getItem:

average latencies uniform run

p99 Latencies, putItem + getItem:

average latencies uniform run

Cache hit ratio:

cache hit ratio, uniform run

CPU utilization:

CPU load

Zipfian distribution, result from one client

[OVERALL], RunTime(ms), 5400031
[OVERALL], Throughput(ops/sec), 7999.443706897238
[TOTAL_GCS_PS_Scavenge], Count, 36081
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 47943
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.8878282365416051
[TOTAL_GCS_PS_MarkSweep], Count, 301
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 3998
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.07403661201204215
[TOTAL_GCs], Count, 36382
[TOTAL_GC_TIME], Time(ms), 51941
[TOTAL_GC_TIME_%], Time(%), 0.9618648485536473
[READ], Operations, 21591496
[READ], AverageLatency(us), 3260.060847057564
[READ], MinLatency(us), 1570
[READ], MaxLatency(us), 215551
[READ], 95thPercentileLatency(us), 6079
[READ], 99thPercentileLatency(us), 8567
[READ], Return=OK, 21591496
[CLEANUP], Operations, 50
[CLEANUP], AverageLatency(us), 0.04
[CLEANUP], MinLatency(us), 0
[CLEANUP], MaxLatency(us), 2
[CLEANUP], 95thPercentileLatency(us), 0
[CLEANUP], 99thPercentileLatency(us), 2
[UPDATE], Operations, 21605748
[UPDATE], AverageLatency(us), 2509.231772905988
[UPDATE], MinLatency(us), 1481
[UPDATE], MaxLatency(us), 213887
[UPDATE], 95thPercentileLatency(us), 4227
[UPDATE], 99thPercentileLatency(us), 5675
[UPDATE], Return=OK,

Cluster-wide throughput, putItem + getItem: throughput zipfian run