Skip to content

asatarin/testing-distributed-systems

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

List of resources on testing distributed systems curated by Andrey Satarin (@asatarin). If you are interested in my other stuff, check out talks page. For any questions or suggestions you can reach out to me on Twitter (@asatarin), Mastodon (https://discuss.systems/@asatarin) or LinkedIn.

Table of Contents

  • A Markdown unordered list which will be replaced with the ToC {:toc}

Overview of Testing Approaches

Research Papers

Bugs

Testing

Fault Tolerance

Technologies for Testing Distributed Systems by Colin Scott

{:.no_toc}

Colin Scott shares his viewpoint from academia on testing distributed systems, specifically regression testing for correctness and performance bugs.

Testing in a Distributed World by Ines Sombra (RICON 2014)

{:.no_toc}

Great overview of techniques for testing distributed systems from practitioner, the video did age well and still an excellent overview of the landscape. Additional materials could be found in this GitHub repo

Resilience In Complex Adaptive Systems

These materials are not directly related to testing distributed systems, but they greatly contribute to general understanding of such systems.

Jepsen

State-of-the-art approach to testing stateful distributed systems.

Elle transactional consistency checker for black-box databases:

Some notable Jepsen analyses:

Jepsen is used by CockroachDB, VoltDB, Cassandra, ScyllaDB and others.

Formal Methods

TLA+

Companies using TLA+ to verify correctness of algorithms:

Deterministic Simulation

Pioneered by FoundationDB, deterministic simulation approach to testing distributed systems gained more popularity in recent years.

More companies and systems adopt deterministic simulation as a primary testing strategy:

Lineage-driven Fault Injection

Netflix adopted lineage-driven fault injection techniques for testing microservices.

Chaos Engineering

Netflix pioneered chaos engineering discipline.

Fuzzing

There are two flavors of fuzzing. First, randomized concurrency testing, where the ordering of messages is fuzzed:

And input fuzzing, where message contents or user inputs are fuzzed:

Microservices

Amazing and comprehensive overview of different strategies to test systems built with microservices by Cindy Sridharan.

Series of blog posts specifically on testing in production — best practices, pitfalls, etc:

Game Days

Performance and Benchmarking

See also benchmarking tools.

Test Case Reduction

Misc

Specific Approaches in Different Distributed Systems

Google

Amazon Web Services

See also formal methods and deterministic simulation sections.

Netflix

Automated failure injection (see also Lineage-driven Fault Injection):

Random/manual failure injection testing:

See also Chaos Engineering and Lineage-driven Fault Injection.

Microsoft

See also formal methods section.

Meta

  • BellJar: A new framework for testing system recoverability at scale — BellJar is a testing framework focused on answering question "What service dependencies are required for the service to recover after large scale disaster?". BellJar puts service in a vacuum environment with only handful of direct dependencies allow-listed to verify that recovery procedures succeed under those constraints. It checks those recovery procedures in CI/CD pipeline preventing unconstrained growth of dependency graph and circular dependencies. Based on BellJar tests one can construct the entire dependency graph of the services allowing to boostrap them in the correct order from bottom to top.

FoundationDB

See also deterministic simulation section.

Cassandra

ScyllaDB

They published series of blog posts on testing ScyllaDB:

Dropbox

  • Mysteries of Dropbox Property-Based Testing of a Distributed Synchronization Service — example of how to use QuickCheck to test synchronization in Dropbox and similar tools (Google Drive). John Hughes gave a talk on this. See also QuickCheck.
  • Data Checking at Dropbox — If you have lots of data, you have to verify that it did not suffer from bit rot and protect it against rare bugs (e.g. race conditions) to guarantee long term durability. This talks explains intricacies of building data consistency checker(s) at Dropbox scale.
  • Dropbox's Exabyte Storage System (aka Magic Pocket) talk by James Cowling — describes number of strategies to achieve extremely high durability. This includes:
    • guard against faulty disks,
    • guard against software defects,
    • guard against black swan events,
    • operational safeguards to reduce blast radius,
    • safeguards against deletes with multi stage soft-delete,
    • comprehensive testing strategy in-depth with increased scale,
    • redundancy across various axis in software and hardware stacks,
    • continuous data integrity validation on many levels,
    • etc
  • Testing sync at Dropbox — comprehensive overview of two test frameworks at Dropbox for new sync engine implementation. CanopyCheck — single threaded and fully deterministic randomized testing framework with minimization for synchronization planner component of the engine. The other framework Trinity focuses on concurrency and larger surface area of components. Great discussion on tradeoffs between determinism, strength of test oracles vs width of coverage and size of the system under test.

Elastic (Elasticsearch)

See also formal methods section.

MongoDB

See also formal methods section.

Confluent (Kafka)

See also formal methods section.

CockroachLabs (CockroachDB)

SingleStore

Formerly known as MemSQL.

Twitter

LinkedIn

Salesforce

VoltDB

Series of post on testing at VoltDB:

Additional resources:

PingCap (TiDB)

See also formal methods section.

Cloudera

Wallaroo Labs

There is also talk from Sean T. Allen on testing stream processing system at Wallaroo Labs (ex. Sendence)

YugabyteDB

FaunaDB

Shopify

Hazelcast

Basho (Riak)

Etcd

Red Planet Labs

See also deterministic simulation section.

Atomix Copycat

Onyx

Druid.io

TigerBeetle

See also deterministic simulation section.

Convex

See also QuickCheck, FoundationDB, Dropbox, Jepsen, deterministic simulation.

RisingWave

In a series of two blog posts, RisingWave team talks about their experience using deterministic simulation for testing distributed SQL-based stream processing platform:

As a result of this work, they open sourced MadSim — Magical Deterministic Simulator for the Rust language ecosystem.

See also deterministic simulation section.

Single Node Systems

These examples are not about distributed systems, but they demonstrate testing concurrency and level of sophistication required in distributed systems.

SQLite

SQLite is not a distributed system by any stretch of the imagination, but provides good example of comprehensive testing of a database implementation.

Sled

See also deterministic simulation section.

Clickhouse

Tools

Network Simulation

QuickCheck

Benchmarking

Linkbench

YCSB