-
Notifications
You must be signed in to change notification settings - Fork 30
Overview
Simulation allows a rigorous, scalable, and reproducible approach to testing:
Simulation begins with statistical models of the use of your system. This model includes facts such as "we have identified four customer profiles, each with different browsing and purchasing patterns" or "the analytics query for the management report must run every Wednesday afternoon." Models are versioned and kept in a database.
The statistical models are used to create tests. A test contains a detailed description of activity in a system over a period of time. All activity is undertaken by agents. Each agent in the system represents a human user or external process interacting serially with the system, and has its own timestamped stream of actions. With a large number of agents, simulations can produce the highly concurrent activity expected in a large production system.
A simulation (sim) is a particular run of a test. A sim describes how to distribute agents across the processes driving the sim, and give access to the system under test. The simulator coordinates time, playing through the activity streams for all the agents.
Every step of the simulation process, including modeling, test generation, simulation, and the code itself, is captured and stored in a database for further analysis. Results might also include the capture of any other side effects you need: logs, metrics, etc.
Since all phases of a simulation are kept in a database, validation can be performed at any time. This differs markedly from many approaches to testing, which require in-the-moment validation against the live system.
The separation of concerns above, and the use of a versioned, time-aware database, gives simulation great power. Imagine that you get a bug report from the field, and you realize that the bug corresponds to a corner case that you failed to consider. With a simulation-based approach, you can write a new validation for the corner case, and run that validation against your past simulation results, without ever running your actual system.