ANTLRv4 grammar-based test generator
Grammarinator is a random test generator / fuzzer that creates test cases according to an input ANTLR v4 grammar. The motivation behind this grammar-based approach is to leverage the large variety of publicly available ANTLR v4 grammars.
The trophy page of the found issues is available from the wiki.
To use Grammarinator in another project, it can be added to setup.cfg
as an install requirement (if using setuptools with declarative config):
[options]
install_requires =
grammarinator
To install Grammarinator manually, e.g., into a virtual environment, use pip:
pip install grammarinator
The above approaches install the latest release of Grammarinator from PyPI. Alternatively, for the development version, clone the project and perform a local install:
pip install .
As a first step, Grammarinator takes an ANTLR v4 grammar and creates a test generator script in Python3. Grammarinator supports a subset of the features of the ANTLR grammar which is introduced in the Grammar overview section of the documentation. The produced generator can be subclassed later to customize it further if needed.
Basic command-line syntax of test generator creation:
grammarinator-process <grammar-file(s)> -o <output-directory> --no-actions
After having generated and optionally customized a fuzzer, it can be executed by the grammarinator-generate
script (or by manually instantiating it in a custom-written driver, of course).
Basic command-line syntax of grammarinator-generate
:
grammarinator-generate <generator> -r <start-rule> -d <max-depth> \
-o <output-pattern> -n <number-of-tests> \
-t <transformer1> -t <transformer2>
Beside generating test cases from scratch based on the ANTLR grammar, Grammarinator is also able to recombine existing inputs or mutate only a small portion of them. To use these additional generation approaches, a population of selected test cases has to be prepared. The preparation happens with the grammarinator-parse
tool, which processes the input files with an ANTLR grammar (possibly with the same one as the generator grammar) and builds grammarinator tree representations from them (with .grt extension). Having a population of such .grt files, grammarinator-generate
can make use of them with the --population
cli option. If the --population
option is set, then Grammarinator will choose a strategy (generation, mutation, or recombination) randomly at the creation of every new test case. If any of the strategies is unwanted, they can be disabled with the --no-generate
, --no-mutate
or --no-recombine
options.
Basic command line syntax of grammarinator-parse
:
grammarinator-parse <grammar-file(s)> -r <start-rule> \
-i <input_file(s)> -o <output-directory>
The repository contains a minimal example to generate HTML files. To give it a try, run the processor first:
grammarinator-process examples/grammars/HTMLLexer.g4 examples/grammars/HTMLParser.g4 \
-o examples/fuzzer/
Then, use the generator to produce test cases:
grammarinator-generate HTMLCustomGenerator.HTMLCustomGenerator -r htmlDocument -d 20 \
-o examples/tests/test_%d.html -n 100 \
-s HTMLGenerator.html_space_serializer \
--sys-path examples/fuzzer/
Grammarinator was tested on:
- Linux (Ubuntu 16.04 / 18.04 / 20.04)
- OS X / macOS (10.12 / 10.13 / 10.14 / 10.15 / 11)
- Windows (Server 2012 R2 / Server version 1809 / Windows 10)
Background on Grammarinator is published in:
- Renata Hodovan, Akos Kiss, and Tibor Gyimothy. Grammarinator: A Grammar-Based Open Source Fuzzer. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating Test Case Design, Selection, and Evaluation (A-TEST 2018), pages 45-48, Lake Buena Vista, Florida, USA, November 2018. ACM. https://doi.org/10.1145/3278186.3278193
Licensed under the BSD 3-Clause License.