The Brown Benchmark for Table Types
This benchmark is documented in the paper Types for Tables: A Language Design Benchmark. Please read it to understand what this benchmark is trying to accomplish and what the following components are about.
To submit an implementation of B2T2, open a pull request with a new folder in
the Media/ directory for your programming medium (which might be a language,
a theoretical model, or some other way of expressing tabular programs).
The folder should include at least two things:
- code that addresses the main components of B2T2 (Tables, API, Example Programs, Errors)
- a datasheet describing the implementation
Refer to Media/TypeScript for an example.
B2T2 strives to be fair, comprehensive, concise, and realistic:
-
Fairness. A benchmark would be unfair if it was designed to favor some type systems over others. Since B2T2 combines operators from untyped languages and we (its authors) don't currently "own" a type system for tables, there is no obvious problem when it comes to fairness.
-
Comprehensiveness. A benchmark would be incomprehensive if it left out things that are obviously important. B2T2 draws from several widely-used programming media and also some educational materials in an effort to cover all the bases.
-
Conciseness. Although we tried to be comprehensive, we also wanted to keep B2T2 within a reasonable size. So we kept important things but also compressed them. For example, if an operation is equivalent to a composition of other operations, the composition is not included.
-
Realistic. An unrealistic benchmark would address synthetic or hypothesized use-cases rather than real ones. B2T2 draws from existing libraries to satisfy real needs.
Future changes to B2T2 will be made with these perspectives in mind.
To refer to this benchmark, please use the following citation:
@article{lgk-pj-2022,
title={Types for Tables: A Language Design Benchmark},
author={Kuang-Chen Lu and Ben Greenman and and Shriram Krishnamurthi},
journal={The Art, Science, and Engineering of Programming},
volume={6},
number={2},
pages={8:1--8:30},
year={2022}
}
You may, if you wish, refer to this repository in addition, but please do refer to the above paper.