Austin Benchmark Suites for Computational Electromagnetics
The CEM benchmark suites are currently being populated. Keep watching this space! To receive updates, you can also subscribe to the email list here: https://utlists.utexas.edu/sympa/subscribe/austincembenchmarks
In software engineering, a "proto-benchmark" is defined as a set of tests used to compare the performance of alternative tools or techniques that is missing one of the components of a proper benchmark; most commonly, a performance measure (S. E. Sim et al., "Using benchmarking to advance research: a challenge to software engineering," in Proc. Int. Conf. Software Eng. May 2003). Both proto-benchmarks and benchmarks can be used to demonstrate features of a new computational system (algorithm, code, supporting software, hardware), while only benchmarks can also systematically combat the ubiquity of error, ameliorate the hazards of specialization, fortify scientific integrity, and inspire research.
More information about benchmarking for advanced research & development can be found at the following references. Please also see the wiki page.
References
[1] J. W. Massey, J. T. Kelley, C. Courtney, D. A. Chamulak, and A. E. Yılmaz, "A benchmark suite for quantifying RCS simulation performance on modern computers," in Proc. USNC/URSI Rad. Sci. Meet., July 2018.
[2] J. T. Kelley, J. W. Massey, and A. E. Yılmaz, “Extending proto-benchmarks to create benchmarks for quantifying modern computational electromagnetics performance,” in Proc. URSI NRSM, Jan. 2018.
[3] A. E. Yılmaz, “Advancing computational electromagnetics research through benchmarking,” in Proc. USNC/URSI Rad. Sci. Meet., July 2017.
[4] J. W. Massey, A. Menshov, and A. E. Yılmaz, “Toward next-generation benchmarking of CEM methods: comparing computational costs,” in Proc. URSI NRSM, Jan. 2017.
[5] J. W. Massey, C. Liu, and A. E. Yılmaz, “Benchmarking to close the credibility gap: A computational BioEM benchmark suite,” in Proc. URSI EMTS, Aug. 2016.