Alpha Version with GUI

This version brings some improvements to the evaluator core, more module options, but most importantly: a graphical user interface (GUI)! It is a co-release of the core project, i.e., the command line tool, and the GUI. Of course, you can also access it via our central repository.

The GUI

Let us talk about the GUI first: The optimizationBenchmarking evaluator, since its first release, is available as command line tool, i.e., you start it from the shell and provide all information it needs in form of files. In order to make using our tool easier, we now provide a graphical user interface (GUI) in form of a locally-running, stand-alone web application.

Our evaluation process needs meta information about your experimental data, such as what your measurement dimensions are, what your benchmark instances are, and what the parameters of the algorithms that you have experimented are, as well as what kind of information you want the evaluator to get for you. The GUI allows you to specify these information by using convenient (HTML) forms, which are annotated with helpful hints. Furthermore, the GUI also allows you to run the evaluator itself and to download several example data sets into your workspace. You can also upload experimental results and download the results of your evaluation. It also comes with an included help. All of this should make it much easier for you to use our system.

The GUI has been written as a stand-alone web application based on the embedded Jetty server. This has a striking advantage: Computing high-level statistics and creating a report document and then compiling it with LaTeX may need some time if you have much experimental data, e.g., from several experiments on many benchmark instances. Now you can either run the system entirely on your local computer, patiently waiting until it has finished. Or you could start the GUI on a strong server (or, basically, any other computer) in your lab. In both cases, you access the GUI via your web browser in exactly the same way, but in the latter case, no computational load is created on your computer (only the server will sweat).

If you work in a research group, the server-based approach has the additional advantage that you can have once centralized repository to store all of your experimental results. This makes sharing of results throughout the group much easier. (In future versions of our system, we may even add support for this.) If you implement a suitable backup strategy for this repository, you will even gain more safety against the loss of precious experimental results.

Like the original command line application, the GUI comes as stand-alone jar, i.e., you do not need to install anything else. Just put the jar on your computer, start it, and you are done. It will even automatically open a browser and navigate it to the application.

Improvements to the Core

One visible improvement to the core is that diagrams which illustrate functions (such as the ECDF or the progress of a statistical parameter over time) now can either be plotted as-is
or as "ranking plots". In a ranking plot, instead of illustrating the actual values of the functions, we plot their rank. This can help you to distinguish visually similar values easily. A situation where this may come in handy is the following: Assume you want to compare optimization algorithms solving the TSP. They may start at really bad solutions, where the objective value may be 200 times as high than the optimum, i.e., f/f*=200. As time goes by, they may get very close to the optimum, 1≤f/f*≤1.001. If we have values as large as 200 and as small as 1.001 in one diagram, it will be virtually impossible to distinguish 1.001 from 1.002. In a ranking plot, that won't be a problem.

Besides this visual changes there have been a few minor improvements and bug fixes. The primary download is optimizationBenchmarkingGui-0.8.4-full.jar.