Test Cases Release v3.4.0

This release contains a number of updates for all three branches of the test suite.

For the 'semantic' tests:

• Tests 1781-1821 added, including tests for:
  • Default values of various elements.
  • Local parameters shadowing reaction IDs and species reference IDs.
  • Case sensitivity.
  • 0D-compartments in Level 3.
  • As suggested in #72: add tests where model elements are given mathematical literal IDs.
• A number of previously-existing tests updated, including:
  • Correct 'comp' model files with extra id's.
  • Correct 'comp' models with external model definitions.
  • Adjust some mathematical tests so they test the function in question and not the encoding of integers as doubles.
  • Add L2v5 and L3v2 models to all relevant tests that didn't already have them.

For the 'syntactic' tests:

• A number of new tests, particularly for the spatial package. (Now up to date as of libsbml 5.19.3) These tests attached as a binary for this release, and are not present in the source release.

For the 'stochastic' tests:

• New tests 40-100, which test implementation of the new 'distrib' package functions.

The SBML Test Suite Runner has not been updated in this release, just the tests themselves. The SBML Test Suite Database of tests and test results from various tools has been updated, so new results from simulators may be uploaded.

For convenience, here are the links to the 3.3.0 installers for the SBML Test Suite Runner. Once installed, you'll need to update the test files by hand:

O.S.	Architecture	Format	File
macOS	64-bit	runnable installer	SBML Test Runner 3.3.0 macOS installer.pkg
Windows	64- or 32-bit	runnable installer	Setup_SBMLTestRunner-3.3.0-Windows.exe
Linux	64-bit	runnable installer	SBMLTestRunner-3.3.0-linux-x64-installer.run
Linux	32-bit	runnable installer	SBMLTestRunner-3.3.0-linux-installer.run
Linux	64-bit	Debian package	SBMLTestRunner_3.3.0-0.linux_amd64.deb

SBML Test Suite release 3.3.0

At long last, we are pleased to announce the full release of version 3.3.0 of the SBML Test Suite. This release brings many updates and some changes. A summary of the changes is listed below; more details can be found in the NEWS.md file.

Installers

The release page on GitHub has the following ready-to-run installers for different operating systems:

O.S.	Architecture	Format	File
macOS	64-bit	runnable installer	SBML Test Runner 3.3.0 macOS installer.pkg
Windows	64- or 32-bit	runnable installer	Setup_SBMLTestRunner-3.3.0-Windows.exe
Linux	64-bit	runnable installer	SBMLTestRunner-3.3.0-linux-x64-installer.run
Linux	32-bit	runnable installer	SBMLTestRunner-3.3.0-linux-installer.run
Linux	64-bit	Debian package	SBMLTestRunner_3.3.0-0.linux_amd64.deb

In addition, archives of the test cases are available for separate download. Note that the SBML Test Runner includes the semantic cases internally so you do not need to download them. Only download the test case archives if you want to run the stochastic or syntactic test cases using your own framework or software, or need the semantic test cases separately for some reason.

Test case archive	Included in Test Runner?	File
Stochastic tests	⬜︎	sbml-stochastic-test-cases-2017-12-12.zip
Syntactic tests	⬜︎	sbml-syntactic-test-cases-2017-12-12.zip
Semantic tests	⬛︎	sbml-semantic-test-cases-2017-12-12.zip

Changes in release 3.3.0

• There is a new SBML Test Runner installer for Mac OS X. The installation is now distributed as an OS X .pkg installer file. By default, it will install the SBML Test Runner into the /Applications folder on the user's computer.

• This release contains hundreds of new semantic test cases, numbers 01219–1780, written by Lucian Smith. They test many more SBML features and rules. Important note: tests 01234–01292 specifically test models that follow the release candidate specification of SBML Level 3 Version 2, which encode elements and situations that were not present or illegal in SBML Level 3 Version 1. If your simulator uses libSBML, it will need version 5.15.0 or later of libSBML, though the latest (5.16.0) must be used to correctly validate some models. Please see the file NEWS.md (provided with the distribution) for more details about the changes to the test cases. Finally, as before, the SBML Test Runner only uses the semantic test cases (and bundles them), but stochastic and syntactic test cases are also available as separate downloadable archives.

• A few things have been cleaned up in past tests. Examples include model identifiers of a few test models that were misleading (now fixed), and a few tests that were designed to test species in multiple compartments were at some point accidentally changed so that all the species were put into a single compartment (they have been redistributed appropriately now).

• With the addition of FBC v2 tests, the packagesPresent line in the .m file can now include fbc_v1 or fbc_v2, depending on which version of the test it is. The original fbc tag is still present. In addition, a new FBC test tag fbc:NonStrict is now provided for FBC v2 models where the strict flag has been set to false.

• The Test Runner's algorithm for comparing test results and displaying the differences between expected and actual result values now properly ignores column ordering. Also, it now considers it an error if the results do not have the expected number of rows. Thanks to Leandro Watanabe and Chris Myers (U. Utah) and Lucian Smith for reporting the problems.

• The Test Runner now allows the combination of filters that include, simultaneously, tags, problematic test cases, and only supported test cases. Thanks to Chris Myers for raising the issue and testing the implementation.

• The SBML Test Runner now monitors for file changes in the currently-displayed test case. If the application's output .csv file is changed outside the runner (e.g., by manually editing the file), it should now notice the changes and update the display. In addition, the Test Runner also monitors the test case settings file, to detect changes that developers may make while experimenting with test case definitions. Note that for architectural reasons, only the case being displayed is so monitored. A setting in the preferences dialog allows this feature to be turned on and off.

• This release of the SBML Test Runner works around a crashing bug caused by an SWT-GTK issue that manifests itself on Ubuntu 16.

• This release of the SBML Test Runner works around a font scaling issue that manifests itself on Windows when using display scaling of 150%.

• The default value of the "Always check for new versions of test cases when application starts" preference option is now "yes" instead of "no" for new users. Users who have used the previous version of the Test Runner will not be affected; this only affects new users of the Test Runner.

• The File → Exit menu item should now work on Windows. Thanks to Thomas Hamm for reporting that it did nothing before. Missing keyboard shortcuts on Windows are now available in the latest version.

• The Test Runner features various other small fixes and improvements.

• The SBML Test Suite home repository has been moved to GitHub. New development will take place in the branch called ‘develop’. Each time we make a new release, we will merge the changes into branch ‘master’ and start the next new developments in ‘develop’.

• The issue tracker for the Test Suite is now the GitHub tracker: https://github.com/sbmlteam/sbml-test-suite/issues. Lucian Smith migrated the previous issues from the SourceForce tracker, so they are now in the GitHub tracker, for history. We thank the guides by Thomas Zajac and Chris Mungall for helpful info about how to achieve this migration.

More information

Please see the SBML Test Suite for general information about the SBML Test Suite, and the GitHub Page for the SBML Test Suite for the software source code.

This software is released under the terms of the LGPL 2.1, but it incorporates code and other materials from other software. Please see the LICENSE.html file for more detailed information about license and distribution terms.

Test cases release 3.3.0 (alpha)

This is an alpha test release of the test cases archives only.

Important: do not run semantic test cases 1199-1204 on systems that use libSBML 5.11.2 or earlier. Test cases 1199-1204 test the use of nested MathML <piecewise> constructs, which unfortunately revealed a bug in libSBML versions prior to 5.11.4. Make sure to use libSBML 5.11.4 or later.

New in this release:

• The SBML Test Suite home repository has been moved to GitHub. New development will take place in the branch called 'develop': https://github.com/sbmlteam/sbml-test-suite/tree/develop Each time we make a new release, we will merge the changes into branch 'master' and start the next new developments in 'develop'.
• New semantic test cases:
  • Cases 01224-01233 test the use of a reaction identifier in the MathML of another construct.
  • Cases 01219-01223 test the use of compartment sizes other than 1 in combination with of species having attribute hasOnlySubstanceUnits=true and various values for attributes initialAmount and initialConcentration in combination with rate rules.
• New semantic and syntactic test cases for the release candidate specification of SBML Level 3 Version 2. To test these, you will need the latest experimental version of libSBML, 5.14.0.
  • All existing tests that could be translated to SBML Level 3 Version 2 (L3V2) now have L3V2 versions.
  • There are tests for the use of elements without child elements, which is allowed in L3V2 but not in L3V1. This includes <math> children especially, but also things like the <trigger> of events. (Tests 1234-1247, and 1271.) This is because some things are allowed in L3V2 that were not allowed in lower Level+Version combinations of SBML.
  • New test cases for the use of Boolean values in numeric contexts, and visa versa. (Tests 01282-01292.)
  • The use of the newly-added MathML operators: max, min, rem, quotient, implies. (Tests 01272-01281.)
  • The use of the newly-added MathML csymbol for <rateof>. (Tests 1248-1270.)

Changes in this release:

• A few things have been cleaned up in past tests. Examples include model identifiers of a few test models that were misleading (now fixed), and a few tests that were designed to test species in multiple compartments were at some point accidentally changed so that all the species were put into a single compartment (they have been redistributed appropriately now).
• The issue tracker for the Test Suite is now the GitHub tracker: https://github.com/sbmlteam/sbml-test-suite/issues. Lucian Smith migrated the previous issues from the SourceForce tracker, so they are now in the GitHub tracker, for history. We thank the guides by Thomas Zajac and Chris Mungall for helpful info about how to achieve this migration.