Skip to content

TestSuiteCleanup

Jump to bottom
Romain Deltour edited this page Jan 27, 2020 · 12 revisions

EPUBCheck Test Suite Cleanup

Why?

EPUBCheck’s test suite has grown organically, it consists of both packaged and unpackaged EPUB content, with no consistent naming convention, and several Java test classes, again poorly organized.

As a result, doing what should be simple maintenance work can become a real ordeal. For instance, pull request #798 shows that changing the severity level of a single message required changes to ~140 test files, when it should have ideally required only a couple changes (e.g. one test for EPUB 2 and one test for EPUB 3).

Note: Previous CG attempt

This page was previously documenting an effort to cleanup the test suite within the EPUB 3 Community group. However, due to various reasons, this initiative wasn’t successful. A previous version of this page documents this past intiative.

Porting the functional tests

Most of the cleanup process consists of porting the old functional tests to the new EPUBCheck test suite.

The steps below must be followed for each test to be ported:

  1. Identify the rule(s) being tested. Information that can help with this include:
    • the name of the test method
    • the name of the related test file
    • the error and warning codes expected to be returned by EPUBCheck, if any
    • the verbose output of EPUBCheck
    • the content of the test files
  2. Determine if the test covers a single rule, or tests several rules at once. If the latter, the test should be ported as several test cases.
  3. Make sure the test file content is as minimal as possible. Remove all unnecessary content. Ideally, all test files should be based on the minimal EPUB content, adding only what is necessary for the feature being tested.
  4. For each rule identified in steps 1 & 2 above, add a Gherkin scenario to the relevant feature file. See the feature files organization to know more on which feature file to use.
  5. Rename the related test file (or directory, for expanded EPUBs) according to the naming convention, and move it to the appropriate location. See the test input files organization to know more on where to move test files.

Rinse and repeat for each JUnit test to be ported. The list of test classes to be ported is available in the next section.

List of test classes to be ported

Historical EPUBCheck functional tests:

  • com.adobe.epubcheck.api.Epub20CheckExpandedTest.java (22 tests)
  • com.adobe.epubcheck.api.Epub20CheckTest.java (33 tests)
  • com.adobe.epubcheck.api.Epub30CheckExpandedTest.java (199 tests)
  • com.adobe.epubcheck.api.Epub30CheckTest.java (48 tests)
  • com.adobe.epubcheck.nav.NavCheckerTest.java (16 tests)
  • com.adobe.epubcheck.ocf.OCFCheckerTest.java (21 tests)
  • com.adobe.epubcheck.opf.OPFCheckerTest.java (164 tests)
  • com.adobe.epubcheck.ops.OPSCheckerTest.java (153 tests)
  • com.adobe.epubcheck.overlay.OverlayCheckerTest.java (12 tests)

Tests coming from the B&N contribution:

  • com.adobe.epubcheck.test.package_Test.java
  • com.adobe.epubcheck.test.single_file_Test.java
  • com.adobe.epubcheck.test.opf_Test.java
  • com.adobe.epubcheck.test.dtBook_Test.java
  • com.adobe.epubcheck.test.encryption_Test.java
  • com.adobe.epubcheck.test.css_Test.java
  • com.adobe.epubcheck.test.toc_Test.java
  • com.adobe.epubcheck.test.xhtml_Test.java
  • com.adobe.epubcheck.test.script_Test.java
Clone this wiki locally