Add support for running & reporting doctest tests

[fatuhoku]

Supporting doctest-haskell can be an invaluable feature for the Haskell development experience, as well as for the development of test-framework itself.

This is because test-framework itself has a number of test reporters that output test results onto the consoles. However, currently there are no tests documenting what the console would actually output.

[sol]

@fushunpoon We once had test-framework integration for Doctest, but it has been deprecated, mainly because it was much slower than running doctest from the command-line. The recommended way to run Doctest is to add a separate test-suite section to you cabal file. This works pretty well.

A basic usage example is included in Doctest's repository (have a look at the corresponding cabal file!).

For other useful stuff (that we should probably include with Doctest at some point) look at some of @ekmett packages.

[fatuhoku]

@sol thanks for your response. However, I maintain that we should supply an integration for doctests.

I mightn't fully understand actually caused the performance situation that motivated the creation of another command line runner called doctest.

However, the mission of test-framework (or so I presume) is to be THE de-facto test runner framework for Haskell. I for one would like to see test-framework be as prominent as 'JUnit' or 'TestNG' is to Java.

With test standardisation, you open the doors to running tests selectively in IDE's, test parallelisation, have a single entry point for CI (e.g. cabal test) without introducing extra test scripts. This has already been made a reality for quickcheck, quickcheck2 and HUnit tests, and I feel doctest should join this family.

What's more, you still can choose to execute quickcheck tests using the quickcheck command line too. Naturally, it has Cabal integration as well. Nothing stops you from doing that. Why support quickcheck then?

Solution?

If a previous project already exists for this, then we have a simple solution: let's just merge the deprecated package into this code-base. My argument for this is: slow is far far better than non-existent.

Even if the final solution calls doctest through a rawSystem call (mitigating the performance problem, we will have gained something: test execution homogeneity.

EDIT: one thing that immediately comes to mind with this sort of delegation is that doctest makes the fact that there are individual tests opaque to the test-runner. Some of these tests should be parallelisable!

[fatuhoku]

I found the relevant issue on test-framework-doctest. The only motivation for deprecation in this case is that there's an interface change in doctest. I think the effort to update test-framework-doctest is worthwhile.

Or perhaps we've raised another question altogether here:

Should the collection of a project's test-suites be wholly defined and declared in a .cabal file, or should test-framework manage the collection of test-suites?

Cabal Test-Suite declarations

For the record, I think that one Cabal Test-Suite declaration is fine; multiple declarations are harmful:
doctest will require one, test-framework will require one, smallcheck will require one.

Simply put, you end up with test-suites organised around the kind of test being written, rather than what it really should be: you organise your test code around the structure of your actual Haskell software modules!

On the contrary, if test-framework were used, it potentially has complete understanding of all the tests it's executing. Reporting, structuring, and execution is far more controllable.

Contrast this with the way Cabal runs tests, which is... well, blindly.