Extensible options

[UnkindPartition]

A nice thing about test-framework is that it can be extended with new test types without modifying the framework itself, thanks to existential types.

One thing that is not yet extensible is options. Here's a sketch of a proposal to address this.

{-# LANGUAGE ScopedTypeVariables, DeriveDataTypeable #-}
module Options
  ( OptionSet
  , IsOption(..)
  , option
  , lookupOption
  , plusTestOption
  , plusTestOptions
  , SomeOptDescr(..)
  ) where

import Data.Typeable
import qualified Data.Map as Map
import Data.Map (Map)
import Data.Monoid
import System.Console.GetOpt
import Test.Framework hiding (plusTestOptions)

class Typeable v => IsOption v where
  defaultValue :: v

data SomeOptDescr = forall v. IsOption v => SomeOption (OptDescr v)

data OptionValue = forall v . IsOption v => OptionValue v

newtype OptionSet = OptionSet (Map TypeRep OptionValue)

instance Monoid OptionSet where
  mempty = OptionSet Map.empty
  mappend (OptionSet s1) (OptionSet s2) =
    OptionSet $ Map.unionWith (flip const) s1 s2 -- right-biased

option :: IsOption v => v -> OptionSet
option v = OptionSet $ Map.singleton (typeOf v) (OptionValue v)

lookupOption :: forall v . IsOption v => OptionSet -> v
lookupOption (OptionSet s) =
  case Map.lookup (typeOf (undefined :: v)) s of
    Just (OptionValue x) | Just v <- cast x -> v
    Just {} -> error "OptionSet: broken invariant (shouldn't happen)"
    Nothing -> defaultValue

{-
class TestResultlike i r => Testlike i r t | t -> i r, r -> i where
    ...
    testOptions :: [SomeOptDescr]
-}

plusTestOptions :: OptionSet -> Test -> Test
plusTestOptions = undefined

plusTestOption :: IsOption v => v -> Test -> Test
plusTestOption = plusTestOptions . option

-- Example option
newtype QCMaxTests = QCMaxTests Int
  deriving Typeable

instance IsOption QCMaxTests where
  defaultValue = QCMaxTests 100

{-
instance Testlike PropertyTestCount PropertyResult Property where
  ...
  testOptions =
    [
      SomeOption $ Option
        ['a']
        ["maximum-generated-tests"]
          (ReqArg (QCMaxTests . read) "NUMBER")
          "how many automated tests QuickCheck should try"
    , ...]
-}

Advantages

• Any test provider can register its own options
• ./mytest --help would automatically show only those options that are relevant for a particular test suite. E.g. if I only use SmallCheck and HUnit, I wouldn't see opitons related to QuickCheck.

Stuff to think about

• I'd like to see similar extensibility for runner options, to allow alternative runners and switching between them.
• There's some probability of clash for short option names. It can be solved this way: by default, only long option names are registered. A function is exposed which allows to add short options conveniently. Then providers may expose convenience functions like "addQuickCheckShortOptions". The user can opt in for those, or she can even define her own option names.

[batterseapower]

I've been thinking about this issue over Christmas. It is one of the main non-extensible parts of test-framework and I always realise that it was a deficiency.

I partially implemented an approach where I had:

class (Monoid o, Typeable o) => TestOptionslike o where
    -- | Prefix affixed to options to disambiguate them if necessary
    optionsConsolePrefix :: o -> String
    optionsConsoleDescription :: o -> [OptDescr o]

class (Typeable t, TestResultlike i r, TestOptionslike o) => Testlike i r o t | t -> i r, t -> o, r -> i where
    runTest :: o -> t -> IO (i :~> r, IO ())
    testTypeName :: t -> TestTypeName    

data Test = ...
            | forall o. TestOptionslike o => PlusTestOptions o Test       -- ^ Add some options to child tests

But then I realised that with this approach I could not easily have an option shared between two providers. For example we may wish to have a Seed option which can be used by both QC and QC2 providers: it would be annoying to force users to supply a separate seed option on the command line for both libraries.

So perhaps your original approach with one data type per option is indeed better.