Run Test Suite classes with method per test

[robodude666]

I really enjoy vba-tdd for it's simplicity, however, I find it limiting when developing large test suites. The fact that everything must be in a single function can either make for very long messy suites or create a ton of small suites. Splitting into sub-suites is also difficult due to VBA's module name size limitation. I've ran into the Procedure too large error several times so far too.

It would be helpful to support running test suite classes where each method is its own test; this will help with a number of things as I'll discuss below.

As an example, the AddTests example could be written as:

Option Explicit

Public Sub ShouldAddTwoNumbers(ByRef Test As TestCase)
    On Error GoTo UnexpectedError
    Test.IsEqual Add(2, 2), 4
    Test.IsEqual Add(3, -1), 2
    Test.IsEqual Add(-1, -2), -3
    On Error GoTo 0
    Exit Sub

    UnexpectedError:
    Test.FailFromError Err
End Sub

Public Sub ShouldAddAnyNumberOfNumbers(ByRef Test As TestCase)
    On Error GoTo UnexpectedError
    Test.IsEqual Add(1, 2, 3), 6
    Test.IsEqual Add(1, 2, 3, 4), 10
    On Error GoTo 0
    Exit Sub

UnexpectedError:
    Test.FailFromError Err
End Sub

Note: Technically error handling within the test subroutine would be optional; see below.

Because VBA does not support programmatically creating classes the user would have to specify a "factory method" to the reporter:

Public Function CreateTestCaseSuite() As Tests_TestCase
    Set CreateTestCaseSuite = New Tests_TestCase
End Function

Public Function CreateTestSuiteSuite() As Tests_TestSuite
    Set CreateTestSuiteSuite = New Tests_TestSuite
End Function

Public Sub RunTests()
    Dim Reporter As New WorkbookReporter
    Reporter.ConnectTo TestRunner
    
    Reporter.AddSuiteForFactory TestSuite.Create("TestCase"), "CreateTestCaseSuite"
    Reporter.AddSuiteForFactory TestSuite.Create("TestSuite"), "CreateTestSuiteSuite"
    Reporter.Done
End Sub

The reporter would be able to:

1. Detect the suite's class name from the return type of the factory.
2. Detect all Public Subroutines within the class.
3. For each subroutine detected:
   1. Call the factory method to create a new instance.
   2. The name of each test can be extrapolated from the PascalCase/SnakeCase name of the test.
      e.g. ShouldAddTwoNumbers and should_add_two_numbers would both convert to "should add two numbers"
   3. Create a new TestCase for the TestSuite based on the above name.
   4. Call each subroutine while passing in the TestCase instance.

Note: To support class reuse SetUp/TearDown and SetUpSuite/TearDownSuite subroutines could be used.

This has several benefits:

1. Allows for large test suites to be broken out into individual test subroutines.
2. Ensures each test runs in a fresh context. You don't have to worry reusing variables that might affect the outcome of your test.
3. Better error handling within the test itself.
4. Provides global error handling for unhandled errors (when the Reporter calls CallByName it can detect if an error occured).
5. Not having to worry about updating NumSuites.
6. Provide the capability of timing how long each test and suite takes to run.
7. Potentially support rerunning only failing suites/tests.

In order to make TestSuite.Create work the PredeclaredId would have to be set to True.

It may also be helpful to add a generic runner class instead of using the WorkbookReporter/ImmediateReporter directly:

Public Sub RunTests()
    Dim Runner As New TestSuiteRunner
    Runner.AddReporter WorkbookReporter.Create(ThisWorkbook.Worksheets("TestRunner"))
    Runner.AddReporter ImmediateReporter.Create() 
    
    Runner.AddSuiteForFactory TestSuite.Create("TestCase"), "CreateTestCaseSuite"
    Runner.AddSuiteForFactory TestSuite.Create("TestSuite"), "CreateTestSuiteSuite"

    Runner.Run
End Sub

Final closing notes: I bring this suggestion up because I really like the all-included aspect of vba-test. While there are more powerful solutions available for vba testing (like RubberDuck/SimplyVBUnit (with modification)), they require external installation which in my particular application is not a feasible option. I strongly feel such an enhancement to vba-test will make it much more flexible.

Thoughts? 😄

[timhall]

I'm heading towards method-per-test too, I've been bitten by variables / errors from other tests causing issues too. I have some plans for how to build the process into vba-test and will definitely take a deep look at your suggestions. I'll get back to you on this

[connerk]

Good plan fellas.
I got something similar working in a different way last December
starting at 4c34095
and improved through 0d8b936

basically, I added a collection of specsuites to the specsuite class, allowing nesting. it's not explicitly one test per method but you could do that if you wanted to.

it made reporting of large numbers of tests more organized in the workbook reporter as well by allowing for collapsible, nested reports.

how to structure the code is in the Readme

example WorkbookReporter output

food for thought at least I hope. 😄

[robodude666]

@timhall good to hear! Looking forward to it 👍

@connerk I originally tried something similar but the issue I have with it is you need to remember to add the "sub" test suite. The benefit of discovery (as outlined above) is all you need to do is add a public sub and the test will automatically run.

[connerk]

any consideration on using comment markers?
Rubber duck calls them Annotations
NUnit calls them Attributes

[robodude666]

@timhall btw, what were you thinking of?

@connerk The way RubberDuck handles Annotations via an ANTLR grammer makes it really easy on the implementation. However, if we do it from VBA you can't really do that very well. In VBA you can very easily iterate through the CodeModule fairly efficiently and get the name of the procedure (via ProcOfLine).

It is technically possible to use the same method and then parse the comments before the beginning of the procedure, but I donno.

I do like the idea through of supporting skipping tests or marking them as expecting to fail, but I don't like the idea of needing to add extra stuff to make a test run. Perhaps a combination of both? Using signature to detect "TestMethod" and annotations to mark tests for skip/fail?

[timhall]

Generally, I see vba-test as a low-level testing library that can provide the backing for various code generation, annotation, and other approaches. With #27 I think both approaches presented here are doable. I imagine @robodude666 example like the following:

Public Sub RunTests()
  Dim Suite As New TestSuite
  Suite.Name = "Project Name"

  Dim Reporter As New WorkbookReporter
  Reporter.ConnectTo ThisWorkbook.Worksheets("TestRunner")
  Reporter.ListenTo Suite

  Dim DebugReporter As New ImmediateReporter
  DebugReporter.ListenTo Suite

  ' From external library
  LoadTestsFromModule Suite.Group("Module Name"), "ModuleName"
  LoadTestsFromClass Suite.Group("Class Name"), "ClassName"
End Sub

' Example

Public Sub LoadTestsFromModule(Suite As TestSuite, Name As String)
  ' Load CodeModule for Name
  ' Iterate through module looking for ' #[test] attribute
  ' -> Get test name (and modifiers)
  On Error Resume Next  

  For Each TestName In FoundTests
    Set Test = Suite.Test(TestName)
    Application.Run Name & "." & TestName, Test
    Test.NotError
    
    Err.Clear
  Next TestName  
End Sub

Public Sub LoadTestsFromClass(Suite As TestSuite, Name As String)
  ' Similar to above, but with CallByName
End Sub

The point being that LoadTestsFromModule can happen outside of vba-test. With that said I plan on integrating test generation into vba-blocks (once I get it working and released). That will generate and inject the test runner code based on ' #[test] "attributes". This would generate output like the following:

Public Sub RunTests()
  Dim Suite As New TestSuite
  Suite.Name = "Project Name"

  Dim Context As New Tests_Context ' <- Generated by vba-blocks test
  
  With Suite.Group("TestCase")
    Context.TestCase_ListenTo .Self
        
    Context.TestCase_ShouldDoA .Self.Test("should do a")
    Context.TestCase_ShouldDoB .Self.Test("should do b")
  End With
    
  With Suite.Group("TestSuite")
    Context.TestSuite_ListenTo .Self

    Context.TestSuite_ShouldDoC .Self.Test("should do c")
    Context.TestSuite_ShouldDoD .Self.Test("should do d")
  End With
End Sub

This would allow writing tests in modules or classes like the following:

' #[test.before_each]
Public Sub BeforeEach(Test As TestCase)
  ' ...
End Function

' #[test]
Public Sub ShouldAddTwoNumbers(Test As TestCase)
  ' ...
End Function

' #[test.skip]
Public Sub SkippedTest(Test As TestCase)

End Function

' #[test("override test name").only]
Public Sub ShouldAddAnyNumberOfNumbers(Test As TestCase)
  ' ...
End Function

' #[test.group]
Public Sub SubGroup(Suite As TestSuite)
  With Suite.Test("...")
    ' ...
  End With
End Sub

With all that said, I think vba-test will stay somewhat minimal, with either separate packages finding and running tests or via an outside tool like vba-blocks generating test runners