skip within before(:all) of isolated example group causes outer example group to be pending

[mlapierre]

Subject of the issue

I'm trying to write tests for before hooks and have run into trouble. When a before(:all) hook is skipped in an isolated example group it sets the outer example group as pending, but it should pass.

Your environment

• Ruby version: ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
• rspec-core version: 3.8.0

Steps to reproduce

describe do
  it 'should pass' do
    RSpec.describe do
      before(:all) { skip }
      it('should be skipped') { }
    end.run
  end
end

Expected behavior

.*

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) should be skipped
     # No reason given
     # ./spec.rb:7


Finished in 0.0013 seconds (files took 0.07034 seconds to load)
2 examples, 0 failures, 1 pending

Actual behavior

F*

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) should be skipped
     # No reason given
     # ./spec.rb:7


Failures:

  1) should pass FIXED
     Expected pending 'No reason given' to fail. No error was raised.
     # ./spec.rb:4

Finished in 0.00124 seconds (files took 0.06776 seconds to load)
2 examples, 1 failure, 1 pending

Failed examples:

rspec ./spec.rb:4 # should pass

More info

Strangely, it's fine if the test has multiple isolated example groups:

describe do
  it 'should pass' do
    RSpec.describe do
      it('passes') { }
    end.run

    RSpec.describe do
      before(:all) { skip }
      it('should be skipped') { }
    end.run
  end
end

..*

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) should be skipped
     # No reason given
     # ./spec.rb:11


Finished in 0.00145 seconds (files took 0.0682 seconds to load)
3 examples, 0 failures, 1 pending

[JonRowe]

👋 If you have a look at the rspec-core specs you will see all sorts of protection and isolation (sand boxing) to allow us to run specs within rspec, its not a "public api" intended to be used in the way you are using it. You will probably need to replicate what we've done to ensure the "inner rspec" works from the outer rspec, in both your examples you should see only 1 spec output for example...

May I ask why you are trying to test before hooks? Is this report psuedo coded from something else?

[godfat]

@JonRowe We're trying to add some new features on top of RSpec, and would like to write tests for them. You can find more context at: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24942#note_140637060

[mlapierre]

Yeah, this is a minimal reproduction of something more complex, as @godfat noted.

Essentially, we're trying to allow tests to be skipped unless focused (e.g., as if running rspec with --tag skip would actually run the skipped tests). We did that via configuration hooks that would skip examples if they had user-defined metadata :quarantine, unless :quarantine was focused, in which case it wouldn't skip the tests.

That worked, but when I tried to write unit tests of the configuration I ran into this issue.

[JonRowe]

Ok so as a minimum you need to setup a reporter of your own, and pass it to run so that the status of the internal groups don't expand into the global. You then need to check the groups pass / fail as desired.

Then you're going to want RSpec::Core::Sandbox.sandboxed and wrap that around every one of the outer examples.

[JonRowe]

You can see examples of this in our spec/spec_helper.rb and spec/support/sandboxing.rb.

[mlapierre]

Ahh, thanks @JonRowe! That explains a few things about the RSpec unit tests that I wasn't understanding, like how tests of configuration didn't affect other tests.

With that I managed to get a simple test passing. It should be enough to fix the rest of out tests, so I'll close this.