Unhandled SystemExit causes susbequent tests to silently not run.

[marcparadise]

This appears related to the change for #2063.

When testing components that might be expected to call Kernel.exit in the normal course of operations, a SystemExit is raised. The policy of not rescuing SystemExit in this case means any tests that were supposed to run after that point will not run -- and will not report a failure. The only indication is that the number of tests executed is lower than it should be.

In automated test runs, this is almost impossible to catch - and even in local runs when the difference is a matter of just a handful of tests not being run, it's easy to miss.

Simple repro:

describe "a scenario that should run two tests and fail one" do 
  it "should report this as failed and  keep running" do 
    Kernel.exit 1 
  end 
  it "does not do either, and I see a succesful exit with one test passed" do
  end 
end

Result:

rspec test_spec.rb

Finished in 0.00039 seconds (files took 0.06405 seconds to load)
1 example, 0 failures

The easy answer is to also handle SystemExit, but that would re-introduce #2063. That led me to consider submitting a patch where a SystemExit handling option could be specified via configuration., and handled similarly to:

 rescue Support::AllExceptionsExceptOnesWeMustNotRescue => ex
...
rescue Support::AnyExceptionWeMustNotRescue  => ex
   raise unless RSpec.configuration.allow_exceptions.include?(ex.class) 

Another option would be a final check of number of executed tests against the number we expected to execute, failing on a mismatch. This may not catch scenarios related to nested runs though.

Before submitting a patch, I wanted to open this issue for feedback and discussion of best approach.

[myronmarston]

Kernel.exit is an instruction to the ruby interpreter to exit. I don't think it makes sense for RSpec to stop you from being able to exit. If you need to test code that does this, you have a few options.

One is to explicit expect that SystemExit is raised:

expect { Kernel.exit 1 }.to raise_error(SystemExit)

Another is to handle this globally with an around hook:

RSpec.configure do |rspec|
  rspec.around(:example) do |ex|
    begin
      ex.run
    rescue SystemExit => e
      puts "Got SystemExit: #{e.inspect}. Ignoring"
    end
  end
end

You could of course choose to convert it to a failure (instead of ignoring it) by raising a different error in the rescue clause.

Given how easy it is to to handle this however you want, I don't think it makes sense for us to provide an option for this. After all, such an option would be less flexible than what you can already accomplish with an around hook.

[JonRowe]

I mostly agree here, if we were to do anything to improve this behaviour it would be to shutdown rspec properly, and then continue to exit.

[marcparadise]

One is to explicit expect that SystemExit is raised:

This is true, and is the usual method. But in the case where someone adds a test that doesn't handle this properly, we can no longer rely on the validity of our tests - something that should always be trustworthy.

Another is to handle this globally with an around hook:

Thanks for that info - I didn't know about this, and will be doing this as a sanity check in my current scenario.

However, it has a similar limitation. If you have a bunch of well behaving tests that don't generate cause for adding this in the first place and someone inadvertently adds a new test that does raise a SystemError, you can't rely on your tests running all the way through.

However, carrying the line of thought further: I assume any handling for this in rspec-core would need to be opt-in to avoid breaking the forked test scenarios. And if it has to be enabled anyway, then there's no way around the need for awareness (whether setting an option, or implementing your suggestion above locally).

While I still some level of default handling of SystemExit in order to minimally guarantee the validity of the test results is important, I can see how it'd introduce other difficulties around the previously fixed issue. What about ensuring it outputs something AllExceptionsThatMustNotBeHandled is generated, so that someone looking at the test result would get some level of hint that all may not be well?

I mostly agree here, if we were to do anything to improve this behaviour it would be to shutdown rspec properly, and then continue to exit.

That also seems reasonable as it would (I think) give a way of knowing that something didn't do what it was supposed to.

[myronmarston]

I'd really prefer not to add an option to RSpec to handle SystemExit in particular situations. If someone is writing code that calls Kernel.exit they should understand what that does. I don't think it makes sense for RSpec to have an option to change Kernel.exit to no longer do what the ruby docs says it does.

I mostly agree here, if we were to do anything to improve this behaviour it would be to shutdown rspec properly, and then continue to exit.

That also seems reasonable as it would (I think) give a way of knowing that something didn't do what it was supposed to.

I think improving how RSpec shuts down in this situation would be a good idea. Want to take a stab at it, @marcparadise?

[marcparadise]

@myronmarston sounds good to me, I'll take a look into it over the next couple of evenings.

[xaviershay]

Closing as stale, please feel free to reopen with a PR!

(fwiw I'm pretty negative on us doing anything here. at_exit hooks have caused so many headaches, particularly with interactions between different libraries such as simplecov.)