nested assert_select reports invalid line number in error backtrace

[kwasimensah]

See rails/rails#6862. Looks like the issue never made it over here and it's still a problem.

[zenspider]

Well… Thank you for actually filing something over here on my side.

From my perspective, this is absolutely intentional. The idea behind it is that custom crafted assertions use more standard assertions to do a majority of the heavy lifting and you’re only interested in the failure from the perspective of the top level assertion.

In the case of assert_select, that flips the script. I don’t have a quick fix for this and would suggest that this should actually fall on rails’ custom backtrace_filter to specifically deal with assert_select.

[matthewd]

I definitely agree this is a (slightly unfortunate) consequence of a desirable intentional behaviour.

It does seem like it'd be nice to be able to avoid it in the rare case it's not what's wanted, though.

@zenspider a couple of approaches come to mind.. maybe one strikes your fancy?

1. A trampoline method (def xx; yield; end) whose presence in the backtrace would "reset" the inside-an-assertion state.
2. A disown mechanism on the assertion exception, which opts caller out of location calculation, for use in a 'rescue, disown, raise' pattern.
3. Automatically give special treatment to "block inside [location]" -- if [location] is under test-author control, then so is a block it contains: which means the inner assertion is something they've written, not part of the outer assertion's implementation.

⌗3 actually seems viable, as I think about it: I'm coming around to the belief this is a general problem with any assertion that yields back to the test. Wisdom of doing so aside, if an author writes an additional assertion inside the block of assert_raises or assert_output, ISTM they would expect that inner line to get the blame.

[zenspider]

Have you filed a bug on rails yet? Can you link to it here? This is not on minitest (at this point).

[matthewd]

@zenspider all of my above suggestions would involve buy-in and/or changes from minitest.

I also think the "that's not the assertion I intuitively expect to be marked as failing" behaviour can be triggered with minitest built-ins only:

if an author writes an additional assertion inside the block of assert_raises or assert_output, ISTM they would expect that inner line to get the blame.

[zenspider]

@matthewd sorry for the delay...

Given:

#!/usr/bin/env ruby -w

require "minitest/autorun"

class TestBug736 < Minitest::Test
  def test_nested_assertions
    assert_silent do
      assert_equal 1, 2
    end
  rescue Minitest::Assertion => e
    Minitest.backtrace_filter.filter e.backtrace
  end
end

The result of that filter is:

["bug736.rb:8:in `block in test_nested_assertions'",
 "bug736.rb:7:in `test_nested_assertions'"]

I could see adding some extra logic to the filter to try to select the deeper of the two if "block in" exists... but the way #location works atm would require some re-jiggering to get that right.

[ghiculescu]

@zenspider for what it's worth, that example behaves a bit differently for me. With the backtrace filter line included, there's no output (it thinks the test passed). Without it:

require "minitest/autorun"

puts "ruby #{RUBY_VERSION}"
puts "minitest #{Minitest::VERSION}"

class TestBug736 < Minitest::Test
  def test_nested_assertions
    assert_silent do # line 8 - minitest claims the failure is here; it should be line 11
    assert_silent do # line 9
    assert_silent do # line 10
      assert_equal 1, 2 # line 11
    end
    end
    end
  end
end

Output:

% ruby /Users/alex/code/test.rb
ruby 2.7.4
minitest 5.14.4
Run options: --seed 12780

# Running:

F

Finished in 0.000979s, 1021.4505 runs/s, 1021.4505 assertions/s.

  1) Failure:
TestBug736#test_nested_assertions [/Users/alex/code/test.rb:8]:
Expected: 1
  Actual: 2