NUnit 3.8+ does not finish running tests #2395
Comments
|
@DavidZidar Can you add details of how you're parallelising your tests? By fixtures/test-cases? Do you have a mix of parallel and non-parallel items? This sounds like it could be related to #2261, which was in 3.8. |
|
Ah... I think I saw this yesterday running tests via ReSharper with 3.8.0. |
|
I'm using I tried running the console runner with --workers=1 but it made no difference. |
|
The mix of parallel fixtures and nonparallel items sounds key. Would be really helpful if you could experiment removing a few attributes - and see if you can find a minimum reproducible example. |
|
I found one offending nonparallizable test fixture that for some reason had 2x Unfortunately I am unable to reproduce this problem in a new project and this project is fairly large and I can't share any of the code. Also, now that I am running my tests over and over again some other tests marked with |
|
I tried one more thing, I deleted all 23 test fixtures with the |
|
I Experienced the same behavior since version 3.8.0. After a downgrade to version 3.7.1. everything works fine again. I used the resharper test runner and the visual studio test runner. Both had the same behavior. If I run all tests the execution seems get into a deadlock. If I run the tests separate all tests will be executed properly. My Tests look like this. They simply create a userControl and check if the datacontext can be created correctly by the prism viewmodellocator [Apartment(ApartmentState.STA)]
public class GuiCreationTests
{
private TestBootstrapper testBootStrapper;
private App app;
[SetUp]
public void SetupTests()
{
testBootStrapper = new TestBootstrapper();
testBootStrapper.Run();
if (Application.Current == null)
{
app = new App();
app.InitializeComponent();
}
}
[Test]
public void CreateMemoryViewWithDataContextCorrectly()
{
ViewShouldHaveCorrectDataContext<MemoryView, MemoryViewModel>();
}
[Test]
public void CreateSensorsViewWithDataContextCorrectly()
{
ViewShouldHaveCorrectDataContext<SensorsView, SensorsViewModel>();
}
[Test]
public void CreateRegisterViewWithDataContextCorrectly()
{
ViewShouldHaveCorrectDataContext<RegisterView, RegisterViewModel>();
}
[Test]
public void CreateRegimeViewWithDataContextCorrectly()
{
ViewShouldHaveCorrectDataContext<RegimeView, RegimeViewModel>();
}
[Test]
public void CreateFlashViewWithDataContextCorrectly()
{
ViewShouldHaveCorrectDataContext<FlashView, FlashViewModel>();
}
private void ViewShouldHaveCorrectDataContext<TView, TViewModel>()
where TView : UserControl, new() where TViewModel : BaseViewModel
{
var sut = (UserControl)new TView();
sut.DataContext.Should().NotBeNull();
sut.DataContext.Should().BeOfType<TViewModel>();
}
} |
|
@Maradaehne Are you using any kind of parallelism in the tests? |
|
In the moment i guess there should be no parallelism. I need to check whether the viewmodels create a task tomorrow when i am at work. If it's what you wanted to know. |
|
I was referring to NUnit parallelism. The sample code doesn't have any |
|
No I did not use the |
|
@Maradaehne You're piggy-backing on an issue about parallelism here. 😄 So your problem is something different. We can try a few fixes here first but if they don't work you should file a separate issue. It looks to me as if your problem is either related to use of the STA or your WRT the STA, create a small test that verifies it is actually running in the STA, as called for on the fixture. It probably is, absent parallel execution, but let's make sure. |
|
Hi just to jump in on this as our tests have also started hanging with NUnit 3.8+, we're also using [assembly: Parallelizable(ParallelScope.Fixtures)]. As of yet we haven't tried the NonParallelizable attribute. I'll have to explore the setup and teardown methods used to see if that helps. There is some test inheritance at play so it is a little messy. |
|
I hope this will also be fixed by #2445 If anyone would be able to test out the fix - it can be downloaded from Appveyor here: https://ci.appveyor.com/project/CharliePoole/nunit/build/3.9.0-ci-04450-issue-2438/artifacts |
ChrisMaddock
added
is:bug
pri:critical
|
I just tested the 3.9.0-ci-04450-issue-2438 build and while it seems to improve things quite a lot it's not there yet. The CodeRush test runner seems to complete every time now, so that's good but a few of our tests with the The console runner still locks up every once in a while. |
|
Thanks @DavidZidar. @rprouse found some additional issues with that PR, I think we have more to look at. As before - if you're able to break out any reproducible examples, that would be invaluable. We've fixed the issue as it manifested in #2438 - but it sounds like your larger test suite may be hitting different scenarios. |
|
I'm also intrigued as to your nonparallel tests running in parallel. Can you produce some logs, using Edit: Please do this against the 3.9.0-ci build - that's hopefully moving in the right direction. |
|
Sorry I wasn't clear, some assertions are randomly failing in some of my tests that really shouldn't be failing unless they are being run in parallel and some tests throw NullReferenceException when they really shouldn't. For instance, some of our tests need to configure A question, are tests marked with |
|
Thanks for clarifying. 🙂
That's the theory, yes. It seems like there may be some problems with this in 3.8 however, with the changes to allow test cases to run in parallel. Two things:
|
|
We are only using the attribute on fixtures, not individual tests Here is a single anonymized test-suite node of a test with a failing assertion that really should not fail. Interestingly the start and stop time is the same on all of them, though I'm not sure if that means anything. NUnit 3.5 seems to behave the same. |
|
Sorry, I don't think I was clear with my log comment. Pass the argument From this file, you should be able to see if the tests you expect to be non-parallel, are being run in parallel with other tests, or not. |
|
Oh, right. I didn't see the log files at first. Here is an anonymized snippet, the failing test was running in thread 22 and it seems to be using "Direct strategy" for all the tests in this particular fixture even though the fixture is marked with If this isn't enough maybe I can email you the raw log file in private. |
|
Direct strategy is correct for tests without an attribute. It just means to run on the same thread, without creating a new queue entry. I assume that the other tests interspersed with thread 22 are Parallel tests in the same fixture. |
|
Ah, I understand. No, the tests on the other threads are from other fixtures. |
|
Then we need to figure out why they are running. If the fixture is non-Parallelizable then no others should be running at the same time. Try...
|
|
|
That narrows down the problem. Your non-parallel fixture is running in parallel with some other fixture or fixtures. The hanging is a secondary effect. Ideally, it would be good if you could verify the above, by running with 3.7.1 and examining the output from If that's the case, we can next examine whether the fixtures run erroneously in parallel with the NP fixture have anything in common or if it's a more general problem in the dispatcher. |
|
Right, tried running with 3.7.1 and it runs our tests just fine every time. The difference I'm seeing is that 3.7.1 seems to be running all non parallelizable tests in series on a single thread at the very end unlike 3.8+ which seems to run them whenever. |
|
In a simple situation like this (most fixtures parallel with a few non-parallel and no parallelizable test cases) that's what should happen. All the parallelizable fixtures should complete then the non-parallelizable ones. @ChrisMaddock This seems like the key to me. We made some fixes for the corner cases but it seems like we messed up a very simple case. |
|
@DavidZidar See issue #2454, which seems like a simpler replication of your problem. |
|
@CharliePoole Very nice, we do indeed have a few SetUpFixtures so it looks like it might be the same issue. |
|
The same is happening in our project where we use SetUpFixture and [assembly: Parallelizable(ParallelScope.Fixtures)]. On nunit 3.7.1 it works fine. When I tried to update to 3.8.1 tests hangs in Resharper as well as in the console. |
|
@Suremaker were you one of the people that was running into the intermittent console runner disappearing and causing a socket exception? I wonder if there is crossover here. |
|
@jnm2 I am currently travelling and for 2 weeks I won't be able to check it. Unfortunately I did not check the details when I observed the issue, however it was it was always hanging on each run, after around 12th-16th finished test (so not at the beginning of the run, but always after a specified, constant amount of the run tests)... |
|
Okay, don't worry about it. |
|
It turns out that restoring the queues was starting all queues, so that parallel and non parallel tests were running at the same time. Fixed in #2476. |
|
Anybody is able to tell when we could expect a new version release containing this fix? |
|
@rprouse will have to speak to the schedule. But in any case, it would be great if you could try it out to make sure your case was covered completely. The latest dev build on our myget feed should have the change. |
|
Hi, I just tried build 3.9.0-dev-04639 and while it's working even better still it still is failing every once in a while. The console runner also locked up once. After checking the trace logs it still seems to be running some tests in parallell even though the fixtures in question are marked with |
|
Does the incorrect parallel execution start immediately after a non-parallel fixture completes? That's when we restore the queues after having created new isolated queues for the non-parallel fixture. An excerpt from your log with annotations about how each test is marked would be helpful here. |
|
I'm getting a few hundred lines of startup info, and among them.. .. which is correct so far. The it looks like it begins by running one non parallel test fixture but then starts running parallel tests. Then later on it looks like this.. .. and so on. |
There seems to be some kind of deadlock going on since NUnit 3.8.0 (I have tried 3.8.1 as well). My tests run fine but when all tests are done NUnit never completes the test run and one CPU remains maxed out until the process is killed.
I am experiencing the same problem with both the console runner and the runner in CodeRush for Roslyn.
I attached a debugger to nunit-agent.exe and this is how my threads look like while this is happening, thread 6488 in my screenshot is the one using all the CPU.
The text was updated successfully, but these errors were encountered: