Future of solution mode

[dupdob]

Solution mode (i.e the ability to use Stryker against a full solution instead of against a specific project) is getting mature, we need to define what it does as a feature.
It is likely that users will perceive limits as they try this somewhat new feature, and they will ask for fixes and improvements. In order to guide upcoming decision, I suggest to have a poll around what solution mode should do.

[CzBuCHi]

Dont see much difference between 1 and 2 ...

my take would be something like this:

• analyze solution and resolve test-tested project pairs, save those into configuration so developer can review them & stryker can skip this phase next time... (maybe hide saving to config behind CLI flag)
• each test-tested pair is tested separately (maybe even in parallel), save its report into file
  • ability skip given pair when all source files are older than report file (generate list of source files -or- simply look for any file newer than report in test & tested project directory excluding big & obj directories)
  • ability to run single test-tested pair (probably by running stryker from test | tested project directory when stryker configuration with test-tested pairs in solution directory exists)
  • when old report exists show total score comparison to detect regresions
• when all pairs are tested single aggregated report is generated by merging individual reports into one (file paths relative to solution)
  • save this report into solution folder

[rouke-broersma]

Those are all good suggestions however I feel that they go beyond the scope of this question.

The question is, if you run stryker with the solution as the target, do you then expect an aggregated report, or do you expect individual reports. Or both.

The current implementation does option 2, you get one complete aggregated report when targetting a solution.

[CzBuCHi]

I would expect to see single report file with file paths relative to solution, so you can clearly see from where given file came from ...

extra features like skipping unchanged parts, running specific project or IDE integration should be implemented on top of this so i think it should be at least mentioned here ...

[rouke-broersma]

We already have feature requests such as #2323 that cover that, and imo it's not necessarily related to solution mode. There are other run-modes that could benefit from such features. So let's keep that out of this question, but feel free to create new feature requests for features that are missing and haven't yet been requested.

[CzBuCHi]

ok .. in that case i changed my vote to 2 and you can delete this thread if you wish to have only related comments in here ...

[psfinaki]

Well the second option is at least more scalable (otherwise any proper enterprise monorepo risks ending up with dozens/hundreds of reports) and yeah it also can give a beautiful overview of the testing situation.

[CzBuCHi]

another tought about this:
say i have solution with those projects:

First.csproj
First.Tests.csproj       // project reference to First.csproj
Second.csproj            // project reference to First.csproj 
Second.Tests.csproj      // project reference to First.csproj and Second.csproj

*.Tests.csproj are test projects (duh)

currently First.csproj mutants will be tested agains tests from First.Tests.csproj and Second.Tests.csproj which is imho incorrect
(i would preffer to test First.csproj mutants with First.Tests.csproj only)

so my idea is to have:

1. convention that pairs *.csproj with *.Tests.csproj (probably configurable - something like "convention": ".Tests")
2. explicit array of tested-test projects in config file { "First.csproj": "First.Tests.csproj", "Second.csproj": "Second.Tests.csproj" }
   • this will also allows me to ignore some projects entirely

PS: this is just my lastest brainfart, there is probably better solution to this ...

[dupdob]

so, you are favoring solution 1 here: treat a solution as a batch of projects.
What bothers me is that then there is no easy way to test a solution as a whole, using integrated tests as well as unit tests.
If one wants to do this, she/he has to manually add relevant test projects for each sub project, and then find a way to aggregate the result...

[psfinaki]

I'd be quite reluctant (although not entirely opposed) to bringing "convention" options here. What you describe largely feels as an ideal world, having one test project per source project. My personal professional experience shows that this stuff is usually all messed up, hence this might be applicable to quite a low percentage of Stryker users. Also there always will be a person who'd name the test project "ProjectX.Test" instead of "ProjectX.Tests" :)

So this would mean that Stryker would need to test and validate all the paths in question and report errors on that, which is doable but something tells me there would be weird edge cases there.

[CzBuCHi]

@dupdob

• using integrated tests as well as unit tests.
  it never occurred to me to use stryker to validate with integration tests - in my mind you write unit tests only to get all mutants killed, then integration tests that verify your sanity :) (i would mark them as ignored by stryker [attribute / another test project], so they do not participate in genocide)

• aggregate the result...
  looking at json structure it looks like 'simple' merging of files into single object (maybe with property name tweaks so it include project name)

@psfinaki

• feels as an ideal world
  agree, so maybe developer would need to explicitly enable this behavior in configuration - probably when starting new project, where enforcing this would not be such PITA ...

• Also there always will be a person who'd name the test project "ProjectX.Test" instead of "ProjectX.Tests" :)
  in that case that person would also add something like "test project suffix": ".Test" to configuration file (default would be ".Tests")

---

PS: how do you quote someone in response? im sure there is better was than hijacking code blocks ..

[rouke-broersma]

@CzBuCHi
Also there always will be a person who'd name the test project "ProjectX.Test" instead of "ProjectX.Tests" :)
in that case that person would also add something like "test project suffix": ".Test" to configuration file (default would be ".Tests")

Configuration is already ballooning quite enough. The only additional project relationship configuration I'm willing to add at this point is multi-level configuration file merging #2323.

[psfinaki]

PS: how do you quote someone in response? im sure there is better was than hijacking code blocks ..

@CzBuCHi Like this :)

[dupdob]

I think we have a consensus here. I keep the discussion open until work is engaged