Q&A General discussion

[SteveGilham]

A place to ask questions and share ideas

Mainly things that would end up as an issue labelled "question" and end up as FAQ items, but possibly for floating concepts that may become "enhancement" issues at some point.

[axunonb]

Hi,
This week I came across a problem where we had resource files (resx) included in the unit tests.
It took me quite a while to figure out, that failing unit tests (not finding the localized DLLs) went back to missing /p:AltCoverInplace=true /p:AltCoverForce=true as dotnet test parameters. I'm wondering, why this is not the default behavior of AltCover.
Would you please elaborate why?
Thank you!

[SteveGilham]

In the case of /p:AltCoverForce, my reasoning was that it highlights to the developer (and here I assume that dedicated build/CI machines will start from clean always) that something had gone amiss with an earlier run, and fail safe without destroying anything that might be of use for forensic analysis of that earlier failure.

The case of /p:AltCoverInPlace is somewhat more complicated. Originally, the default was false everywhere except the dotnet test scenario, which could use the MSBuild context to perform the necessary tidy and copy-back operation after the tests had run. It then transpired that /p:AltCoverInPlace=true does not play well with the concurrent instrument-and-test behaviour of dotnet test [multipletestprojects].sln /p:AltCover="true" --output [commonArtifactsFolder] -- an entirely plausible use-case.

Making the in-place operation be opt-in was then simplest way to make the tool "just work"; that change also avoids the need to delete or overwrite files in the project output directory, when restoring the uninstrumented original files at the end of the process. Not only does this save two copy operations (copy the as-built files out, and copy them back), it also avoids the occasional glitch where the delete operation is attempted before the in-use state has been relinquished at the file-system level (the usual sluggishness where a file state can linger for a humanly perceptible time, often encountered when performing recursive directory deletes). Supporting this change, every file mentioned in the project as CopyToOutputDirectory either "always", or "if newer", is also copied to the appropriate relative directory.

Having done that, and called it a day, I see that I have indeed overlooked the satellite assembly copying, despite having unit tests for AltCover's own localization support (which go directly to the as-built location and thus bypass this). The same would also be true of any platform specific (e.g. x86 or x64) subdirectories too, now I come to think of it.

So I have a bug affecting all invocation methods to fix, involving recursively copying subdirectories as part of the instrumentation process, with the workround being to explicitly select the in-place option.

[axunonb]

Hi, thanks a lot for your detailed feedback, which was really helpful for understanding. For now I'll just work with AltCoverInPlace, until there is a bug fix available.

[SteveGilham]

Here's a pre-release (but feature complete) build with the copy process now recursively copying sub-directories of the input directory (but excluding the output directory if it is a sub-directory too)

altcover.8.2.399-github-pre.zip

[SteveGilham]

Now available in release v8.2.825.

[axunonb]

Wonderful, Steve, works perfectly now, with all satellite assemblies nicely in place.
Thanks for your ultra fast fix.

[Lanayx]

Hi, what is a most up-to-date way to include just several assemblies (Say Library1 and Library2)?

[SteveGilham]

Taken literally, --assemblyExcludeFilter "?^Library(1|2)$" will exclude all other assemblies than Library1 and Library2

In general while the default is to exclude all items matching the regex, the leading ? -- an otherwise invalid character -- inverts the filter to exclude all non-matching items.

[Lanayx]

Another question - is it possible to make a solution wide report, e.g. choose 2 UnitTests projects and merge their reports into one file?

[SteveGilham]

I'm assuming that the motivation here is to show the aggregate coverage for a shared assembly after a series of tests.

One way is to do the individual tests e.g. by dotnet test MySolution.sln /p:AltCoverReport=<report folder>/$(ProjectName).coverage.xml then consolidate the reports from that folder with the Merge-OpenCover cmdlet -- that groups the assemblies by their hash, so it doesn't matter that the same assembly may be copied to different locations.

Another way that should work -- but has not been rigorously tested -- would be to forgo the convenience of the dotnet test integration and use the command line facility to specify multiple input and output directories against one report also with the --save option, then run the unit tests against the various instrumented locations, and then altcover runner --collect specifying any one of the output locations for the recorder directory. This relies on the instrumentation selecting just one of any multiple instances of an assembly (again grouping by hash) to instrument, and copying it to all relevant outputs.

Finally, depending what you are looking for in a consolidated report, the reportgenerator tool could do a sufficient job to replace the Merge-OpenCover cmdlet, also handling other coverage output formats.

[ajkimberley]

Hi,

I'm trying to run AltCover against a .NET (Framework) 4.8 project (the fsproj files have been converted to SDK-style). Should I be able to run AltCover using dotnet test -p:AltCover=true?

When I run dotnet test against my unit test library, all tests pass. But when I run it with -p:AltCover=true they all fail with the following error:

System.IO.FileLoadException : Could not load file or assembly 'FSharp.Core, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
----> System.IO.FileLoadException : Could not load file or assembly 'FSharp.Core, Version=4.4.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

All fs projects in the solution are using FSharp.Core 7.0.300. I'm probably doing something wrong but haven't been able to figure out what yet. Any ideas?

Thanks.

[SteveGilham]

That gives me somewhere to start looking. Meanwhile release 8.6.61 is now out, and there is a small chance that might have some effect on the issue (tidying of some obsolete code to do with dotnet test integration that would impact on F# projects).

[ajkimberley]

It works!

I updated to v8.6.61 and now the tests all pass as expected.

It would be great to know what was causing the issue if you ever find out. Please let me know if I can be of any use in that regard.

Thanks!

[SteveGilham]

It was a piece of MSBuild script from the days before I had a reliable static linkage of FSharp.Core in the recorder, so I needed to move a copy of that assembly around during instrumentation while attempting not to interfere with any version already present from the system under test. It was only while looking at this issue that I realised I hadn't tidied that bit of script away, and that it was left there potentially doing something unexpected.

Your report was a "something unexpected" of a plausible kind, so I removed that part of the MSBuild script, and saw that, as expected, doing so had no ill effect on the build validation process, and thought it would be worth your trying. I'm glad that the hunch proved right, though not happy that I'd left a problem there for several years.

[ajkimberley]

Superb! Thanks for the explanation and for your help. And I wouldn't be too hard on yourself - this sounds like quite a complex project to manage.

[ajkimberley]

The plot thickens. The tests are now running fine locally, but they are failing with the same error on the build pipeline - same branch, using dotnet test -p:AltCover=true. I currently have no idea what the relevant difference could be between the two environments. But will keep you posted if I find out what the root cause is in case it indicates something that needs fixing in AltCover.