2.12.0

What's Changed

🚀 New adapter for Reqnroll
⚠️ Namespace changes in Allure.NUnit and Allure.Xunit
🎯 New target frameworks of Allure.Xunit
🔬 Improvements
    New implementation of AddScreenDiff (#435)
    Standard failed/broken classification
    Other improvements
🐞 Bug fixes
Deprecations and removals
Other changes
    ⬆️ Dependency Updates
    📦 Packaging changes
    👻 Internal changes
Pull requests and contributors
New contributors

🚀 New adapter for Reqnroll

Meet Allure.Reqnroll - a new Allure adapter for Reqnroll.

Give it a try by installing the Allure.Reqnroll package to your Reqnroll project. Once it's done, you're all set! Run your tests and create the report from the allure-results in the project output directory.

The documentation is currently in progress. Once it's done, the link will be added to this description and the root README file of the repo.

Notable differences with Allure.SpecFlow

Since Reqnroll is a fork of SpecFlow (it was forked back by Gáspár Nagy (gasparnagy), the original creator of SpecFlow), adapters for Reqnroll and SpecFlow have some similarities. However, some notable changes are listed below.

Support for xUnit.net

Allure.Reqnroll correctly works with all unit test frameworks supported by Reqnroll out-of-the-box: NUnit, xUnit.net, and MSTest. In contrast, Allure.SpecFlow currently can't work with xUnit.net due to problems with Allure context isolation (see #431).

Zero configuration

Allure.Reqnroll integrates with Reqnroll in a way that differs from how Allure.SpecFlow integrates with SpecFlow. Because of that, users are not required to alter the Reqnroll configuration to enable the adapter.

SpecFlow, on the other hand, requires users to add an entry to the stepAssemblies list in specflow.json. A missing entry leads to an error and is a common source of problems.

allureConfig.json format changes

Allure.SpecFlow features the specflow section of allureConfig.json where you may define Gherkin tag patterns and configure the table-to-parameter conversion.

In Allure.Reqnroll that section was moved to allure/gherkinPatterns. Some properties were renamed. The packages sub-section was removed (the package label is automatically filled by Allure.Reqnroll).

An example:

{
  "allure": {
    "gherkinPatterns": {
      "stepArguments": {
        "createFromDataTables": false,
        "nameColumn": "ParameterName",
        "valueColumn": "ParameterValue"
      },
      "grouping": {
        "suites": {
          "parentSuite": "allure\\.parentSuite:(.+)",
          "suite": "allure\\.suite:(.+)",
          "subSuite": "allure\\.subSuite:(.+)"
        },
        "behaviors": {
          "epic": "allure\\.epic:(.+)",
          "story": "allure\\.story:(.+)"
        }
      },
      "metadata": {
        "owner": "allure\\.owner:(.+)",
        "severity": "(normal|blocker|critical|minor|trivial)",
        "label": "allure\\.label\\.([^:]+):(.+)"
      },
      "links": {
        "link": "allure\\.link:(.+)",
        "issue": "allure\\.issue:(.+)",
        "tms": "allure\\.tms:(.+)"
      }
    }
  }
}

The schema can be accessed at https://raw.githubusercontent.com/allure-framework/allure-csharp/2.12.0/Allure.Reqnroll/Schemas/allureConfig.schema.json.

Gherkin pattern defaults

Allure.SpecFlow allows users to define patterns to convert Gherkin tags to Allure data. Allure.Reqnroll provides reasonable defaults for such patterns. That also contributes to the zero-configuration experience of Allure Reqnroll (see #439).

The list of the default patterns

The default patterns used by Allure Reqnroll are equivalent to having the following allureConfig.json in place:

{
  "allure": {
    "gherkinPatterns": {
      "stepArguments": {
        "createFromDataTables": false
      },
      "grouping": {
        "suites": {
          "parentSuite": "allure\\.parentSuite:(.+)",
          "suite": "allure\\.suite:(.+)",
          "subSuite": "allure\\.subSuite:(.+)"
        },
        "behaviors": {
          "epic": "allure\\.epic:(.+)",
          "story": "allure\\.story:(.+)"
        }
      },
      "metadata": {
        "owner": "allure\\.owner:(.+)",
        "severity": "(normal|blocker|critical|minor|trivial)",
        "label": "allure\\.label\\.([^:]+):(.+)"
      },
      "links": {
        "link": "allure\\.link:(.+)",
        "issue": "allure\\.issue:(.+)",
        "tms": "allure\\.tms:(.+)"
      }
    }
  }
}

Reliable test plans

Unlike Allure SpecFlow, Allure.Reqnroll doesn't utilize runtime patching to implement test plans. That means features that depend on test plans (e.g., selective run) will work with Allure.Reqnroll in areas where they might not work with Allure.SpecFlow. That includes:

• ARM machines
• Release builds
• New versions of .NET

⚠️ Namespace changes in Allure.NUnit and Allure.Xunit

In this release, we target some historical discrepancies between package names and namespaces of Allure.NUnit and Allure.Xunit.

Old namespaces

Allure.NUnit

The root namespace of Allure.NUnit has been NUnit.Allure up until now. That's the namespace Nick Chursin (unickq) used in his original allure-nunit project until the project was transferred to allure-framework. It matched the package name at the time (NUnit.Allure; it is still available at nuget.org, although deprecated).

Once we changed the package name according to our general convention Allure.<Framework>, the namespace stopped matching the package name. That has created an extra barrier for users who usually expect the root namespace to be the same as the package name. That may also cause users to mix the code of Allure.NUnit and the code of NUnit.

Allure.Xunit

In Allure.XUnit, there are two namespaces: Allure.Xunit and Allure.XUnit (note the casing of u). The public API has been split between both of them at the same time. The attributes have been contained in Allure.Xunit while some other APIs, like AllureStepAttribute - are in Allure.XUnit.

New namespaces

Now, the root namespace of Allure.NUnit is Allure.NUnit. The root namespace of Allure.Xunit is Allure.Xunit.

Additionally, the NUnit.Allure.Core.AllureNUnitAttribute should now be accessed via the root namespace instead of the Core sub-namespace, i.e., as Allure.NUnit.AllureNUnitAttribute.

We advise you to transition accordingly.

Examples

For example, given the following code:

using NUnit.Framework;
using NUnit.Allure.Attributes;
using NUnit.Allure.Core;

namespace MyNamespace;

[AllureNUnit]
class MyTestClass
{
    [Test]
    public void MyTest()
    {
        /* ... */
    }

    [AllureStep]
    void MyStep()
    {
        /* ... */
    }
}

It should be change to the following one:

using NUnit.Framework;
using Allure.NUnit.Attributes;
using Allure.NUnit;

namespace MyNamespace;

[AllureNUnit]
class MyTestClass
{
    [Test]
    public void MyTest()
    {
        /* ... */
    }

    [AllureStep]
    void MyStep()
    {
        /* ... */
    }
}

IDE features like "find & replace all" might assist you in the transition.

The transition becomes trivial if you've added Allure namespaces to the list of global usings. Given the following part of the .csproj file:

<Project Sdk="Microsoft.NET.Sdk">
  <!-- ... -->

  <ItemGroup>
    <Using Include="Allure.Xunit.Attributes">
    <Using Include="Allure.XUnit.Attributes.Steps">

    <!-- Other usings -->
    <!-- ... -->
  </ItemGroup>

</Project>

Just a single character should be changed:

<Project Sdk="Microsoft.NET.Sdk">
  <!-- ... -->

  <ItemGroup>
    <Using Include="Allure.Xunit.Attributes">
    <Using Include="Allure.Xunit.Attributes.Steps"> <!-- Note the casing here -->

    <!-- Other usings -->
    <!-- ... -->
  </ItemGroup>

</Project>

Backward compatibility

The old namespaces remain in place for both Allure.NUnit and Allure.Xunit. Accessing the API through them now triggers the CS0618 warning. While we advise you to transition to new namespaces, you may ignore the warnings and continue using the old ones.

We will remove the public API access from the old namespaces in the next major release.

🎯 New target frameworks of Allure.Xunit

Allure.Xunit now targets netstandard2.0 (down from netstandard2.1) and netcoreapp3.1 (increased from netcoreapp2.0 because of security issues; for example: CVE-2021-26701).

Due to how xUnit.net dependencies are managed, that means that the package works in the following runtimes:

• .NET Core 3.1
• .NET 5.0 or greater

Other runtimes are excepte...

2.11.0

What's Changed

🚀 New features

Runtime API

This release introduces Runtime API that can be used to affect the report during test execution. Previously, users had to utilize AllireLifecycle's methods, such as AllureLifecycle.UpdateTestCase, to achieve that. Now we advise all users to migrate to the new API whenever possible.

The API is accessible as static methods of Allure.Net.Commons.AllureApi. The full list of the class's methods includes:

• Metadata
  • SetTestName(string newName)
  • SetFixtureName(string newName)
  • SetStepName(string newName)
  • SetDescription(string description)
  • SetDescriptionHtml(string descriptionHtml)
  • Labels
    • AddLabels(params Label[] labels)
    • AddLabel(string name, string value)
    • AddLabel(Label newLabel)
    • SetSeverity(SeverityLevel severity)
    • SetOwner(string owner)
    • SetAllureId(int allureId)
    • AddTags(params string[] tags)
  • Links
    • AddLink(string url)
    • AddLink(string name, string url)
    • AddIssue(string url)
    • AddIssue(string name, string url)
    • AddTmsItem(string url)
    • AddTmsItem(string name, string url)
    • AddLink(string name, string type, string url)
    • AddLinks(params Link[] links)
• Hierarchies
  • Suites hierarchy
    • AddParentSuite(string parentSuite)
    • AddSuite(string suite)
    • AddSubSuite(string subSuite)
  • BDD hierarchy
    • AddEpic(string epic)
    • AddFeature(string feature)
    • AddStory(string story)
• Steps
  • Step(string name)
  • Lambda steps
    • Step(string name, Action action)
    • Step<T>(string name, Func<T> function): T
    • async Step(string name, Func<Task> action): Task
    • async Step<T>(string name, Func<Task<T>> function): Task<T>
• Attachments
  • AddAttachment(string name, string type, string path)
  • AddAttachment(string name, string type, byte[] content, string fileExtension = "")
  • AddAttachment(string path, string? name = null)
  • AddScreenDiff(string expectedPng, string actualPng, string diffPng)
• Parameters
  • AddTestParameter(string name, object? value)
  • AddTestParameter(string name, object? value, ParameterMode mode)
  • AddTestParameter(string name, object? value, bool excluded)
  • AddTestParameter(string name, object? value, ParameterMode mode, bool excluded)
  • AddTestParameter(Parameter parameter)

Some less frequently used API methods are available as static methods of Allure.Net.Commons.ExtendedApi. Those are:

• Low-level fixtures API
  • StartBeforeFixture(string name)
  • StartAfterFixture(string name)
  • PassFixture()
  • PassFixture(Action<FixtureResult> updateResults)
  • FailFixture()
  • FailFixture(Action<FixtureResult> updateResults)
  • BreakFixture()
  • BreakFixture(Action<FixtureResult> updateResults)
• Low-level steps API
  • StartStep(string name)
  • StartStep(string name, Action<StepResult> updateResults)
  • PassStep()
  • PassStep(Action<StepResult> updateResults)
  • FailStep()
  • FailStep(Action<StepResult> updateResults)
  • BreakStep()
  • BreakStep(Action<StepResult> updateResults)
• Lambda fixtures
  • Before(string name, Action action)
  • Before<T>(string name, Func<T> function): T
  • async Before(string name, Func<Task> action): Task
  • async Before<T>(string name, Func<Task<T>> function): Task<T>
  • After(string name, Action action)
  • After<T>(string name, Func<T> function): T
  • async After(string name, Func<Task> action): Task
  • async After<T>(string name, Func<Task<T>> function): Task<T>

Note

Allure.Net.Commons.Steps.CoreStepsHelper as its derivatives (NUnit.Allure.Core.StepsHelper and Allure.Xunit.Steps) are now deprecated in favor of the new API. The attachment-related methods from Allure.Net.Commons.AllureLifecycle are also deprecated.

(implements #404 by @delatrie via #414)

Parameter's mode and exclude fields

Allure Report supports masked and hidden parameters of a test. Allure TestOps does, and the future major release of Allure Report will support excluded parameters of a test. Additionally, many other Allure integrations support excluded parameters of a test when calculating historyId of the test.

This release adds the mode and exclude fields to the Allure.Net.Commons.Parameter class as well as some overloads of the new AllureApi.AddTestParameter method to fill those fields. Allure NUnit, Allure xUnit.net, and Allure SpecFlow now ignore excluded parameters when calculating historyId (implements #425 by @delatrie via #414).

🔬 Improvements

• Xml documentation was added to the Allure.Net.Commons package (partially implements #426 by @delatrie via #414).
• Now Allure SpecFlow doesn't interrupt the run if an exception is thrown during an attempt to patch SpecFlow with Harmony to enable selective run. Selective run won't work in such a case. This allows to workaround #434 until a new version of MonoMod that supports net8.0 is released (implements #436 by @delatrie via #437).

🐞 Bug fixes

• BOM was removed from JSON schemas to comply with RFC 8259 (fixes #415 by @delatrie via #414).
• Link.Issue(string name) and Link.Tms(string name) now correctly fill the url field of the link they create. The parameter was renamed to url for both methods (fixes #416 by @delatrie via #414).
• InvalidOperationException that sometimes occurs when Allure SpecFlow is run in a multithreaded environment was fixed (fixes #433 by @delatrie via #414).
• Skipped xUnit theories now get their historyId filled correctly (fixes #422 by @delatrie via #414).

New Contributors

• @slang25 made their first contribution in #411

Full Changelog: 2.10.0...2.11.0

2.10.0

What's Changed

🚀 Core new features

Concurrent steps

Concurrency support by allure-csharp integrations was very limited. In this release we extend it. Steps from parallel/async tests as well as steps from parallel/async methods, sub-steps and operations now work as expected. That fixes a lot of concurrency-related errors that were usually manifested as NullPointerException (fixes #83, #106 and #367 by @delatrie via #371, #383 and #388 by @delatrie via #393).

Selective run

Selective run is now supported for all integrations (implements #372 by @delatrie via #392).

🔬 Core improvements

Improved identification properties

Id properties (fullName, testCaseId, historyId) are fixed to prevent the following potential problems:

• test results of different test cases are interpreted as reruns of a single test case
• multiple test cases are created from test results of a single parametrized test case

(implements #373 and #387 by @delatrie via #395)

Improved parameter formatting

Test and step parameters are now converted to strings using JSON conversion algorithm. It is much more versatile than the plain ToString() call.
The same goes for step parameters interpolated into the step's title (by @delatrie via #395).

Custom type formatters

A user can add an implementation of Allure.Net.Commons.TypeFormatter[T] via the AllureLifecycle.Instance.AddTypeFormatter[T] method. Previously, such formatters were just ignored. Now, they are automatically used to format test and step parameters as well as step titles in case the type of a parameter value is the same as the formatter's generic type argument (fixes #377 by @delatrie via #395).

JSON schema for allureConfig.json

JSON schemas are available for allureConfig.json at https://raw.githubusercontent.com/allure-framework/allure-csharp/<version>/<package>/Schemas/allureConfig.schema.json. You can use the schemas to validate the config file or explore available properties (by @delatrie via #395).

Language and framework labels

The language and framework labels are now added to all test results by all integrations (by @delatrie via #395).

Debug symbols

Debug symbols (.snupkg files) are now available on NuGet for all packages (by @delatrie via #382).

🐞 Core bug fixes

• Fixed InvalidCastException when underlying step returns Task masked as non-generic Task (by @eranikid via #343).

Allure-nunit

🔬 Improvements

• Creating steps in a [OneTimeSetUp] method of a [TestFixture] is now supported (fixes #286 and #374 by @delatrie via #380).
• Attachments are now automatically created from console output (implements #305 by @undron via #306).
• Disable step logging to the console by default (fixes #312 by @undron via #313).
• The AllureId attribute added to the API (by @alekskulakov via #291).

🐞 Bug fixes

• Skipped tests now include historyId and thus preserve their history across runs (fixes #345 by @delatrie via #395).
• An empty package label is added for a test class without a namespace. Previously ArgumentOutOfRangeException was thrown (fixes #375 by @delatrie via #380).

Allure-specflow

🔬 Improvements

• Now if an after-feature hook fails, the scenarios' statuses remain intact instead of being changed to broken. A placeholder scenario is added instead to indicate an error (by @delatrie via #371).

Allure-xunit

🚀 New features

• Allure-xunit now works as a runner reporter. Native xUnit attributes Fact and Theory are supported out-of-the-box (implements #344 by @delatrie via #366).
• Static test methods are now supported (by @delatrie via #366).

🔬 Improvements

• Additional runner reporter can be run side-by-side with allure-xunit (implements #368 by @delatrie via #382). This behavior is configurable (see here).
• Status details of a skipped test now include the skip reason (by @tkeerthivel via #298).

🐞 Bug fixes

• Now the default configuration is used if allureConfig.json is missing. Previously, a catastrophic failure occurred in such a case leading to the test run abortion (fixes #381 by @delatrie via #393).

⬆️ Dependency Updates

• Bump Microsoft.NET.Test.Sdk from 16.10.0 to 17.4.1 by @dependabot in #311
• Bump CsvHelper from 27.1.0 to 30.0.1 by @dependabot in #302
• Bump Newtonsoft.Json from 13.0.1 to 13.0.3 by @dependabot in #335
• Bump SpecFlow from 3.9.8 to 3.9.74 by @dependabot in #331
• Bump coverlet.collector from 3.1.2 to 3.2.0 by @dependabot in #324
• Bump AspectInjector from 2.7.3 to 2.8.1 by @delatrie in #382

👻 Internal changes

• Steps for Allure.XUnit by @neparij in #273
• Add Allure.NUnit package. Big thanks to @unickq for great work on it! by @neparij in #280

New Contributors

• @IvanWR1995 made their first contribution in #271
• @andrey-brovko made their first contribution in #288
• @evgeniy-skt made their first contribution in #284
• @tkeerthivel made their first contribution in #298
• @dependabot made their first contribution in #311
• @danilov-sv made their first contribution in #327
• @askazakov made their first contribution in #330
• @smolchanovsky made their first contribution in #310
• @eranikid made their first contribution in #343
• @delatrie made their first contribution in #366
• @overlord made their first contribution in #314
• @undron made their first contribution in #306
• @alekskulakov made their first contribution in #291

Full Changelog: 2.9.1...2.10.0

2.10.0-preview.1

What's Changed

🚀 New Features

• Replace allure-xunit's custom attributes with runner reporter (fixes #344) by @delatrie in #366
• New context-based state management model for allure-csharp by @delatrie in #371

🔬 Improvements

• Allure-xunit: support for the second reporter and xunit 2.5.0. Updated packaging and AspectInjector (fixes 368) by @delatrie in #382

🐞 Bug Fixes

• NUnit: Fix inconsistent one-time fixtures behavior (fixes #286, #374 and #375) by @delatrie in #380

New Contributors

• @eranikid made their first contribution in #343
• @delatrie made their first contribution in #366

Full Changelog: 2.9.5-preview.1...2.10.0-preview.1

2.9.5-preview.1

What's Changed

🔬 Improvements

• Add parameters and custom name attributes for XUnit by @smolchanovsky in #310

⬆️ Dependency Updates

• Bump Newtonsoft.Json from 13.0.1 to 13.0.3 by @dependabot in #335
• Bump SpecFlow from 3.9.8 to 3.9.74 by @dependabot in #331
• Bump coverlet.collector from 3.1.2 to 3.2.0 by @dependabot in #324
• Bump AspectInjector from 2.7.3 to 2.8.1 by @dependabot in #320

New Contributors

• @askazakov made their first contribution in #330
• @smolchanovsky made their first contribution in #310

Full Changelog: 2.9.4-preview.6...2.9.5-preview.1

2.9.4-preview.6

What's Changed

⬆️ Dependency Updates

• Bump xunit.runner.reporters from 2.4.1 to 2.4.2 by @dependabot in #319

New Contributors

• @danilov-sv made their first contribution in #327

Full Changelog: 2.9.4-preview.5...2.9.4-preview.6

2.9.4-preview.5

Full Changelog: 2.9.4-preview.4...2.9.4-preview.5

2.9.4-preview.4

Full Changelog: 2.9.4-preview.3...2.9.4-preview.4

2.9.4-preview.3

Full Changelog: 2.9.4-preview.2...2.9.4-preview.3

2.9.4-preview.2

Full Changelog: 2.9.4-preview.1...2.9.4-preview.2