Error: No tests were executed when using aliased signal inputs in an Angular 17 application

[kal-rein]

Summary

While testing an Angular component or directive, if an input is aliased and uses the new signal inputs introduced in Angular 16 Stryker fails to run any test, even those outside the affected element. But, if the same element is tested using an annotated input instead of a signal input Stryker runs all tests correctly, including the affected element.

This is a strange behavior because both inputs works correctly while testing directly using ng test.

I've prepared a repository where the problem can be reproduced here. The latest commit includes both inputs and fails to run Stryker while this commit includes only the annotated input and runs correctly.

Stryker config

{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "_comment": "This config was generated using 'stryker init'. Please see the guide for more information: https://stryker-mutator.io/docs/stryker-js/guides/angular",
  "mutate": [
    "src/**/*.ts",
    "!src/**/*.spec.ts",
    "!src/test.ts",
    "!src/environments/*.ts"
  ],
  "testRunner": "karma",
  "karma": {
    "configFile": "karma.conf.js",
    "projectType": "angular-cli",
    "config": {
      "browsers": [
        "ChromeHeadless"
      ]
    }
  },
  "reporters": [
    "progress",
    "clear-text",
    "html"
  ],
  "concurrency": 16,
  "concurrency_comment": "Recommended to use about half of your available cores when running stryker with angular",
  "coverageAnalysis": "perTest",
  "checkers": ["typescript"],
  "tsconfigFile": "tsconfig.json"
}

Test runner config

// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html

module.exports = function (config) {
  config.set({
    basePath: '',
    frameworks: ['jasmine', '@angular-devkit/build-angular'],
    plugins: [
      require('karma-jasmine'),
      require('karma-chrome-launcher'),
      require('karma-jasmine-html-reporter'),
      require('karma-coverage'),
      require('@angular-devkit/build-angular/plugins/karma')
    ],
    client: {
      jasmine: {
        // you can add configuration options for Jasmine here
        // the possible options are listed at https://jasmine.github.io/api/edge/Configuration.html
        // for example, you can disable the random execution with `random: false`
        // or set a specific seed with `seed: 4321`
      },
      clearContext: false // leave Jasmine Spec Runner output visible in browser
    },
    jasmineHtmlReporter: {
      suppressAll: true // removes the duplicated traces
    },
    coverageReporter: {
      dir: require('path').join(__dirname, './coverage/angular-stryker-input-alias'),
      subdir: '.',
      reporters: [
        { type: 'html' },
        { type: 'text-summary' }
      ]
    },
    reporters: ['progress', 'kjhtml'],
    browsers: ['ChromeHeadless'],
    restartOnFileChange: true
  });
};

Stryker environment

├── @stryker-mutator/karma-runner@8.2.6
├── @stryker-mutator/typescript-checker@8.2.6

├── @angular/cli@17.2.2
├── karma@6.4.3

Test runner environment

ng test

{
  "$schema": "./node_modules/@angular/cli/lib/config/schema.json",
  "version": 1,
  "newProjectRoot": "projects",
  "projects": {
    "angular-stryker-input-alias": {
      "projectType": "application",
      "schematics": {},
      "root": "",
      "sourceRoot": "src",
      "prefix": "app",
      "architect": {
        "build": {
          "builder": "@angular-devkit/build-angular:application",
          "options": {
            "outputPath": "dist/angular-stryker-input-alias",
            "index": "src/index.html",
            "browser": "src/main.ts",
            "polyfills": [
              "zone.js"
            ],
            "tsConfig": "tsconfig.app.json",
            "assets": [
              "src/favicon.ico",
              "src/assets"
            ],
            "styles": [
              "src/styles.css"
            ],
            "scripts": []
          },
          "configurations": {
            "production": {
              "budgets": [
                {
                  "type": "initial",
                  "maximumWarning": "500kb",
                  "maximumError": "1mb"
                },
                {
                  "type": "anyComponentStyle",
                  "maximumWarning": "2kb",
                  "maximumError": "4kb"
                }
              ],
              "outputHashing": "all"
            },
            "development": {
              "optimization": false,
              "extractLicenses": false,
              "sourceMap": true
            }
          },
          "defaultConfiguration": "production"
        },
        "serve": {
          "builder": "@angular-devkit/build-angular:dev-server",
          "configurations": {
            "production": {
              "buildTarget": "angular-stryker-input-alias:build:production"
            },
            "development": {
              "buildTarget": "angular-stryker-input-alias:build:development"
            }
          },
          "defaultConfiguration": "development"
        },
        "extract-i18n": {
          "builder": "@angular-devkit/build-angular:extract-i18n",
          "options": {
            "buildTarget": "angular-stryker-input-alias:build"
          }
        },
        "test": {
          "builder": "@angular-devkit/build-angular:karma",
          "options": {
            "polyfills": [
              "zone.js",
              "zone.js/testing"
            ],
            "tsConfig": "tsconfig.spec.json",
            "assets": [
              "src/favicon.ico",
              "src/assets"
            ],
            "styles": [
              "src/styles.css"
            ],
            "scripts": [],
            "karmaConfig": "karma.conf.js"
          }
        }
      }
    }
  }
}

Your Environment

software	version(s)
node	20.11.0
npm	10.2.4
Operating System	Windows 11 23H2 - WSL2 Ubuntu 22.04.4 LTS

Add stryker.log

stryker.log

[kal-rein]

After some debugging I've confirmed the error is due a mutation introduced by Stryker in the source code, before the ivy compiler parses and bind the template with the controller.

When ran through Stryker the original component is modified to introduce the mutations as expected, this includes the object with the input options which contains the alias. Because of the mutations introduced there, the compiler seems to be unable to parse it and fails leading the tests to fail as well.

Original component

import { Component, input } from '@angular/core';

@Component({
  selector: 'app-new-input',
  standalone: true,
  template: ``,
  styles: ``
})
export class NewInputComponent {
  aliasedInput = input<string | undefined>(undefined, { alias: 'aliased-input' });

  splitInput(): string[] {
    return (this.aliasedInput() ?? '').trim().split(/\s+/);
  }
}

Modified component

// @ts-nocheck
function stryNS_9fa48() {
  var g = typeof globalThis === 'object' && globalThis && globalThis.Math === Math && globalThis || new Function("return this")();
  var ns = g.__stryker__ || (g.__stryker__ = {});
  if (ns.activeMutant === undefined && g.process && g.process.env && g.process.env.__STRYKER_ACTIVE_MUTANT__) {
    ns.activeMutant = g.process.env.__STRYKER_ACTIVE_MUTANT__;
  }
  function retrieveNS() {
    return ns;
  }
  stryNS_9fa48 = retrieveNS;
  return retrieveNS();
}
stryNS_9fa48();
function stryCov_9fa48() {
  var ns = stryNS_9fa48();
  var cov = ns.mutantCoverage || (ns.mutantCoverage = {
    static: {},
    perTest: {}
  });
  function cover() {
    var c = cov.static;
    if (ns.currentTestId) {
      c = cov.perTest[ns.currentTestId] = cov.perTest[ns.currentTestId] || {};
    }
    var a = arguments;
    for (var i = 0; i < a.length; i++) {
      c[a[i]] = (c[a[i]] || 0) + 1;
    }
  }
  stryCov_9fa48 = cover;
  cover.apply(null, arguments);
}
function stryMutAct_9fa48(id) {
  var ns = stryNS_9fa48();
  function isActive(id) {
    if (ns.activeMutant === id) {
      if (ns.hitCount !== void 0 && ++ns.hitCount > ns.hitLimit) {
        throw new Error('Stryker: Hit count limit reached (' + ns.hitCount + ')');
      }
      return true;
    }
    return false;
  }
  stryMutAct_9fa48 = isActive;
  return isActive(id);
}
import { Component, input } from '@angular/core';
@Component({
  selector: 'app-new-input',
  standalone: true,
  template: ``,
  styles: ``
})
export class NewInputComponent {
  aliasedInput = input<string | undefined>(undefined, stryMutAct_9fa48("4") ? {} : (stryCov_9fa48("4"), {
    alias: stryMutAct_9fa48("5") ? "" : (stryCov_9fa48("5"), 'aliased-input')
  }));
  splitInput(): string[] {
    if (stryMutAct_9fa48("6")) {
      {}
    } else {
      stryCov_9fa48("6");
      return stryMutAct_9fa48("7") ? (this.aliasedInput() ?? '').split(/\s+/) : (stryCov_9fa48("7"), (stryMutAct_9fa48("8") ? this.aliasedInput() && '' : (stryCov_9fa48("8"), this.aliasedInput() ?? (stryMutAct_9fa48("9") ? "Stryker was here!" : (stryCov_9fa48("9"), '')))).trim().split(stryMutAct_9fa48("11") ? /\S+/ : stryMutAct_9fa48("10") ? /\s/ : (stryCov_9fa48("10", "11"), /\s+/)));
    }
  }
}

I believe this wasn't a problem using the annotations because they are excluded from being mutated here. This means aliased inputs and outputs using annotations are safe from being modified, but their recently introduced signal counterparts, input and model, are not since they use an object parameter inside the function call.

Now, I'm not sure where to add this exclusion since it is only necessary to support Angular, my best guess is the karma-runner package, although there is recent support for jest too. In the meantime I've made a custom ignore plugin to avoid introducing mutations in objects inside functions named input or model so Stryker can be used with these, it can be seen in action here.

import { declareValuePlugin, PluginKind } from '@stryker-mutator/api/plugin';

export const strykerPlugins = [declareValuePlugin(PluginKind.Ignore, 'angular.signal-model-input-options', {
  shouldIgnore(path) {
    const inputOrModelExpression = path.findParent((path) =>
      path.isCallExpression() &&
      path.node.callee.type === 'Identifier' &&
      (path.node.callee.name === 'input' || path.node.callee.name === 'model'),
    );

    if (path.isObjectExpression() && inputOrModelExpression != null) {
      return 'Angular signal or model input options cannot be mutated as that causes issues with the ivy compiler.';
    }
  },
})];

[nicojs]

Wow, great find and workaround, @kal-rein. I'm very glad to see you were able to use the new 'ignore-plugin' type. These kinds of use cases were exactly why I added it.

StrykerJS should ideally come with an "angular" ignore-plugin and configure this automatically when users choose angular in npm init stryker. @kal-rein, would you feel comfortable contributing this feature?

[kal-rein]

I can give it a try in the upcoming days and implement this workaround as a plugin when the option karma.projectType equals angular-cli. Although, that would only fix the problem when using Karma, if someone were to use Jest I'm pretty sure they will experience this issue, maybe a warning can be added in the Jest documentation so users can add it manually.

Also, I'm not experienced with AST and there is a problem that I don't know how to address in my workaround. If the user were to define another function named input or model which accepts an object as one of the parameters, the plugin will force Stryker to ignore said object when called. Is there any way to know the source of an identifier using AST?

[nicojs]

I can give it a try in the upcoming days and implement this workaround as a plugin when the option karma.projectType equals angular-cli. Although, that would only fix the problem when using Karma, if someone were to use Jest I'm pretty sure they will experience this issue, maybe a warning can be added in the Jest documentation so users can add it manually.

I see that this ignorer plugin would be shipped with @stryker-mutator/core. So you could configure it with

{
  "ignorers": ["angular"]
}

So configure it for jest would be the same.

Btw, don't focus on jest too much; as of today, jest support is still experimental, see https://stryker-mutator.io/docs/stryker-js/guides/angular/#angular-with-experimental-jest-support.

Also, I'm not experienced with AST and there is a problem that I don't know how to address in my workaround. If the user were to define another function named input or model which accepts an object as one of the parameters, the plugin will force Stryker to ignore said object when called. Is there any way to know the source of an identifier using AST?

This plugin would only be active for Angular projects, so I think model(), input(), and output(), etc are most probably from Angular. That being said, I think you should be able to identify that these methods are being called from a class property in a class decorated with a @Component or @Directive decorator. I think those limitations are pretty valid, we can always relax them if users experience issues.

See https://astexplorer.net/#/gist/76889bb1091d862b334873658a938f4f/5802b6615a5b6b01e908f32553afc15e1ad70708 for an example.