Mutant still survive although test failed

[AkiraNoob]

Question

Hello everyone, this is my first question so if i make any mistake, pls let me know. Peace
Back to the question, can s.o explain for me why this mutant keep survived although when i try that mutant to the code itself, the test is failed.
My repo: https://github.com/AkiraNoob/kiem_chung

Test case:

    describe('Given valid payload', () => {
      const spyedRefreshCreate = jest.spyOn(RefreshTokenModel, 'create');
      const spyedRefreshDelete = jest.spyOn(RefreshTokenModel, 'deleteMany');
      const spyedFindUser = jest.spyOn(UserModel, 'findOne');
      const spyedBcryptCompare = jest.spyOn(bcryptCommon, 'bcryptCompareSync');
      it('should return statusCode 200 and data should contain {token, refreshToken} and message is "Login successfully"', async () => {
        const data = {
          token: {
            token: expect.any(String),
            expires: expect.any(String),
          },
          refreshToken: {
            token: expect.any(String),
            expires: expect.any(String),
          },
        };

        await expect(authService.loginWithEmailAndPassword(mockLocalLoginPayload)).resolves.toStrictEqual({
          statusCode: EHttpStatus.OK,
          data,
          message: expect.stringMatching('Login successfully'),
        });

        expect(spyedBcryptCompare).toHaveBeenCalledWith(mockLocalLoginPayload.password, expect.any(String));
        expect(spyedBcryptCompare).toHaveReturnedWith(true);

        expect(spyedFindUser).toHaveBeenCalledWith({
          email: expect.any(String),
        });

        expect(spyedRefreshDelete).toHaveBeenCalledWith({
          userId: expect.any(mongoose.Types.ObjectId),
        });

        return expect(spyedRefreshCreate).toHaveBeenCalledWith({
          userId: expect.any(mongoose.Types.ObjectId),
          refreshToken: expect.any(String),
          expiredAt: expect.any(String),
        });
      });
    });

service:

loginWithEmailAndPassword: async (
    reqBody: TLocalLoginPayload,
  ): Promise<TServiceResponseType<{ token: TReturnJWTType; refreshToken: TReturnJWTType }>> => {
    const user = await UserModel.findOne({ email: reqBody.email }).select('+password');

    if (!user) {
      throw new AppError(EHttpStatus.BAD_REQUEST, 'Wrong email');
    }

    if (!bcryptCompareSync(reqBody.password, user.password)) {
      throw new AppError(EHttpStatus.BAD_REQUEST, 'Wrong password');
    }

    const userData = {
      id: user._id.toString(),
      email: user.email,
      fullName: user.fullName,
    };

    const token = signJWT(userData);
    const refreshToken = signRefreshJWT(userData);

    await RefreshTokenModel.deleteMany({ userId: new mongoose.Types.ObjectId(userData.id) });

    await RefreshTokenModel.create({
      userId: user._id,
      refreshToken: refreshToken.token,
      expiredAt: refreshToken.expires,
    });

    return {
      data: {
        token,
        refreshToken,
      },
      statusCode: EHttpStatus.OK,
      message: 'Login successfully',
    };
  },

mutant

- await RefreshTokenModel.deleteMany({ userId: new mongoose.Types.ObjectId(userData.id) });
+ await RefreshTokenModel.deleteMany({})

Stryker environment

+-- @stryker-mutator/core@7.3.0
+-- @stryker-mutator/jest-runner@7.3.0
+-- @stryker-mutator/typescript-checker@7.3.0

+--jest@29.7.0
+--supertest@6.3.3

Additional context

my stryker config:

{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "_comment": "This config was generated using 'stryker init'. Please take a look at: https://stryker-mutator.io/docs/stryker-js/configuration/ for more information.",
  "packageManager": "npm",
  "reporters": ["html", "progress"],
  "testRunner": "jest",
  "testRunner_comment": "Take a look at (missing 'homepage' URL in package.json) for information about the jest plugin.",
  "coverageAnalysis": "perTest",
  "mutate": ["src/**"],
  "ignoreStatic": true,
  "checkers": ["typescript"],
  "tsconfigFile": "tsconfig.json",
  "typescriptChecker": {
    "prioritizePerformanceOverAccuracy": false
  }
}

my jest config

/** @type {import('ts-jest').JestConfigWithTsJest} */
module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  testMatch: ['**/**/*.test.ts'],
  forceExit: true,
  verbose: true,
  clearMocks: true,
  setupFiles: ['<rootDir>/.jest/setEnv.ts'],
  coverageDirectory: 'reports/coverage',
  testTimeout: 5000,
};

Also node that i currently use node 18.18.2

[odinvanderlinden]

Hi @AkiraNoob , thanks for opening this issue. I'll have a look at your source code and try to figure out what is happening.

[odinvanderlinden]

So far there is one thing that has caught my attention, when i run Stryker with the change that should let a test fail. The initial test run fails. This means that the problem is not with finding the test. Besides that i don't know what's going wrong here. @nicojs do you have a clue?

[nicojs]

I think i've found the issue. Your tests seem to share state.

This change:

--- a/server/src/api/service/auth.service.ts
+++ b/server/src/api/service/auth.service.ts
@@ -68,7 +68,7 @@ const authService = {
       fullName: verifiedRefreshToken.fullName,
     };
 
-    await RefreshTokenModel.deleteMany({ userId: new mongoose.Types.ObjectId(userData.id) });
+    await RefreshTokenModel.deleteMany({});

Makes your tests fail when you run them all serially (the test 'Given valid payload should return statusCode 200 and data should contain {token, refreshToken} and message is "Login successfully"' fails), but the failing test passes when you focus it:

--- a/server/__test__/api/service/auth.service.test.ts
+++ b/server/__test__/api/service/auth.service.test.ts
@@ -55,7 +55,7 @@ describe('Testing auth service', () => {
       const spyedRefreshDelete = jest.spyOn(RefreshTokenModel, 'deleteMany');
       const spyedFindUser = jest.spyOn(UserModel, 'findOne');
       const spyedBcryptCompare = jest.spyOn(bcryptCommon, 'bcryptCompareSync');
-      it('should return statusCode 200 and data should contain {token, refreshToken} and message is "Login successfully"', async () => {
+      it.only('should return statusCode 200 and data should contain {token, refreshToken} and message is "Login successfully"', async () => {

Focussing tests is what StrykerJS does when you set coverageAnalysis to "perTest".

I would suggest making your unit test independent of each other. You can do this by removing the global state, and instead initialing variables inside beforeEach hooks.

For example:

--- a/server/__test__/api/service/auth.service.test.ts
+++ b/server/__test__/api/service/auth.service.test.ts
@@ -11,20 +11,26 @@ import UserModel from '../../../src/model/user';
 import { TLocalLoginPayload, TRegisterPayload } from '../../../src/types/api/auth.types';
 import { TUserSchema } from '../../../src/types/schema/user.schema.types';
 
-const userPayload: TUserSchema = {
-  email: 'tester.001@company.com',
-  password: bcryptCommon.bcryptHashSync('Tester@001'),
-  fullName: 'Tester 001',
-  avatar: 's3_img_string',
-  dateOfBirth: new Date(),
-};
-
-const mockRegisterPayload: TRegisterPayload = {
-  ...omit(userPayload, ['avatar', 'dateOfBirth']),
-  password: 'Tester@001',
-};
-
-const mockLocalLoginPayload: TLocalLoginPayload = omit(mockRegisterPayload, 'fullName');
+let userPayload: TUserSchema;
+let mockRegisterPayload: TRegisterPayload;
+let mockLocalLoginPayload: TLocalLoginPayload;
+
+beforeEach(() => {
+  userPayload = {
+    email: 'tester.001@company.com',
+    password: bcryptCommon.bcryptHashSync('Tester@001'),
+    fullName: 'Tester 001',
+    avatar: 's3_img_string',
+    dateOfBirth: new Date(),
+  };
+
+  mockRegisterPayload = {
+    ...omit(userPayload, ['avatar', 'dateOfBirth']),
+    password: 'Tester@001',
+  };
+
+  mockLocalLoginPayload = omit(mockRegisterPayload, 'fullName');
+});

[AkiraNoob]

thanks @nicojs and @odinvanderlinden for help me out, i really appriciate it! Seem like that is the reason

[AkiraNoob]

sorry for re-open this issue but somehow i think it not solves my case, i provide a brief version of my code:

const generateMockPayload = () => {
  const userPayload: TUserSchema = {
    email: 'tester.001@company.com',
    password: bcryptCommon.bcryptHashSync('Tester@001'),
    fullName: 'Tester 001',
    avatar: 's3_img_string',
    dateOfBirth: new Date(),
  };

  const mockRegisterPayload: TRegisterPayload = {
    ...omit(userPayload, ['avatar', 'dateOfBirth']),
    password: 'Tester@001',
  };

  const mockLocalLoginPayload: TLocalLoginPayload = omit(mockRegisterPayload, 'fullName');

  return {
    userPayload,
    mockRegisterPayload,
    mockLocalLoginPayload,
  };
};

describe('Testing auth service', () => {
  beforeAll(async () => {
    const mongoServer = await MongoMemoryServer.create();
    await mongoose.connect(mongoServer.getUri());
  });

  afterAll(async () => {
    await mongoose.disconnect();
    await mongoose.connection.close();
  });

  beforeEach(() => {
    jest.clearAllMocks();
  });

  describe('Register service', () => {
    afterAll(async () => {
      const { userPayload } = generateMockPayload();

      await UserModel.deleteMany({ email: userPayload.email });
    });

    describe('Given valid payload', () => {
      it('should return statusCode 200 and data is null and message is "Register successfully"', async () => {
        const { mockRegisterPayload } = generateMockPayload();

        const spyedUserModelCreate = jest.spyOn(UserModel, 'create');
        const resolveData = {
          statusCode: EHttpStatus.OK,
          data: null,
          message: expect.stringMatching('Register successfully'),
        };

        await expect(authService.register(mockRegisterPayload)).resolves.toStrictEqual(resolveData);
        return expect(spyedUserModelCreate).toHaveBeenCalledWith({
          ...mockRegisterPayload,
          password: expect.any(String),
        });
      });
    });

  });
});

[AkiraNoob]

i belive that if it runs to return statement, it means that API is resolves successfully and MUST return message in image

[AkiraNoob]

Sorry for my mistake, i try research more and try some approaches: #2989 #3068 and trouble shooting but my issue still occurs. I have created a smaller repository to focus on a single unit here. You can see that if i run

npm run mutation-test

1 mutant still survive but the test is failed (i try to isolate the testcase as suggest). Hope this repository will help to solve my issue

[AkiraNoob]

Sorry for my hurry but any update on this?

[odinvanderlinden]

Hi @AkiraNoob sorry for the late response, I haven't had a lot of time to work on Stryker lately. I have taken a look at your new repository. And it seems that you still make use of shared state. For every single test you should clear your in memory database. This is not the case right now. For instance the after all method where you clear the user should be a AfterEacht method to make sure you start every test with a clean sheet. I hope this helps!