Automate Graceful Recovery NFR #1832

bjee19 · 2024-04-17T00:20:36Z

Problem: We want to automate the Graceful Recovery NFR.

Solution: Extend existing automation to cover the following test cases in the Graceful Recovery NFR: "restart nginx-gateway container" and "restart NGINX container".

Testing: Test correctly matches results of manual run of graceful-recovery NFR test.

Closes #1365

Output of test w/o Skip:

Will run 2 of 12 specs
------------------------------
[BeforeSuite] 
/home/username/nginx-gateway-fabric/tests/suite/system_suite_test.go:238
[BeforeSuite] PASSED [73.895 seconds]
------------------------------
SSSSSSSSSS
------------------------------
Graceful Recovery test recovers when NGF container is restarted [nfr, graceful-recovery]
/home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:70
• [45.174 seconds]
------------------------------
Graceful Recovery test recovers when nginx container is restarted [nfr, graceful-recovery]
/home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:74
  [FAILED] in [It] - /home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:233 @ 04/30/24 18:39:04.063
• [FAILED] [3.790 seconds]
Graceful Recovery test [It] recovers when nginx container is restarted [nfr, graceful-recovery]
/home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:74

  [FAILED] /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
  /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
  /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
  10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
  /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
  /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
  /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
  /docker-entrypoint.sh: Configuration complete; ready for start up
  2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/run/nginx/nginx-config-version.sock failed (98: Address in use)
  2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/lib/nginx/nginx-502-server.sock failed (98: Address in use)
  2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/lib/nginx/nginx-500-server.sock failed (98: Address in use)
  2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/run/nginx/nginx-status.sock failed (98: Address in use)
  2024/04/30 18:39:03 [notice] 92#92: try again to bind() after 500ms
  2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/run/nginx/nginx-config-version.sock failed (98: Address in use)
  2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/lib/nginx/nginx-502-server.sock failed (98: Address in use)
  2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/lib/nginx/nginx-500-server.sock failed (98: Address in use)
  2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/run/nginx/nginx-status.sock failed (98: Address in use)
  2024/04/30 18:39:03 [notice] 92#92: try again to bind() after 500ms

  Expected
      <string>: /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
      /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
      /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
      10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
      /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
      /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
      /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
      /docker-entrypoint.sh: Configuration complete; ready for start up
      2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/run/nginx/nginx-config-version.sock failed (98: Address in use)
      2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/lib/nginx/nginx-502-server.sock failed (98: Address in use)
      2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/lib/nginx/nginx-500-server.sock failed (98: Address in use)
      2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/run/nginx/nginx-status.sock failed (98: Address in use)
      2024/04/30 18:39:03 [notice] 92#92: try again to bind() after 500ms
      2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/run/nginx/nginx-config-version.sock failed (98: Address in use)
      2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/lib/nginx/nginx-502-server.sock failed (98: Address in use)
      2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/lib/nginx/nginx-500-server.sock failed (98: Address in use)
      2024/04/30 18:39:03 [emerg] 92#92: bind() to unix:/var/run/nginx/nginx-status.sock failed (98: Address in use)
      2024/04/30 18:39:03 [notice] 92#92: try again to bind() after 500ms
      
  not to contain substring
      <string>: emerg
  In [It] at: /home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:233 @ 04/30/24 18:39:04.063

  Full Stack Trace
    github.com/nginxinc/nginx-gateway-fabric/tests/suite.checkContainerLogsForErrors({0xc0005d6090, 0x2e})
        /home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:233 +0x1ba
    github.com/nginxinc/nginx-gateway-fabric/tests/suite.runRecoveryTest({0x1bc67de, 0x1c}, {0x1bc9929, 0x1e}, {0xc0005d6090, 0x2e}, {0x1ba931f, 0x5}, {0xc000228800, 0x4, ...}, ...)
        /home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:92 +0x112
    github.com/nginxinc/nginx-gateway-fabric/tests/suite.init.func3.5()
        /home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:75 +0x59
------------------------------
[AfterSuite] 
/home/username/nginx-gateway-fabric/tests/suite/system_suite_test.go:264
[AfterSuite] PASSED [56.574 seconds]
------------------------------

Summarizing 1 Failure:
  [FAIL] Graceful Recovery test [It] recovers when nginx container is restarted [nfr, graceful-recovery]
  /home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:233

Ran 2 of 12 Specs in 179.434 seconds
FAIL! -- 1 Passed | 1 Failed | 0 Pending | 10 Skipped
--- FAIL: TestNGF (179.44s)
FAIL

Output of test with Skip:

• [26.809 seconds]
------------------------------
Graceful Recovery test recovers when nginx container is restarted [nfr, graceful-recovery]
/home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:81
  [SKIPPED] in [It] - /home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:84 @ 05/01/24 15:59:36.41
S [SKIPPED] [0.872 seconds]
Graceful Recovery test [It] recovers when nginx container is restarted [nfr, graceful-recovery]
/home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:81

  [SKIPPED] Test currently fails due to this issue: https://github.com/nginxinc/nginx-gateway-fabric/issues/1108
  In [It] at: /home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:84 @ 05/01/24 15:59:36.41

  Full Stack Trace
    github.com/nginxinc/nginx-gateway-fabric/tests/suite.init.func3.5()
        /home/username/nginx-gateway-fabric/tests/suite/graceful_recovery_test.go:84 +0x92
------------------------------
SS
------------------------------
[AfterSuite] 
/home/username/nginx-gateway-fabric/tests/suite/system_suite_test.go:264
[AfterSuite] PASSED [63.160 seconds]
------------------------------

Ran 1 of 12 Specs in 163.326 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 11 Skipped
PASS

Checklist

Before creating a PR, run through this checklist and mark each as complete.

I have read the CONTRIBUTING doc
I have added tests that prove my fix is effective or that my feature works
I have checked that all unit tests pass after adding my changes
I have updated necessary documentation
I have rebased my branch onto main
I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

NONE

tests/framework/request.go

deploy/helm-chart/templates/deployment.yaml

tests/suite/graceful_recovery_test.go

sjberman

Be sure to move all existing results for this test to the results/ directory, and you can delete the graceful-recovery/ directory.

tests/framework/request.go

tests/suite/graceful_recovery_test.go

tests/framework/request.go

tests/suite/manifests/graceful-recovery/node-debugger-job.yaml

tests/suite/graceful_recovery_test.go

bjee19 · 2024-05-01T17:42:49Z

Be sure to move all existing results for this test to the results/ directory, and you can delete the graceful-recovery/ directory.

Should I put this on hold until #1910 and #1901 go through?

sjberman · 2024-05-01T18:02:35Z

@bjee19

Should I put this on hold until #1910 and #1901 go through?

Yep

tests/suite/graceful_recovery_test.go

github-actions bot added tests Pull requests that update tests helm-chart Relates to helm chart labels Apr 17, 2024

github-advanced-security bot found potential problems Apr 17, 2024

View reviewed changes

tests/framework/request.go Fixed Show resolved Hide resolved

pleshakov requested changes Apr 17, 2024

View reviewed changes

bjee19 mentioned this pull request Apr 17, 2024

Keep OTel collector version updated in product telemetry functional test #1665

Open

github-actions bot removed the helm-chart Relates to helm chart label Apr 25, 2024

bjee19 force-pushed the tests/automate-graceful branch from 4eb991d to a1e885a Compare April 25, 2024 21:30

bjee19 commented Apr 25, 2024

View reviewed changes

tests/suite/graceful_recovery_test.go Outdated Show resolved Hide resolved

bjee19 marked this pull request as ready for review April 25, 2024 21:36

bjee19 requested a review from a team as a code owner April 25, 2024 21:36

bjee19 requested review from pleshakov and sjberman April 25, 2024 22:06

sjberman reviewed Apr 26, 2024

View reviewed changes

bjee19 commented Apr 26, 2024

View reviewed changes

tests/suite/graceful_recovery_test.go Show resolved Hide resolved

bjee19 force-pushed the tests/automate-graceful branch from 9bf650c to 92d4ebe Compare April 29, 2024 20:27

github-advanced-security bot found potential problems Apr 29, 2024

View reviewed changes

tests/framework/request.go Dismissed Show dismissed Hide dismissed

pleshakov reviewed Apr 29, 2024

View reviewed changes

bjee19 mentioned this pull request May 1, 2024

NGF Pod cannot recover if NGINX master process fails without cleaning up #1108

Open

bjee19 requested review from pleshakov and sjberman May 1, 2024 16:02

pleshakov reviewed May 1, 2024

View reviewed changes

tests/suite/graceful_recovery_test.go Outdated Show resolved Hide resolved

tests/suite/graceful_recovery_test.go Outdated Show resolved Hide resolved

tests/suite/graceful_recovery_test.go Outdated Show resolved Hide resolved

tests/suite/graceful_recovery_test.go Show resolved Hide resolved

bjee19 requested a review from pleshakov May 1, 2024 21:41

pleshakov reviewed May 2, 2024

View reviewed changes

tests/suite/graceful_recovery_test.go Show resolved Hide resolved

tests/suite/graceful_recovery_test.go Outdated Show resolved Hide resolved

bjee19 requested a review from pleshakov May 2, 2024 22:17

pleshakov approved these changes May 2, 2024

View reviewed changes

sjberman reviewed May 3, 2024

View reviewed changes

tests/suite/graceful_recovery_test.go Show resolved Hide resolved

bjee19 requested a review from sjberman May 3, 2024 15:37

sjberman approved these changes May 3, 2024

View reviewed changes

bjee19 added 27 commits May 3, 2024 10:49

Revert runAsNonRoot to true

801a187

Change test to allow system suite to deploy NGF

8134a94

Add propagation policy when deleting Job

2df9b80

Refactor multiple functions into restartContainer function

3f4d37c

Refactor test cases using shared runTest function

24eec08

Change runTest name to runRecoveryTest

8ae6c60

Add noling:gosec to satisfy CodeQL

f47df82

Remove accidentally added files

020a0b8

Add placeholder values in node debugger job

1f33460

Add check for job spec container count

851de6b

Remove regex from pgrep commands

cf6bd61

Remove leader lease checking from nginx container restart

afb3af9

Add ngfPodName to BeforeAll

1ad5271

Fix argument refactor mistake

7eaad5b

Add ContainerRestartTimeout

d513eeb

Move batchv1 scheme to system suite test

b4c7628

Refactor constants to local variables

bbd7448

Add additional comments

4544fb5

Refactor error checking

1c3ecee

Remove sinceSeconds when checking container logs for errors

6508425

Refactor waiting functions using gomega Eventually assertions

cefead3

Add Skip of failing test

587b237

Refactored Eventually functions to return errors instead of booleans

dc07841

Add check for lease holder identity being empty

d311b9d

Add filter of error messages in nginx logs to allow expected errors

dbba349

Add ubuntu image version

b5dc80e

Add teardown and setup of NGF and check NGF pod length is one

e326e96

bjee19 force-pushed the tests/automate-graceful branch from 478f5b1 to e326e96 Compare May 3, 2024 17:49

bjee19 merged commit fdde6ba into nginxinc:main May 3, 2024
40 checks passed

bjee19 deleted the tests/automate-graceful branch May 3, 2024 18:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automate Graceful Recovery NFR #1832

Automate Graceful Recovery NFR #1832

bjee19 commented Apr 17, 2024 •

edited

sjberman left a comment

bjee19 commented May 1, 2024 •

edited

sjberman commented May 1, 2024

Automate Graceful Recovery NFR #1832

Automate Graceful Recovery NFR #1832

Conversation

bjee19 commented Apr 17, 2024 • edited

Checklist

Release notes

sjberman left a comment

Choose a reason for hiding this comment

bjee19 commented May 1, 2024 • edited

sjberman commented May 1, 2024

bjee19 commented Apr 17, 2024 •

edited

bjee19 commented May 1, 2024 •

edited