When activating the identity-auth profile, the health and readiness probes fail with status code 401

[romansmirnov]

Describe the bug

With version 8.6.0-alpha1, the readiness probe may fail with a 401. In a nutshell, the security fail chain defined for the REST API is applied when processing the readiness probe request:

https://github.com/camunda/zeebe/blob/c3110d98798853f0b471a7a6bcc22a66f70e4d71/dist/src/main/java/io/camunda/zeebe/shared/security/SecurityConfiguration.java#L42-L63

Instead, it should apply the Management security filter chain that permits all incoming requests on the management port:

https://github.com/camunda/zeebe/blob/c3110d98798853f0b471a7a6bcc22a66f70e4d71/dist/src/main/java/io/camunda/zeebe/shared/security/SecurityConfiguration.java#L84-L96

However, there might be situations where the readiness probe request does not fail with a 401, and the expected Management security filter chain is applied. Why? It depends on the order in which the beans are loaded, initiated, and registered. Sometimes, they are applied in the "right" order so that the expected security filter chain is applied on the readiness probe request.

To Reproduce

1. Download the camunda-zeebe-8.6.0-alpha1 distribution
2. Activate the profiles identity-auth by adding it to the config/application.yaml

spring:
  profiles:
    active: identity-auth

3. Start the standalone gateway by executing bin/gateway.sh
4. Execute the readiness probe http://localhost:9600/actuator/health/readiness

Expected behavior

The readiness probe succeeds with the status code 200.

Log/Stacktrace

When the beans are loaded, initiated, and registered in the "right" order, the following is logged:

Overriding bean definition for bean 'springSecurityFilterChain' with a different definition: replacing [Root bean: class [null]; scope=;
abstract=false; lazyInit=null; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false;
factoryBeanName=org.springframework.boot.actuate.autoconfigure.web.servlet.ServletManagementChildContextConfiguration$ServletManagementContextSecurityConfiguration;
factoryMethodName=springSecurityFilterChain; initMethodNames=null; destroyMethodNames=[(inferred)]; defined in class path resource
[org/springframework/boot/actuate/autoconfigure/web/servlet/ServletManagementChildContextConfiguration$ServletManagementContextSecurityConfiguration.class]]
with [Root bean: class [null]; scope=; abstract=false; lazyInit=null; autowireMode=3; dependencyCheck=0; autowireCandidate=true;
primary=false; factoryBeanName=org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration;
factoryMethodName=springSecurityFilterChain; initMethodNames=null; destroyMethodNames=[(inferred)]; defined in class path
resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class]]

Meaning, the ServletManagementChildContextConfiguration$ServletManagementContextSecurityConfiguration.class loads the springSecurityFilterChain from the application context and registers it in its management context. Afterward, the management security filter chain is initiated and registered with the WebSecurityConfiguration, which defines a bean called springSecurityFilterChain and overrides it in the management context's registry. That way, the "correct" springSecurityFilterChain is applied when executing the readiness probe request.

If the override happens in the other order, the wrong springSecurityFilterChain is applied.

Hints

• According to Spring Documentation, that should be how to configure the security filter chain for the actuator endpoints: https://docs.spring.io/spring-boot/reference/actuator/endpoints.html#actuator.endpoints.security

Environment:

• Zeebe Version: 8.6.0-alpha1
• Configuration: Activate the profiles identity-auth