GE UI unable to retrieve Envoy configuration when gateway-proxies not in default gloo-system namespace

[DuncanDoyle]

Gloo Edge Product

Enterprise

Gloo Edge Version

1.16.6

Kubernetes Version

1.27.8

Describe the bug

When you deploy your gateway proxies in any other namespace than gloo-system, the GE UI will be not be able to recognize and display the Envoy configuration.

Using the following configuration, in which we only deploy the default gateway in gloo-system and the other 2 gateway-proxies in gloo-external and gloo-internal, the UI will only be able to display the Envoy config of the GW in gloo-system namespace:

gloo:
  settings:
    disableKubernetesDestinations: true
  discovery:
    enabled: true
  gateway:
    persistProxySpec: true
    readGatewaysFromAllNamespaces: true
  gatewayProxies:      
    gatewayProxy:
      disabled: false
      readConfig: true
      readConfigMulticluster: true
      gatewaySettings:
        customHttpGateway:
          virtualServiceSelector:
            gateway-type: default
        customHttpsGateway:
          virtualServiceSelector:
            gateway-type: default
    publicGw:
      namespace: gloo-external
      disabled: false
      readConfig: true
      readConfigMulticluster: true
      kind:
        deployment:
          replicas: 1
      service:
        httpPort: 81
        httpsFirst: true
        httpsPort: 444
        type: LoadBalancer
      tcpKeepaliveTimeSeconds: 5
      gatewaySettings:
        customHttpGateway:
          virtualServiceSelector:
            gateway-type: public
        customHttpsGateway:
          virtualServiceSelector:
            gateway-type: public
    corpGw:
      namespace: gloo-internal
      disabled: false
      readConfig: true
      readConfigMulticluster: true
      kind:
        deployment:
          replicas: 1
      service:
        httpPort: 8080
        httpsFirst: true
        httpsPort: 8443
        type: LoadBalancer
      tcpKeepaliveTimeSeconds: 5
      gatewaySettings:
        customHttpGateway:
          virtualServiceSelector:
            gateway-type: private
        customHttpsGateway:
          virtualServiceSelector:
            gateway-type: private

When you deploy all 3 gateway-proxies in the gloo-system namespace, the UI will see all of them:

gloo:
  settings:
    disableKubernetesDestinations: true
  discovery:
    enabled: true
  gateway:
    persistProxySpec: true
    readGatewaysFromAllNamespaces: true
  gatewayProxies:      
    gatewayProxy:
      disabled: false
      readConfig: true
      readConfigMulticluster: true
      gatewaySettings:
        customHttpGateway:
          virtualServiceSelector:
            gateway-type: default
        customHttpsGateway:
          virtualServiceSelector:
            gateway-type: default
    publicGw:
      # namespace: gloo-external
      disabled: false
      readConfig: true
      readConfigMulticluster: true
      kind:
        deployment:
          replicas: 1
      service:
        httpPort: 81
        httpsFirst: true
        httpsPort: 444
        type: LoadBalancer
      tcpKeepaliveTimeSeconds: 5
      gatewaySettings:
        customHttpGateway:
          virtualServiceSelector:
            gateway-type: public
        customHttpsGateway:
          virtualServiceSelector:
            gateway-type: public
    corpGw:
      # namespace: gloo-internal
      disabled: false
      readConfig: true
      readConfigMulticluster: true
      kind:
        deployment:
          replicas: 1
      service:
        httpPort: 8080
        httpsFirst: true
        httpsPort: 8443
        type: LoadBalancer
      tcpKeepaliveTimeSeconds: 5
      gatewaySettings:
        customHttpGateway:
          virtualServiceSelector:
            gateway-type: private
        customHttpsGateway:
          virtualServiceSelector:
            gateway-type: private

When you change the writeNamespace of GE to another namespace than gloo-system, the Proxy CRs get written to a namespace that is different from gloo-system. The dataplane works fine, but the UI can't see any Envoy config with this setting.

gloo:
  settings:
    disableKubernetesDestinations: true
    writeNamespace: gloo-external
  discovery:
    enabled: true
  gateway:
    persistProxySpec: true
    readGatewaysFromAllNamespaces: true
  gatewayProxies:      
    gatewayProxy:
      disabled: false
      readConfig: true
      readConfigMulticluster: true
      gatewaySettings:
        customHttpGateway:
          virtualServiceSelector:
            gateway-type: default
        customHttpsGateway:
          virtualServiceSelector:
            gateway-type: default
    publicGw:
      # namespace: gloo-external
      disabled: false
      readConfig: true
      readConfigMulticluster: true
      kind:
        deployment:
          replicas: 1
      service:
        httpPort: 81
        httpsFirst: true
        httpsPort: 444
        type: LoadBalancer
      tcpKeepaliveTimeSeconds: 5
      gatewaySettings:
        customHttpGateway:
          virtualServiceSelector:
            gateway-type: public
        customHttpsGateway:
          virtualServiceSelector:
            gateway-type: public
    corpGw:
      # namespace: gloo-internal
      disabled: false
      readConfig: true
      readConfigMulticluster: true
      kind:
        deployment:
          replicas: 1
      service:
        httpPort: 8080
        httpsFirst: true
        httpsPort: 8443
        type: LoadBalancer
      tcpKeepaliveTimeSeconds: 5
      gatewaySettings:
        customHttpGateway:
          virtualServiceSelector:
            gateway-type: private
        customHttpsGateway:
          virtualServiceSelector:
            gateway-type: private

Expected Behavior

GE UI is able to display the Envoy configuration of all my gateway-proxies, regardless of the namespace they are deployed in.

Steps to reproduce the bug

Use the Helm values provided in the description to deploy the different scenarios.

Also, see reproducer project here: https://github.com/DuncanDoyle/ge-gloo-9378-9379

Additional Environment Detail

No response

Additional Context

No response

[nfuden]

if we are tackling this we should also make sure that it works with GGV2 and also migrates from using the proxy cr to instead using either the grpc endpoint or the xds endpoints.
If we are tackling this we need to make sure that we dont leak enterprise xds configuration that is not envoy. In other words extauth configuration likely would not display well if attempted to be parsed as proxy.