Layer preview using WMS causes service exception

[AlexTzk]

Bug description

A cloud geoserver deployment on GKE will stop rendering layers in "Layer Preview" after an upgrade or scaling up/down operations.
The following exception will be shown:

<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE ServiceExceptionReport SYSTEM "https://somedomain.com/geoserver/schemas/wms/1.1.1/WMS_exception_1_1_1.dtd"> <ServiceExceptionReport version="1.1.1" > <ServiceException code="LayerNotDefined" locator="layers"> Could not find layer site_2:gm_n_anomalies_1y_2 </ServiceException></ServiceExceptionReport>

Steps to reproduce

1. Install geoserver cloud 1.6.0 or 1.7.0 with profile datadir
2. Upload a few layers
3. Check that Layer preview works gracefully
4. Perform a few scale up/down operations on WMS, WEBUI and GWC
5. Upgrade to 1.7.1 the whole stack
6. Check if Layer Preview is working

Expected behaviour

Layer preview works after upgrades or scale up/down operations.

YAML Files

I have created each deployment file manually from the docker-compose file.
JNDI and JDBC sources are not used, instead we link the layers which are stored in a PGSQL DB with a Geoserver store.

It's entirely possible that I am doing something wrong but I was able to replicate this in 3 different environments quite easily hence the issue submit.

If the YAML files are useful for the documentation, please let me know and I can create a PR with them.

discovery

apiVersion: "apps/v1"
kind: "StatefulSet"
metadata:
  name: "discovery"
  namespace: "default"
  labels:
    app: "discovery"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "discovery"
  template:
    metadata:
      labels:
        app: "discovery"
        environment: "production"
        tier: "geoserver"
    spec:
      containers:
        - image: geoservercloud/geoserver-cloud-discovery:1.7.1
          name: discovery
          ports:
            - containerPort: 8761
---
apiVersion: v1
kind: Service
metadata:
  name: "discovery"
  labels:
    app: "discovery"
spec:
  selector:
    app: "discovery"
  ports:
  - name: http
    port: 8761
    targetPort: 8761

config

apiVersion: "apps/v1"
kind: Deployment
metadata:
  name: "config"
  namespace: "default"
  labels:
    app: "config"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "config"
  template:
    metadata:
      labels:
        app: "config"
        environment: "production"
        tier: "geoserver"
    spec:
      containers:
      - name: "config"
        image: geoservercloud/geoserver-cloud-config:1.7.1
        ports:
          - containerPort: 8080
        env:
          - name: CONFIG_GIT_URI
            valueFrom:
              secretKeyRef:
               name: "geogituri"
               key: "password"
          - name: SPRING_CLOUD_CONFIG_SERVER_GIT_DEFAULT_LABEL
            value: config_server_git_repo
          - name: SPRING_PROFILES_ACTIVE
            value: git
          - name: CONFIG_GIT_BASEDIR
            value: /tmp/git_config
          - name: CONFIG_NATIVE_PATH
            value: /tmp/config
          - name: XDG_CONFIG_HOME
            value: /tmp
---
apiVersion: v1
kind: Service
metadata:
  name: "config"
  labels:
    app: "config"
spec:
  selector:
    app: "config"
  ports:
  - name: http
    port: 8080
    targetPort: 8080

gateway

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "gateway"
  namespace: "default"
  labels:
    app: "gateway"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "gateway"
  template:
    metadata:
      labels:
        app: "gateway"
        environment: "production"
        tier: "geoserver"
        monitoringActuator: enabled
    spec:
      containers:
      - name: gateway
        image: geoservercloud/geoserver-cloud-gateway:1.7.1
        ports:
         - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /actuator/health
            port: 8081
          initialDelaySeconds: 50
        livenessProbe:
          httpGet:
            path: /actuator/health
            port: 8081
          initialDelaySeconds: 50  
        env:
            - name: GEOSERVER_BASE_PATH
              value: /geoserver
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: gateway
  labels:
    app: gateway
    monitoringActuator: enabled
spec:
  ports:
  - name: health
    port: 8081
    targetPort: 8081
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: gateway
  type: ClusterIP

gwc

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "gwc"
  namespace: "default"
  labels:
    app: "gwc"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "gwc"
  template:
    metadata:
      labels:
        app: "gwc"
        environment: "production"
        tier: "geoserver"
        monitoringActuator: enabled
    spec:
      containers:
      - name: gwc
        image: geoservercloud/geoserver-cloud-gwc:1.7.1
        ports:
         - containerPort: 8080
        env:
            - name: GEOSERVER_DATA_DIR
              value: "/opt/app/data_directory"
            - name: GEOWEBCACHE_CACHE_DIR
              value: "/data/geowebcache"
            - name: SPRING_PROFILES_ACTIVE
              value: "datadir"
        volumeMounts:
          - name: shared-data-directory
            mountPath: "/opt/app/data_directory"
          - name: geowebcache-data
            mountPath: "/data/geowebcache"
      restartPolicy: Always
      volumes:
        - name: shared-data-directory
          persistentVolumeClaim:
            claimName: nfs-sh
        - name: geowebcache-data
          persistentVolumeClaim:
            claimName: gwc-share
---
apiVersion: v1
kind: Service
metadata:
  name: gwc
  labels:
    app: gwc
    monitoringActuator: enabled
spec:
  ports:
  - name: health
    port: 8081
    targetPort: 8081
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: gwc
  type: ClusterIP

webui

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "webui"
  namespace: "default"
  labels:
    app: "webui"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "webui"
  template:
    metadata:
      labels:
        app: "webui"
        environment: "production"
        tier: "geoserver"
        monitoringActuator: enabled
    spec:
      containers:
      - name: webui
        image: geoservercloud/geoserver-cloud-webui:1.7.1
        ports:
          - containerPort: 8080
        env:
          - name: GEOSERVER_DATA_DIR
            value: "/opt/app/data_directory"
          - name: GEOWEBCACHE_CACHE_DIR
            value: "/data/geowebcache"
          - name: SPRING_PROFILES_ACTIVE
            value: "datadir"
        volumeMounts:
          - name: shared-data-directory
            mountPath: "/opt/app/data_directory"
          - name: geowebcache-data
            mountPath: "/data/geowebcache"
      restartPolicy: Always
      volumes:
        - name: shared-data-directory
          persistentVolumeClaim:
            claimName: nfs-sh
        - name: geowebcache-data
          persistentVolumeClaim:
            claimName: gwc-share
---
apiVersion: v1
kind: Service
metadata:
  name: webui
  labels:
    app: webui
    monitoringActuator: enabled
spec:
  ports:
  - name: health
    port: 8081
    targetPort: 8081
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: webui
  type: ClusterIP

wms

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "wms"
  namespace: "default"
  labels:
    app: "wms"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "wms"
  template:
    metadata:
      labels:
        app: "wms"
        environment: "production"
        tier: "geoserver"
        monitoringActuator: enabled
    spec:
      containers:
        - name: wms
          image: geoservercloud/geoserver-cloud-wms:1.7.1
          ports:
          - containerPort: 8080
          env:
            - name: GEOSERVER_DATA_DIR
              value: "/opt/app/data_directory"
            - name: SPRING_PROFILES_ACTIVE
              value: "datadir"
          volumeMounts:
            - name: shared-data-directory
              mountPath: "/opt/app/data_directory"
      restartPolicy: Always
      volumes:
        - name: shared-data-directory
          persistentVolumeClaim:
            claimName: nfs-sh
---
apiVersion: v1
kind: Service
metadata:
  name: wms
  labels:
    app: wms
    monitoringActuator: enabled
spec:
  ports:
  - name: health
    port: 8081
    targetPort: 8081
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: wms
  type: ClusterIP

rest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "rest"
  namespace: "default"
  labels:
    app: "rest"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "rest"
  template:
    metadata:
      labels:
        app: "rest"
        environment: "production"
        tier: "geoserver"
        monitoringActuator: enabled
    spec:
      containers:
      - name: rest
        image: geoservercloud/geoserver-cloud-rest:1.7.1
        ports:
        - containerPort: 8080
        env:
          - name: GEOSERVER_DATA_DIR
            value: "/opt/app/data_directory"
          - name: SPRING_PROFILES_ACTIVE
            value: "datadir"
        volumeMounts:
          - name: shared-data-directory
            mountPath: "/opt/app/data_directory"
      restartPolicy: Always
      volumes:
        - name: shared-data-directory
          persistentVolumeClaim:
            claimName: nfs-sh
---
apiVersion: v1
kind: Service
metadata:
  name: rest
  labels:
    app: rest
    monitoringActuator: enabled
spec:
  ports:
  - name: health
    port: 8081
    targetPort: 8081
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: rest
  type: ClusterIP

wfs

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "wfs"
  namespace: "default"
  labels:
    app: "wfs"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "wfs"
  template:
    metadata:
      labels:
        app: "wfs"
        environment: "production"
        tier: "geoserver"
        monitoringActuator: enabled
    spec:
      containers:
        - name: wfs
          image: geoservercloud/geoserver-cloud-wfs:1.7.1
          ports:
          - containerPort: 8080
          env:
            - name: GEOSERVER_DATA_DIR
              value: "/opt/app/data_directory"
            - name: SPRING_PROFILES_ACTIVE
              value: "datadir"
          volumeMounts:
            - name: shared-data-directory
              mountPath: "/opt/app/data_directory"
      restartPolicy: Always
      volumes:
        - name: shared-data-directory
          persistentVolumeClaim:
            claimName: nfs-sh
---
apiVersion: v1
kind: Service
metadata:
  name: wfs
  labels:
    app: wfs
    monitoringActuator: enabled
spec:
  ports:
  - name: health
    port: 8081
    targetPort: 8081
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: wfs
  type: ClusterIP

wcs

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "wcs"
  namespace: "default"
  labels:
    app: "wcs"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "wcs"
  template:
    metadata:
      labels:
        app: "wcs"
        environment: "production"
        tier: "geoserver"
        monitoringActuator: enabled
    spec:
      containers:
        - name: wcs
          image: geoservercloud/geoserver-cloud-wcs:1.7.1
          ports:
          - containerPort: 8080
          env:
            - name: GEOSERVER_DATA_DIR
              value: "/opt/app/data_directory"
            - name: SPRING_PROFILES_ACTIVE
              value: "datadir"
          volumeMounts:
            - name: shared-data-directory
              mountPath: "/opt/app/data_directory"
      restartPolicy: Always
      volumes:
        - name: shared-data-directory
          persistentVolumeClaim:
            claimName: nfs-sh
---
apiVersion: v1
kind: Service
metadata:
  name: wcs
  labels:
    app: wcs
    monitoringActuator: enabled
spec:
  ports:
  - name: health
    port: 8081
    targetPort: 8081
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: wcs
  type: ClusterIP