5.5.1 broken upgrade

[Elyytscha]

Overview

autoupgrade to 5.5.1 (Mar 8, 2024, 4:17 PM UTC) broke my postgres

Failed to pull image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.11-0": rpc error: code = Unknown desc = reading manifest ubi8-14.11-0 in registry.developers.crunchydata.com/crunchydata/crunchy-postgres: manifest unknown

Environment

okd 4.14

Please provide the following details:

• Platform: okd
• Platform Version: 4.14
• PGO Image Tag: (e.g. ubi8-5.x.y-0)
• Postgres Version 14
• Storage: rook block storage (ceph)

Steps to Reproduce

REPRO

Provide steps to get to the error condition:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: postgresql
  namespace: openshift-operators
spec:
  channel: v5
  installPlanApproval: Automatic
  name: postgresql
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: postgresoperator.v5.5.0
---
apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: keycloak-db
  namespace: openshift-sso
  labels:
    app.kubernetes.io/instance: keycloak
spec:
  backups:
    pgbackrest:
      repos:
        - name: repo1
          volume:
            volumeClaimSpec:
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 5Gi
  instances:
    - dataVolumeClaimSpec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
      name: ''
      replicas: 1
  port: 5432
  postgresVersion: 14

EXPECTED

a working postgres instance

ACTUAL

Logs

Please provided appropriate log output or any configuration files that may help troubleshoot the issue. DO NOT include sensitive information, such as passwords.

Additional Information

Please provide any additional information that may be helpful.

[andrewlecuyer]

hi @Elyytscha! Sorry to hear you are having trouble.

Unfortunately the v5.5.1 community OLM installers went out with a reference to an image (specifically for PG 14, as you're seeing) that doesn't exist in the Developer Registry. Per the Supported Platforms page within the CPK documentation, the latest two versions of Postgres are supported via the Crunchy Developer Program.

Additionally, I'll also reference the following GitHub issues for similar threads and conversations related to this:

• registry.developers.crunchydata.com not availible anymore.. #3859 (comment)
• postgres_14 image is not available postgres-operator-examples#255

Fortunately, you should be able to quickly get back up and running using the PG v14.10 and pgBackRest v2.47 images. This means you will need to update your PostgresCluster specs as follows (this is the Postgres version your cluster would have stayed at if the incorrect image reference hadn't been included in the OLM installer):

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: hippo
spec:
  backups:
    pgbackrest:
      image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.47-0
  image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.10-0
   ...

And from there you can perform a major PG upgrade to onto a supported version of PG using CPK's Major Postgres Version Upgrade functionality.

[Elyytscha]

thanks for the help, i will close

didn't noticed that only the latest 2 postgres version are supported

(it feels a little bit harsh to drop support in an upgrade from 5.5.0 -> 5.5.1 just saying)

my 2 cents

1. maybe support postgres 14 a little bit longer then usual until users had time to adapt https://www.crunchydata.com/blog/be-ready-public-schema-changes-in-postgres-15
2. support a convenient way to adapt 1 via postgrescluster custom resource, maybe

kind: PostgresCluster
apiVersion: postgres-operator.crunchydata.com/v1beta1
metadata:
  name: hippo
spec:
...
  defaultPublicAccess: true
  postgresVersion: 16
...

which defaults to no with the actual behaviour, while true gives the default user access to the public schema
(#3567 (comment))