[outline on kubernetes] Regular write error on PG Cluster setup

[bmm-alc]

Hello,

we recently migrated Outline and the dependencies (postgresql and redis) to Kubernetes, and since then we observe error 500 due to issue in the Db layer.
Our Db setup involves PgBouncer and a 3 nodes postgresql cluster (version 16) managed by percona postgresql operator. Outline is configured to connect to the PgBouncer and PgBouncer connects to the primary of the cluster.
We set DATABASE_URL to postgresql://user:pass@postgresql-pgbouncer.outline.svc:5432/outline

Randomly we get write errors which prevents documents creation or modification. The problem appears and disappears without being able to have a reproducer to trigger the problem by ourselves.

The error is the following

could not read block 687 in file "base/16447/33617": read only 0 of 8192 bytes

The block number increase by one each time we get an error.

one of the exceptions we get on outline

Error
    at Query.run (/opt/outline/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)
    at /opt/outline/node_modules/sequelize/lib/sequelize.js:315:28
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async PostgresQueryInterface.insert (/opt/outline/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)
    at async Revision.save (/opt/outline/node_modules/sequelize/lib/model.js:2490:35)
    at async /opt/outline/build/server/commands/revisionCreator.js:16:22
    at async /opt/outline/node_modules/sequelize/lib/sequelize.js:507:18
    at async RevisionsProcessor.perform (/opt/outline/build/server/queues/processors/RevisionsProcessor.js:39:11)
    at async Queue.<anonymous> (/opt/outline/build/server/services/worker.js:86:9)

We modified the setup to make outline skip pgbouncer and directly connect to the primary postgresql, we still had the same issue.

we can dump the postgresql data without any issue, so unlikely to be a corruption on database side.

On docker we never had such issue 😓

Is someone having the same setup in order there is some tuning to do ?
I'm no expert in postgreql, so is the outline could be "incompatible" with such Db setup ?

Best

[bmm-alc]

Turned out to be a corruption on the table revisions since day 1 we migrated to kubernetes....
id 33617 is the OID of the revisions table in postgresql. It was possible to read the table but not to write to it. Backup (so reading all data) does not generate any error, no error eith on postgresql side. To fix we deleted all the database and restored it.

Impact: We lost all the revisions since we migrated, we just have the last version of each document if I understand correctly.

What could be improve in outline: That would have being great that outline provides the database query when it fails executing it.