feat: prevent failovers when disk space is exhausted #4404

leonardoce · 2024-04-29T15:46:43Z

PostgreSQL will shut down cleanly when there is not enough disk space to store WAL files.

The operator did not recognize this condition and, since the primary failed, was performing a failover to the most advanced replica. This action will not fix the underlying issue.

Only a manual disk resize, initiated by the user, can ultimately lead to a fully working PostgreSQL cluster.

This patch makes the instance manager recognize this condition and report it to the operator. Upon detecting it, the operator will not trigger a switchover and set a phase describing the situation.

After the PVCs are resized, the cluster will restart working correctly.

Closes: #4521

github-actions · 2024-04-29T15:46:58Z

❗ By default, the pull request is configured to backport to all release branches.

To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

leonardoce · 2024-05-13T10:10:05Z

I tested this using Longhorn in a Fedora VM, but any storage enforcing the PV capacity will do the trick.

To test the patch, you need to finish your WAL storage. To keep things easy, I used:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-example
spec:
  instances: 1

  storage:
    size: 256Mi

And then:

CREATE TABLE storage_area (t text);

-- repeat the following query 20-30 times (you need to be fast!)
INSERT INTO storage_area (t) (select repeat('Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do', 5*1024*1024));

With the predefined WAL settings, you'll finish your WAL disk space before you finish the space for PGDATA.

controllers/cluster_controller.go

pkg/management/postgres/instance.go

controllers/cluster_controller.go

armru · 2024-05-16T10:13:57Z

/test limit=local

github-actions · 2024-05-16T10:14:12Z

@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/9110497781

jsilvela

About to start adding documentation and going over the E2E, but left a few comments on the implementation bits.
IMO the "WALDisk" nomenclature could get confusing as it seems to imply there is a separate WAL volume, which may or may not be the case.

internal/cmd/manager/instance/run/lifecycle/run.go

pkg/fileutils/directory.go

pkg/management/postgres/instance.go

pkg/utils/fencing.go

controllers/cluster_controller.go

jsilvela

I still think it's worth renaming the ensureSufficientDiskSpace method, but otherwise give this an enthusiastic 👍

docs/src/instance_manager.md

docs/src/troubleshooting.md

PostgreSQL will shutdown cleanly when there is no enough disk space to store WAL files. The operator was not recognizing this condition and, since the primary failed, was performing a failover to the most advanced replica. This action will not fix the underlying issue. Only a manual disk resize, initiated by the user, can ultimately lead to a fully working PostgreSQL cluster. This patch makes the instance manager recognize this condition, and report it back to the operator. Upon detecting it, the operator will fence the primary instance and set a phase describing the situation. Since the primary instance is fenced, no failovers will be done. Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enteprisedb.com>

Add an e2e to test the recovery in case a primary runs out of disk space. Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enteprisedb.com>

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>

Co-authored-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@gmail.com>

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

Signed-off-by: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enteprisedb.com>

leonardoce · 2024-06-04T12:04:53Z

E2e tests are fine!

…4404) PostgreSQL will shut down cleanly when there is not enough disk space to store WAL files. The operator did not recognize this condition and, since the primary failed, was performing a failover to the most advanced replica. This action will not fix the underlying issue. Only a manual disk resize, initiated by the user, can ultimately lead to a fully working PostgreSQL cluster. This patch makes the instance manager recognize this condition and report it to the operator. Upon detecting it, the operator will not trigger a switchover and set a phase describing the situation. After the PVCs are resized, the cluster will restart working correctly. Closes: cloudnative-pg#4521 Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enteprisedb.com> Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com> Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Signed-off-by: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enteprisedb.com> Co-authored-by: Francesco Canovai <francesco.canovai@enterprisedb.com> Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Co-authored-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Co-authored-by: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com> Signed-off-by: Douglass Kirkley <dkirkley@eitccorp.com>

github-actions bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.21 release-1.22 release-1.23 labels Apr 29, 2024

leonardoce force-pushed the dev/space branch 3 times, most recently from 4012706 to 126566d Compare May 13, 2024 09:54

leonardoce marked this pull request as ready for review May 13, 2024 10:02

leonardoce requested a review from a team as a code owner May 13, 2024 10:02

gbartolini reviewed May 13, 2024

View reviewed changes

controllers/cluster_controller.go Outdated Show resolved Hide resolved

gbartolini reviewed May 13, 2024

View reviewed changes

pkg/management/postgres/instance.go Outdated Show resolved Hide resolved

armru reviewed May 15, 2024

View reviewed changes

controllers/cluster_controller.go Outdated Show resolved Hide resolved

armru force-pushed the dev/space branch 2 times, most recently from 9b2cea6 to e625d8f Compare May 15, 2024 15:25

armru requested review from jsilvela, NiccoloFei and litaocdl as code owners May 16, 2024 10:04

github-actions bot added the ok to merge 👌 This PR can be merged label May 16, 2024

armru approved these changes May 20, 2024

View reviewed changes

jsilvela reviewed May 20, 2024

View reviewed changes

controllers/cluster_controller.go Outdated Show resolved Hide resolved

leonardoce force-pushed the dev/space branch 2 times, most recently from 172cfe7 to 3cdc43a Compare May 21, 2024 09:47

jsilvela approved these changes May 21, 2024

View reviewed changes

jsilvela reviewed May 21, 2024

View reviewed changes

docs/src/instance_manager.md Outdated Show resolved Hide resolved

jsilvela reviewed May 21, 2024

View reviewed changes

docs/src/troubleshooting.md Outdated Show resolved Hide resolved

leonardoce added do not backport This PR must not be backported - it will be in the next minor release and removed release-1.23 labels Jun 4, 2024

Leonardo Cecchi and others added 23 commits June 4, 2024 13:53

test: out of disk space recovery scenario

eb6f98c

Add an e2e to test the recovery in case a primary runs out of disk space. Signed-off-by: Francesco Canovai <francesco.canovai@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enteprisedb.com>

review: bulk fencing and noWalDiskSpace status

5a8789f

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

chore: more structured approach to size probing

c943173

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

chore: rename size_probe -> directory

0b4fa29

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

docs: add top-level documentation

e0b076e

Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>

docs: commas

cbcdbb7

Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>

chore: fix grammar in pkg/fileutils/directory.go

71562a9

Co-authored-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@gmail.com>

chore: fix grammar in pkg/fileutils/directory.go

bc9b8bc

Co-authored-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@gmail.com>

chore: fix pkg/utils/fencing.go

5c3ffa0

Co-authored-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@gmail.com>

chore: address Gabriele's comments

2664c9e

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

chore: address Jaime's comments

b7b57c0

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

chore: improve naming

c9fd9a9

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

review: clarify documentation

f99b807

Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>

Update docs/src/instance_manager.md

1fcbcaf

Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>

Update docs/src/troubleshooting.md

120456e

Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com>

chore: directory vs diskprobe

5e2c31c

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

chore: rename ensureSufficientDiskSpace to ensureNoFailoverOnFullDisk to

304ee69

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

docs: cosmetic changes

f8c1263

Signed-off-by: Gabriele Bartolini <gabriele.bartolini@enterprisedb.com>

feat: implementation using exit codes and no fencing

3a3ec2a

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

fix: reduce required space to a single wal

1bb5ac2

docs: improve documentation

bdc766d

chore: remove WALSpaceAvailable field

f98905a

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enteprisedb.com>

leonardoce force-pushed the dev/space branch from 957d3eb to f98905a Compare June 4, 2024 11:53

leonardoce removed the do not merge 🙅 This PR cannot be merged (yet) label Jun 4, 2024

leonardoce merged commit bf42946 into cloudnative-pg:main Jun 4, 2024
25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: prevent failovers when disk space is exhausted #4404

feat: prevent failovers when disk space is exhausted #4404

leonardoce commented Apr 29, 2024 •

edited

github-actions bot commented Apr 29, 2024

leonardoce commented May 13, 2024

armru commented May 16, 2024

github-actions bot commented May 16, 2024

jsilvela left a comment •

edited

jsilvela left a comment

leonardoce commented Jun 4, 2024

feat: prevent failovers when disk space is exhausted #4404

feat: prevent failovers when disk space is exhausted #4404

Conversation

leonardoce commented Apr 29, 2024 • edited

github-actions bot commented Apr 29, 2024

leonardoce commented May 13, 2024

armru commented May 16, 2024

github-actions bot commented May 16, 2024

jsilvela left a comment • edited

Choose a reason for hiding this comment

jsilvela left a comment

Choose a reason for hiding this comment

leonardoce commented Jun 4, 2024

leonardoce commented Apr 29, 2024 •

edited

jsilvela left a comment •

edited