Greenplum: Restoration GP from a backup with the --restore-only option

[Sanikadze]

Greenplum

Greenplum Database 6.23.3
WAL-G 3.0.0

What am I doing wrong? Or how do I restore GP from backup with the --restore-only option?

After restoring from a backup with the --restore-only option, all segments go into panic mode.

backup-fetch

wal-g backup-fetch LATEST --restore-only="gpdb/public.rnd_nocomp9_distrnd_ao1" --in-place --restore-point "202403250853" --config=/home/gpadmin/gpdb_configs/wal-g.json
INFO: 2024/03/25 20:18:21.880238 Selecting the latest backup...
INFO: 2024/03/25 20:18:21.906460 LATEST backup is: 'backup_20240325T082654Z_D_20240324T194524Z'
INFO: 2024/03/25 20:18:21.906526 Starting backup-fetch for backup_20240325T082654Z_D_20240324T194524Z
INFO: 2024/03/25 20:18:21.912862 [Unpack] Running wal-g on segments and master...
INFO: 2024/03/25 20:18:44.638679 [Prepare] Updating pg_hba configs on segments...
INFO: 2024/03/25 20:18:44.922102 [Prepare] Creating recovery.conf files...
INFO: 2024/03/25 20:18:44.922125 Recovery target is 202403250853

When attempting to execute gpstart after backup-fetch, I receive the following error:

20240325:20:19:48:009955 gpstart:mdw:gpadmin-[INFO]:-Commencing parallel segment instance shutdown, please wait...
20240325:20:19:48:009955 gpstart:mdw:gpadmin-[ERROR]:-gpstart error: Do not have enough valid segments to start the array.

Segment logs

2024-03-25 20:28:55.248368 MSK,,,p9274,th1452865664,,,,0,,,seg0,,,,,"LOG","00000","restored log file ""00000008.history"" from archive",,,,,,,0,,"xlogarchive.c",270,
2024-03-25 20:28:55.311838 MSK,,,p9273,th1452865664,,,,0,,,seg0,,,,,"LOG","00000","3rd party error log:
INFO: 2024/03/25 20:28:55.311442 Files will be read from storages: [default]
",,,,,,,,"SysLoggerMain","syslogger.c",752,
2024-03-25 20:28:55.352276 MSK,,,p9274,th1452865664,,,,0,,,seg0,,,,,"LOG","00000","restored log file ""00000009.history"" from archive",,,,,,,0,,"xlogarchive.c",270,
2024-03-25 20:28:55.364426 MSK,,,p9274,th1452865664,,,,0,,,seg0,,,,,"LOG","00000","redo starts at 0/3C00CB50",,,,,,,0,,"xlog.c",7216,
2024-03-25 20:28:55.365156 MSK,,,p9274,th1452865664,,,,0,,,seg0,,,,,"WARNING","01000","page 129 of relation base/16384/57479 does not exist",,,,,,,0,,"xlogutils.c",75,
2024-03-25 20:28:55.365167 MSK,,,p9274,th1452865664,,,,0,,,seg0,,,,,"WARNING","01000","page 1 of relation base/16384/57479 does not exist",,,,,,,0,,"xlogutils.c",75,
2024-03-25 20:28:55.366754 MSK,,,p9273,th1452865664,,,,0,,,seg0,,,,,"LOG","00000","3rd party error log:
sh: addr2line: command not found
",,,,,,,,"SysLoggerMain","syslogger.c",752,
2024-03-25 20:28:55.368750 MSK,,,p9274,th1452865664,,,,0,,,seg0,,,,,"PANIC","XX000","WAL contains references to invalid pages",,,,,,,0,,"xlogutils.c",247,"Stack trace:
1    0xbdbe58 postgres errstart + 0x2b8
2    0xbddd08 postgres elog_finish + 0xb8
3    0x777be6 postgres <symbol not found> + 0x777be6
4    0x7679e8 postgres <symbol not found> + 0x7679e8
5    0x76f7ed postgres StartupXLOG + 0x151d
6    0xa1d533 postgres StartupProcessMain + 0x123
7    0x7aabf1 postgres AuxiliaryProcessMain + 0x4e1
8    0xa1739f postgres <symbol not found> + 0xa1739f
9    0xa1c8d1 postgres PostmasterMain + 0xd11
10   0x6de1ab postgres main + 0x42b
11   0x7f97533ffd85 libc.so.6 __libc_start_main + 0xe5
12   0x6e9f8e postgres _start + 0x2e
"
2024-03-25 20:28:55.557336 MSK,,,p9272,th1452865664,,,,0,,,seg0,,,,,"LOG","00000","startup process (PID 9274) was terminated by signal 6: Aborted",,,,,,,0,,"postmaster.c",4009,
2024-03-25 20:28:55.557352 MSK,,,p9272,th1452865664,,,,0,,,seg0,,,,,"LOG","00000","terminating any other active server processes",,,,,,,0,,"postmaster.c",3729,

[x4m]

Yup, we totally forgot to document, that GP build must include greenplum-db/gpdb#15942

[Sanikadze]

Thank you for the prompt response.

[x4m]

Thanks for using wal-g and letting us know about the problem. PR with updates to documentation would be appreciated.
Also, if you are interested in the feature - you can bump up PR in GP repo :)