wal-g symlink postgresql.conf outside of target directory

[LotusIodid]

Database name

postgresql v15

Issue description

Describe your problem

INFO: 2023/10/10 02:17:58.472020 Finished extraction of part_1291.tar.lz4                                                                                                          
ERROR: 2023/10/10 02:17:58.472062 Extraction error in part_1291.tar.lz4: extractOne: Interpret failed: Interpret: failed to create symlink /var/lib/postgresql/15/postgresql.conf: 
symlink /postgresql.conf /var/lib/postgresql/15/postgresql.conf: file exists
ERROR: 2023/10/10 02:17:58.472111 Failed to fetch backup: failed to extract files:   

The symlink does indeed exist, but the target of the file does not exist. And the target also cannot be created. The expected behavior would be that, wal-g checks only if the symlink exists, but not if the target exists. Because the target is outside /var/lib/postgresql/15/.

Please provide steps to reproduce

1. Move postgresql.conf outside of /var/lib/postgresql/15/.
2. Create a symlink to this file.
3. Backup
4. Try to restore (on a different machine, with an empty restore target)

Please add config and wal-g stdout/stderr logs for debug purpose

If you can, provide logs

There are no other error messages above.

WARNING: 2023/10/10 01:58:38.852068 1 files failed to download: [part_1291.tar.lz4]. Going to sleep and retry downloading them.
INFO: 2023/10/10 02:01:29.274289 Finished extraction of part_1291.tar.lz4
ERROR: 2023/10/10 02:01:29.274315 Extraction error in part_1291.tar.lz4: extractOne: Interpret failed: Interpret: failed to create symlink /var/lib/postgresql/15/postgresql.conf: 
symlink /postgresql.conf /var/lib/postgresql/15/postgresql.conf: file exists
WARNING: 2023/10/10 02:01:29.274331 1 files failed to download: [part_1291.tar.lz4]. Going to sleep and retry downloading them.                                                    
INFO: 2023/10/10 02:05:18.731594 Finished extraction of part_1291.tar.lz4
ERROR: 2023/10/10 02:05:18.731623 Extraction error in part_1291.tar.lz4: extractOne: Interpret failed: Interpret: failed to create symlink /var/lib/postgresql/15/postgresql.conf: 
symlink /postgresql.conf /var/lib/postgresql/15/postgresql.conf: file exists
WARNING: 2023/10/10 02:05:18.731640 1 files failed to download: [part_1291.tar.lz4]. Going to sleep and retry downloading them.                                                    
INFO: 2023/10/10 02:11:08.421455 Finished extraction of part_1291.tar.lz4                
ERROR: 2023/10/10 02:11:08.421477 Extraction error in part_1291.tar.lz4: extractOne: Interpret failed: Interpret: failed to create symlink /var/lib/postgresql/15/postgresql.conf: 
symlink /postgresql.conf /var/lib/postgresql/15/postgresql.conf: file exists                                                                                                       
WARNING: 2023/10/10 02:11:08.421494 1 files failed to download: [part_1291.tar.lz4]. Going to sleep and retry downloading them.
INFO: 2023/10/10 02:17:58.472020 Finished extraction of part_1291.tar.lz4                                                                                                          
ERROR: 2023/10/10 02:17:58.472062 Extraction error in part_1291.tar.lz4: extractOne: Interpret failed: Interpret: failed to create symlink /var/lib/postgresql/15/postgresql.conf: 
symlink /postgresql.conf /var/lib/postgresql/15/postgresql.conf: file exists
ERROR: 2023/10/10 02:17:58.472111 Failed to fetch backup: failed to extract files:   
part_1291.tar.lz4

[x4m]

Hi! Generally, we expect PGDATA to be empty before restore. In this particular case we can do special actions of checking symlink, but it's kind of unwanted exception. It seems to me we must add check that directory is empty before fetching backup.

[LotusIodid]

PGDATA was empty at the beginning of the restore process. wal-g tried to create the symlink.
The original symlink goes to /nix/store/.../postgresql.conf. The symlink that was restored by wal-g did go to /postgresql.conf. So /var/lib/postgresql/15/postgresql.conf -> /postgresql.conf. I'm testing right now if touch /postgresql.conf before the restore is a workaround.

I do think the correct solution would be to not check if the symlink target exists at all. Just check that the symlink was recovered, but ignore the target. The target could be a file that will be later recovered by wal-g, but it could also be a file that will not be recovered at all because it is managed outside of PGDATA dir.

[LotusIodid]

I clarified the steps to reproduce part.

[x4m]

Uhm...I see. Seems like a bug.

[x4m]

@usernamedt I think we could recruit someone from university to fix this. Do we have some kind of special tag for such bugs? TLDR: recovery fails if backup contains symlink to nonexistent external file.

[LotusIodid]

touch /postgresql.conf is not a working workaround. Error message has not failed.

@x4m Is there a way to ignore errors and just continue with the recovery?

[LotusIodid]

I have found a workaround. watch rm /var/lib/postgresql/15/postgresql.conf. I think that creating the symlink just works. But the symlink is created multiple times. This second creation fails at (https://github.com/wal-g/wal-g/blob/master/internal/file_tar_interpreter.go#L46C13-L46C24).

Shall the https://github.com/wal-g/wal-g/blob/master/internal/extract.go#L78C6-L78C19 extractOneTar function overwrite already existing files?