index is not complete

[saberya]

edit by @MichaelEischer: moved from #3400 (comment)

I encounter this almost everyday and cant reslove it, or even understand what trigger it;
It really took me a lot of time, restic is a good tool, I hope that sb can give me a little help;
Well, I'll describe my problem and operations as follows:

---

restic 0.12.0 CentOS release 6.6
I backup my mysql-data via restic with mount remote repostory to local file system

---

Firstly, restic -r /mnt/backup/repo1 backup /root/mysql_bak/ --password ...
OK，it succeed.but i think there is something wrong with this snapshot.
Secondly,restic -r /mnt/backup/repo1 check --password-file=...
it failed with following error log

using temporary cache in /tmp/restic-check-cache-956472566
repository 6890ac2a opened successfully, password is correct
created new cache in /tmp/restic-check-cache-956472566
create exclusive lock for repository
load indexes
check all packs
pack daac2bc1: unexpected file size: got 0, expected 4765734
check snapshots, trees and blobs
[0:00] 100.00%  6 / 6 snapshots
Fatal: repository contains errors

umhhh,,,,maybe it caused by Corrupted file，but I dont know what lead to this error;
this problem appeared many times with high frequency;
i think my server is stable and disk is abundant；

Thirdly,i tried to resolve this problem
restic -r /mnt/backup/repo1 prune 75f16341 --password-file=... --verbose
75f16342 is the error snapshot
it failed with following error

repository 6890ac2a opened successfully, password is correct
loading indexes...
loading all snapshots...
finding data that is still in use for 6 snapshots
[0:00] 100.00%  6 / 6 snapshots
searching used packs...
{<data/423e668c> <data/4c5f1a17> <data/a24f19fc>} not found in the index

Integrity check failed: Data seems to be missing.
Will not start prune to prevent (additional) data loss!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new/chooseFatal: index is not complete


searching used packs...
collecting packs for deletion and repacking
pack daac2bc1: calculated size 4765732 does not match real size 0

Forthly ,restic rebuild-index
also failed, but i miss the log

Fifthly, restic backup again, it falied

Save(<data/e893418011>) returned error, retrying after 552.330144ms: Sync: sync /mnt/backup/repo1/data/e8/e893418011982ce20fa4064d1: input/output errorak/test/file1_210712_040001.sql.gz_split021
Save(<data/952f81102b>) returned error, retrying after 720.254544ms: Sync: sync /mnt/backup/repo1/data/95/952f81102b181b2e503f22724d8424d2974c3c38794f9bab0ef972d20d5aee25: host
is downysql_bak/test/file2_210712_040001.sql.gz_split021
Save(<data/fb88f1e58b>) returned error, retrying after 582.280027ms: Sync: sync /mnt/backup/autodb2/repo1/data/fb/fb88f1e58bf4034a1076c5cdf391e17d4a8d6016acd9b2779b9bc7ce39253e4f: input/output error

I don't understand what's going on with those message

Sixthly, restic check

repository 6890ac2c opened successfully, password is correct
loading indexes...
loading all snapshots...
finding data that is still in use for 6 snapshots
[0:00] 100.00%  6 / 6 snapshots
searching used packs...
{<data/423e668d> <data/4c5f1a17> <data/a24f19fc>} not found in the index

Integrity check failed: Data seems to be missing.
Will not start prune to prevent (additional) data loss!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new/chooseFatal: index is not complete

Oh, man, intersting I was completely dumbstruck，i had rebuild the repository, but id didnt work;
who can help me, at least,tell me how to operate to resolve this problem，Thank you very much
i have used resic for three months, i really ran into a lot of problems and solved them manually every time

[MichaelEischer]

@saberya On which filesystem is "/mnt/backup" stored? If this is a SMB mount then you have to set the environment variable GODEBUG to asyncpreemptoff=1 as documented in https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#local .

pack daac2bc1: unexpected file size: got 0, expected 4765734

This error should normally be resolved by rebuild-index however it usually indicates a problem with the storage backend / filesystem.

Save(<data/e893418011>) returned error, retrying after 552.330144ms: Sync: sync /mnt/backup/repo1/data/e8/e893418011982ce20fa4064d1: input/output error

restic complains here that it could not properly store the pack file, due to an input/output error. This kind of error is normally a problem of the underlying hardware / filesystem, which fails to write the data it was supposed to, and not of restic.

[saberya]

thanks for you reply.
i guess the point of your reply is that my storage filesystem leads to the problem?
but my backend filesystem is stable as the network, and the backup user is root, but this problem appears frequnently,but i'll try to open debug model.
I backupserver mysql data with the file format of *.sql.gz， and the size of one file is 20GB-40GB approximately and backup everyday,i wonder maybe it too big for a file, so i _split -b 512000k $files -d -a 3 "${files}split" split it for many 500MB files.
And i have no idea how can i avoid this problem, it is tedious to reslove it manually everytime.
change a backup filesystem? Or dont use mount way? Or some better way?

[rawtaz]

@saberya You should start by answering the question that the person who is trying to help you (@MichaelEischer) is asking you, what else is the point of him asking them and trying to help you. Please answer his questions, and follow his advice and suggestions. That's a good start to approach the problem you're having.

[saberya]

yummmm,i checked my server;
i use a specialized backup machine, there is 2 repositories is this machine,with same filesystem and network, only the backup directory is different(repo path);
but the other repo is good and stable all the time,Not once did a problem come up
The biggest difference between the two repos is the file size,but i dont think 500MB is a size for big file.
so i guess the reason is not the filesystem or hardware
thanks for you to take this into consideration

[saberya]

thank you, @rawtaz i am trying to follow MichaelEischer's suggestion to reslove my problem.
the stored filesytem is not SMB，it just is FTP @MichaelEischer

[rawtaz]

@saberya It's still not clear how you mounted that path. Please show the output from mount | grep '/mnt/backup'.

[saberya]

this is my mount command
"mount -t cifs -o "rw,username=db,password=abc，uid=root" //backupserver01.aaa.net/backup/db1/mnt/backup/db1"

[root@autodb2 ~]# mount | grep '/mnt/backup'
//backupserver01.aaa.net/backup/db1 on /mnt/backup/db1 type cifs (rw)

[root@autodb2 ~]# df -h
....
//backupserver01.aaa.net/backup/db1
                       55T   25T   31T  44% /mnt/backup/db1

and in this path //backupserver01.aaa.net/backup/db1，there are two repositories:repo1,repo2
i execute backup command like this

/usr/local/bin/restic -r /mnt/backup/autodb2/**repo1** backup /root/mysql_bak/db1 --exclude="*.sql" --password-file=/root/scripts/restic-password-file --verbose

/usr/local/bin/restic -r /mnt/backup/autodb2/**repo2** backup /root/mysql_bak/db2 --exclude="*.sql" --password-file=/root/scripts/restic-password-file --verbose

one repo is normal,and the other often error

[rawtaz]

@saberya @MichelEischer asked you if your /mnt/backup mount was an SMB share. You said no. Yet your latest information above (thanks for that) shows that it is indeed an SMB/CIFS mount. So Michael's suspicion is correct, and you need to act on the information he provided, to see if it helps - in particular this:

If this is a SMB mount then you have to set the environment variable GODEBUG to asyncpreemptoff=1 as documented in https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#local .

Why you say that the mount is not SMB, and that it is FTP, when it is not FTP and it is SMB/CIFS, is beyond my comprehension. Please be more accurate when you reply.

Please re-read what Michael wrote in #3455 (comment) to make sure you follow his advice the best you can.

[MichaelEischer]

Closing this issue as it seems to be resolved, feel free to complain if this is not the case. The underlying problem is tracked in #2659.