borg prune : Object with key not found in repository

[tconstans]

Hello,

I am try to get myself out of a complicated situation.

A few weeks ago, i violently interrupted a borg backup. Repository got in a state where i had to do several borg repair to make the backup functional again.

But despite all effort, i am unable to prune my old backups, I get numerous errors likethe following errors when trying:

Remote: Borg 1.0.10: exception in RPC call:
Remote: Traceback (most recent call last):
Remote: File "borg/remote.py", line 145, in serve
Remote: File "borg/repository.py", line 550, in delete
Remote: borg.repository.Repository.ObjectNotFound: Object with key b'\xfd\x8d\x96\xd0a\xdfo\x85\xf6\xee\xc5\xb6\xb6&\x15\x92r\t\xe2\x8b*U\x96D+d\x8a\xa6J\xe5\x14\x0e' not found in repository /srv/nas/CUSTOMER/mail.
Remote: Platform: Linux FOO.BAR.fr 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64
Remote: Linux: CentOS Linux 7.3.1611 Core
Remote: Borg: 1.0.10 Python: CPython 3.5.3
Remote: PID: 11395 CWD: /srv/nas/CUSTOMER
Remote: sys.argv: ['/usr/bin/borg1.0.10', 'serve', '--restrict-to-path', '/srv/nas/CUSTOMER/']
Remote: SSH_ORIGINAL_COMMAND: '/usr/bin/borg1.0.10 serve --umask=077 --info'

borg check --repair is actually emptying some chunks, but fails to make my backup manageable

Thanks for your help,

Best regards

[ThomasWaldmann]

ObjectNotFound exception here means that it tried to delete a key from the repo index - but the key was not in the index. The delete operation is likely called from the borg client because you are pruning.

So, either your repo is corrupt (== the data storage actually misses stuff that should be there) or at least the indexes are somehow inconsistent with the repository.

What you can try (on your on risk - if it is important, do all experiments on a COPY of the repo):

• kill the clientside caches (not the repository): borg delete --cache-only repo
• try the prune again

If that still throws same/similar error there is a small chance that your repo index is corrupt, but the data in the repo is fine. Then you could try:

• kill the repo index and the hints file (they are inside the repository directory, called hints.N and index.N)
• use borg check --repair repo to rebuild the index.
  If your repo is big and you work from remote, this can take a long time. But after that, IF the data in your repo was still fine, your repo index will be consistent with the data. If the repair crashes / is unsuccessful, then the data in your repo is corrupted.

[ThomasWaldmann]

BTW, usually interrupting borg shouldn't get a repo into such a miserable state. If there is just some uncommitted stuff, that usually is rolled back automatically when you use the repo next time.
So maybe consider other reasons, like hardware issues - check your filesystem, your disk(s) (SMART values) and your RAM.

[tconstans]

Thanks for the quick feedback, i'll try that this evening.

It might be possible that repo is in such state because of 2 concurrent borg backup running

[tconstans]

Thomas,
I followed your instructions successfully. I had to do the whole process, including "killing" index and hints files on repo, then repairing.

I have then been able to successfully prune older backups.

The whole process ( kill local cache, try prune, kill remote hints and index file, repair, prune --keep-daily20 and then prune --keep-daily 7) took about 120 minutes. Data size is 40GB, repo is remote.

Thanks for the quick and effective feedback.

Keep up the good work

[tuxick]

Maybe have the error message print something like "Trying running borg check" ?