Easier way to recover from 'transaction': Bitmap already exists: virtnbdbackup.0] in cluster setup

[wkernpixel]

I see the FAQ on this issue, but I already have several VMs in this state and am looking at fixing them. Obviously going forward I will look at using a shared folder as suggested.
We already understood that we need to user a fresh backup folder and start again with a Level 0 on the new host when a migration occurs.

The VMs in question were live migrated from similar servers U22LTS Stock qemu/libvirt on shared media (Gluster). The migration works fine and we can go back and forth but when our virt backup script ran the newly migrated VMs failed with the above error

virtnbdbackup version is 1.42 (stable Ubuntu) the level was set to auto

if I shutdown the VM and use qemu-img info THERE IS NO BITMAP listed.

If we run the backup on a stopped VM it DOES do a full backup, but if the VM is running we get the bitmap exists issue, even after a previous shutdown or even a undefine/define of the XML to recreate the VM domain.

One trick that did seem to work was to use qemu-img to --add a bitmap file, check with qemu-img info to see that it is there and then --remove it. That seems to solve the problem but as is suggested in your notes messing with qemu-img is not recommended.

Is there an easier way to clear this problem? Preferably without having to shutdown the migrated VMs.

[wkernpixel]

correction on virtndbbackup version forgot the .9. but is still odd because the PACKAGE identifies itself as 1.9.42 in the deb package listing but 1.9.32 on the Version line

https://packages.debian.org/trixie/virtnbdbackup

virtnbdbackup -V

1.9.32

[abbbi]

thats strange, i cant reproduce this in a fresh debian/sid instance:

root@radiohead-unstable-amd64-nh1u:/srv# dpkg -l virtnbdbackup
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  virtnbdbackup  1.9.42-1     all          Backup utility for libvirt
root@radiohead-unstable-amd64-nh1u:/srv# virtnbdbackup --version
1.9.42

i guess you have some leftover files with a different libvirtndbackup version in your python search path.

[wkernpixel]

yes, I did apt purge and reinstalled 1.9.42 deb and it shows correctly

[abbbi]

Is there an easier way to clear this problem? Preferably without having to shutdown the migrated VMs.

in any case if the virtual machines migrate between systems, you need to use the transient virtual machine
checkpoint folder which you need to share between hosts as described in:

https://github.com/abbbi/virtnbdbackup#transient-virtual-machines-checkpoint-persistency

IF the virtual machine is OFFLINE, only a type "copy" backup is created, which will not create any bitmap nor will
create any checkpoint: you simply dont run into the problem.

If you store the checkpoints metadata within a folder shared between systems, it at least allows you to create
a new full backup on the new host the virtual machine has been moved to without the bitmap mess happening.

[wkernpixel]

understood. I saw that in the FAQ when I was researching the issue. Going forward we need to use the shared folder on a migration.

I guess for the existing VMs that are in that state I need to do add the fake bitmap and remove it trick.
I wonder if that is a bug in libvirt/qemu?

There are similar issues mentioned on RedHat bugzilla where they talk about a leftover bitmap but they all indicate the problem was solved by 2020.

[abbbi]

if you have that issue now just remove the bitmaps from the qcow files using qemu-img while vm is offline.

Its not quite an issue in libvirt, its just that the checkpoints are not shared between libvirt environments during
migration. I dont know wether if this has already been addressed in newer libvirt versions or will. If you migrate
a virtual machine to a different system the checkpoints information is not migrated to the target system. So the qcow
images contain the bitmaps tied to the checkpoint, but libvirt doesnt know about them anymore.

This results in the situation that virtnbdbackup doenst see the checkpoint via libvirt api and attempts to create a new
one. This fails because the bitmap is already existant. In case you use the checkpoint-folder it attempts to re-define
the checkpoints with different flags (metadata only) so the existant bitmap is tied to the new checkpoint and a new
full backup can happen on the new host.

[wkernpixel]

yep, all is good.
Thanks all for clueing me in on this bitmap issue. I understand the issue now and have made and tested the appropriate changes to the virtnbd script and everything works fine

[abbbi]

note from IRC in #virt channel:

21:10 <abi> if there is a virtual machine mirated from host a to host b, using live migration, the existant checkpoints are not migrated? But the qcow images on the new host will still 
            contain the bitmap information. Is this something that will be handled in future versions?
21:17 <pkrempa> abi: there are no current plans that I know of to work on the incremental backup in the near future
21:19 <pkrempa> but it's pretty simple (but tedious) to copy and redefine the metadata on the destination

so it seems for current libvirt versions theres no other solution.