Not able to backup a running VM

[welsh1lad]

Version used
Provide output of virtnbdbackup -V
1.9.15

Describe the bug
When issuing the following command
virtnbdbackup -d terraform-jenkins-1 -l auto -o /media/rob/KvmBackups/MonthlyBackups/Jenkins
And the destination folder is empty a Full back is expected . This dose in deed happen when the command is ran again the auto switch is meant to complete an incremental backup - error

ERROR root virtnbdbackup - main [MainThread]: Cant execute switch to auto incremental backup: target folder seems not to include full backup.

Directory -
/media/rob/KvmBackups/MonthlyBackups/Jenkins$ ls -lsa
total 7318636
4 drwxrwxr-x 3 rob rob 4096 Aug 13 00:03 .
0 drwxrwxr-x 5 rob rob 49 Aug 6 17:55 ..
4 -rw-r--r-- 1 root root 1909 Aug 12 16:35 backup.auto.08122023163555.log
4 -rw-r--r-- 1 root root 2983 Aug 12 19:39 backup.auto.08122023193643.log
4 -rw-r--r-- 1 root root 448 Aug 12 23:01 backup.auto.08122023230101.log
4 -rw-r--r-- 1 root root 448 Aug 13 00:01 backup.auto.08132023000120.log
4 -rw-r--r-- 1 root root 407 Aug 13 00:03 backup.full.08132023000304.log
0 drwxr-xr-x 2 root root 6 Aug 12 16:35 checkpoints
7318604 -rw-r--r-- 1 root root 7494246574 Aug 12 19:39 vda.copy.data
4 -rw-r--r-- 1 root root 614 Aug 12 19:39 vda.copy.qcow.json
4 -rw-r--r-- 1 root root 2689 Aug 12 19:39 vmconfig.copy.xml

If I then run a full backup i get the following error
2023-08-13 00:03:04] INFO root virtnbdbackup - main [MainThread]: Backup level: [full]
[2023-08-13 00:03:04] ERROR root virtnbdbackup - main [MainThread]: Target directory already contains full or copy backup.
backup.full.08132023000304.log (END)

Contradicting the first error.

Running libvirtd - libvirtd (libvirt) 9.0.0
uname -a
Linux dsk-nix 6.4.7 #2 SMP PREEMPT_DYNAMIC Sun Jul 30 01:19:44 BST 2023 x86_64 x86_64 x86_64 GNU/Linux

Expected behavior
On first run of the command
virtnbdbackup -d terraform-jenkins-1 -l auto -o /media/rob/KvmBackups/MonthlyBackups/Jenkins
Full backup to be created
upon the second run of the above command auto switches mode to complete a incremental backup of the running VM.

Hypervisor information:

• OS: ubuntu

• Linux 6.4.7 No way to exclude disks #2 SMP PREEMPT_DYNAMIC Sun Jul 30 01:19:44 BST 2023 x86_64 x86_64 x86_64 GNU/Linux

• HV type = libvirtd (libvirt) 9.0.0

Logfiles:
[2023-08-12 23:01:01] INFO lib common - printVersion [MainThread]: Version: 1.9.15 Arguments: /usr/bin/virtnbdbackup -d terraform-jenkins-1 -l auto -o /media/rob/KvmBackups/MonthlyBackups/Jenkins [2023-08-12 23:01:01] INFO root virtnbdbackup - main [MainThread]: Backup level: [auto] [2023-08-12 23:01:01] ERROR root virtnbdbackup - main [MainThread]: Cant execute switch to auto incremental backup: target folder seems not to include full backup.

Workaround:
Delete backup folder contents. Turnoff the the virtual machine , then run the command to complete a full backup .
both issueing auto or inc as a subiquant command will fail.

[abbbi]

hi,

this looks strange to me, the backup folder includes a type copy backup, which cant be used as base
for incremental backups:

7318604 -rw-r--r-- 1 root root 7494246574 Aug 12 19:39 vda.copy.data

so there was in fact a copy backup executed instead of a full backup. I would need the log from the first backup
( backup.auto.08122023163555.log) executed on this folder to see what happened. Also: please update to the latest version available, and check
if you can reproduce this behavior.

I guess the virtual machine was offline (powered off) during the first backup, then the backup mode "auto" did
fallback to a copy backup backup because thats the only thing that works in this situation.

If the folder already includes an copy backup, no inc or full backups can be executed on this folder anymore, so this
error is intended.

[welsh1lad]

Hi there ,

Upgraded to - virtnbdbackup -V 1.9.36

Replicate Issue

1. Delete all files in the back up destination folder
2. Vm running on libvirtd (libvirt) 9.0.0
3. VM terraform-jenkins-1 in running - terraform-jenkins-1 stopped
4. backup command - sudo virtnbdbackup -d terraform-jenkins-1 -l auto -o /media/rob/KvmBackups/MonthlyBackups/Jenkins -v
5. Files created #KvmBackups/MonthlyBackups/Jenkins$ ls -lst
   total 7318720
   44 -rw-r--r-- 1 root root 43153 Aug 14 09:20 backup.auto.08142023091717.log
   4 -rw-r--r-- 1 root root 614 Aug 14 09:20 vda.copy.qcow.json
   4 -rw-r--r-- 1 root root 2689 Aug 14 09:20 vmconfig.copy.xml
   7318668 -rw-r--r-- 1 root root 7494312110 Aug 14 09:20 vda.copy.data
   0 drwxr-xr-x 2 root root 6 Aug 14 09:17 checkpoints
   6 Only a vda.copy.data is created and not full backup .

Expected : On first run into a empty folder in auto to be a fullcopy
As per instructions:
auto: If the target folder is empty, attempt to execute full backup, otherwise switch to backup mode incremental: allows rotation of backup into monthly folders.

logs
backup.auto.08142023091717.log

[abbbi]

hi,

the behavior is expected if the virtual machine is offline like it was in your case. You attempted an auto mode backup
on an offline virtual machine, which then did a fallback to a copy type backup:

   [2023-08-14 09:17:17] WARNING root virtnbdbackup - setOfflineArguments [MainThread]: Domain is offline, resetting backup options.
   [2023-08-14 09:17:17] INFO root virtnbdbackup - setOfflineArguments [MainThread]: Backup level: [copy].

if a virtual machine is offline no full backup can be created because we cannot add an checkpoint for a full backup
if its offline. Thats an limitation on libvirt side, ..

So you have to make sure virtual machine is online if you want the auto backup mode to create a full backup
for a new chain.

See the Notes section in:

https://github.com/abbbi/virtnbdbackup#backup-operation

[..]
Note:

If the virtual domain is not in running state (powered off) virtnbdbackup supports copy and inc/diff backup modes. Incremental and differential backups will then save the changed blocks since last created checkpoint. As no new checkpoints can be defined for offline domains, the Backup mode full is changed to mode copy.

[welsh1lad]

Hi ,
Many thanks for the quick feed back , but if the VM is online , no backups can be created due to dirty bitmap error

sudo virtnbdbackup -d terraform-jenkins-1 -l auto -o /media/rob/KvmBackups/MonthlyBackups/Jenkins -v
[sudo] password for rob:
[2023-08-14 09:15:53] INFO lib common - printVersion [MainThread]: Version: 1.9.36 Arguments: /usr/bin/virtnbdbackup -d terraform-jenkins-1 -l auto -o /media/rob/KvmBackups/MonthlyBackups/Jenkins -v
[2023-08-14 09:15:53] INFO root virtnbdbackup - main [MainThread]: Backup level: [auto]
[2023-08-14 09:15:53] INFO root virtnbdbackup - main [MainThread]: Backup mode auto, target folder is empty: executing full backup.
[2023-08-14 09:15:53] DEBUG virt client - _connect [MainThread]: Libvirt URI: [qemu:///system]
[2023-08-14 09:15:53] DEBUG virt client - _connect [MainThread]: Hostname: [dsk-nix]
[2023-08-14 09:15:53] DEBUG virt client - _connect [MainThread]: Connect libvirt using open function.
[2023-08-14 09:15:53] INFO root virtnbdbackup - main [MainThread]: Libvirt library version: [9000000]
[2023-08-14 09:15:53] DEBUG virt client - getDomainDisks [MainThread]: Disk [vda]: volume notation
[2023-08-14 09:15:53] DEBUG virt client - getDomainDisks [MainThread]: Device list: [DomainDisk(target='vda', format='qcow2', filename='terraform-jenkins-1-jenkins.qcow2', path='/var/lib/libvirt/images/jenkins/terraform-jenkins-1-jenkins.qcow2', backingstores=['/var/lib/libvirt/images/master_terraform_vol.qcow2'])]
[2023-08-14 09:15:53] DEBUG virt client - getDomainInfo [MainThread]: No setting [loader] found: 'NoneType' object has no attribute 'text'
[2023-08-14 09:15:53] DEBUG virt client - getDomainInfo [MainThread]: No setting [nvram] found: 'NoneType' object has no attribute 'text'
[2023-08-14 09:15:53] DEBUG virt client - getDomainInfo [MainThread]: No setting [kernel] found: 'NoneType' object has no attribute 'text'
[2023-08-14 09:15:53] DEBUG virt client - getDomainInfo [MainThread]: No setting [initrd] found: 'NoneType' object has no attribute 'text'
[2023-08-14 09:15:53] DEBUG virt client - getDomainInfo [MainThread]: Domain Info: [{}]
[2023-08-14 09:15:53] INFO root virtnbdbackup - main [MainThread]: Backup will save [1] attached disks.
[2023-08-14 09:15:53] INFO root virtnbdbackup - main [MainThread]: Concurrent backup processes: [1]
[2023-08-14 09:15:53] INFO root checkpoint - redefine [MainThread]: Loading checkpoint list from: [/media/rob/KvmBackups/MonthlyBackups/Jenkins/checkpoints]
[2023-08-14 09:15:53] INFO root checkpoint - create [MainThread]: Checkpoint handling.
[2023-08-14 09:15:53] DEBUG root checkpoint - removeAll [MainThread]: Cleaning up persistent storage /media/rob/KvmBackups/MonthlyBackups/Jenkins/checkpoints
[2023-08-14 09:15:53] INFO root checkpoint - create [MainThread]: Using checkpoint name: [virtnbdbackup.0].
[2023-08-14 09:15:53] DEBUG root checkpoint - create [MainThread]: Checkpoint info: [{'name': 'virtnbdbackup.0', 'parent': '', 'file': '/media/rob/KvmBackups/MonthlyBackups/Jenkins/terraform-jenkins-1.cpt'}].
[2023-08-14 09:15:53] INFO root virtnbdbackup - main [MainThread]: Local NBD Endpoint socket: [/var/tmp/virtnbdbackup.2119196]
[2023-08-14 09:15:53] INFO root virtnbdbackup - main [MainThread]: Temporary scratch file target directory: [/var/tmp]
[2023-08-14 09:15:53] INFO root virtnbdbackup - startBackupJob [MainThread]: Starting backup job.
[2023-08-14 09:15:53] DEBUG virt client - _createBackupXml [MainThread]: Using scratch file: /var/tmp/backup.6FG23.vda
[2023-08-14 09:15:53] DEBUG root xml - indent [MainThread]:








[2023-08-14 09:15:53] DEBUG root xml - indent [MainThread]:

Backup checkpoint
virtnbdbackup.0




[2023-08-14 09:15:53] DEBUG fs fs - freeze [MainThread]: Attempting to freeze filesystems.
[2023-08-14 09:15:53] INFO fs fs - freeze [MainThread]: Freezed [2] filesystems.
[2023-08-14 09:15:53] DEBUG virt client - startBackup [MainThread]: Starting backup job via libvirt API.
[2023-08-14 09:15:53] DEBUG fs fs - thaw [MainThread]: Attempting to thaw filesystems.
[2023-08-14 09:15:53] INFO fs fs - thaw [MainThread]: Thawed [2] filesystems.
[2023-08-14 09:15:53] ERROR root virtnbdbackup - startBackupJob [MainThread]: Failed to start backup: [internal error: unable to execute QEMU command 'transaction': Can't make bitmap 'virtnbdbackup.0' persistent in 'libvirt-1-format': Cannot store dirty bitmaps in qcow2 v2 files]

[abbbi]

Can't make bitmap 'virtnbdbackup.0' persistent in 'libvirt-1-format': Cannot store dirty bitmaps in qcow2 v2 files

your qcow image files are too old, they still use the old v2 format.
For persistent bitmaps, required for full and inc backup, you need to convert them to v3 format using qemu-img.

See also:

https://github.com/abbbi/virtnbdbackup#backup-fails-with-cannot-store-dirty-bitmaps-in-qcow2-v2-files

[welsh1lad]

Hi , Please can you re-open issue . Many thanks Rob

…

On 14 Aug 2023, at 09:43, Michael Ablassmeier ***@***.***> wrote: Can't make bitmap 'virtnbdbackup.0' persistent in 'libvirt-1-format': Cannot store dirty bitmaps in qcow2 v2 files+ your qcow image files are too old, they still use the old v2 format. For persistent bitmaps, you need to convert them to v3 format using qemu-img. See also: https://github.com/abbbi/virtnbdbackup#backup-fails-with-cannot-store-dirty-bitmaps-in-qcow2-v2-files <https://github.com/abbbi/virtnbdbackup#backup-fails-with-cannot-store-dirty-bitmaps-in-qcow2-v2-files> — Reply to this email directly, view it on GitHub <#125 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZR7OOVONAXNGAY3CLHJE3XVHQL7ANCNFSM6AAAAAA3OHNL7M>. You are receiving this because you modified the open/close state.

[abbbi]

Hi , Please can you re-open issue . Many thanks Rob

hi,

its not an issue in virtnbdbackup, you have to adjust your image files qcow version using qemu-img like
described in the README. Ive converted this into an discussion.