New permissions handling in 1.2.0 causes re-backup of files that haven't changed #988
Comments
|
Hi Germar, hope all is good! 😀
I too downgraded to one version lower available on my ppa: After this, peace and quiet has returned, all is good, and BiT performs with its previous accuracy, speed, and without flaws. FYI: currently I'm on Xubuntu 18.04 LTS HWE (kernel 4.18.0-18). Hope this helps... |
|
I have found one more issue after downgrading: the cron settings were
completely reset by 1.2.0.
--
Sent from my Samsung Galaxy S7
…On 2019. április 28. 20:04:49 BelladonnavGFH ***@***.***> wrote:
Hi Germar, hope all is good!
Like @kosal75 I too got updated today, and ran into many issues. These were:
My previous snapshots (over 2 years worth) were no longer recognised/
found. Although they were visible in the BiT-GUI, a new update triggered a
new "virgin" rsync of all files on all disks.
Having said that, $HOME/.config/backintime/config was found and recognised
though and executed without errors.
As @kosal75 writes, the rsync is extremely slow. With my backup disk on a
type C connection, write speeds are blow < 1MB/s
After the BiT/ rsync has been running for a while, my backup disk is
unmounted. As a result BiT throws endless notification errors.
I too downgraded to one version lower available on my ppa:
sudo apt install backintime-qt4=1.1.12-2 backintime-common=1.1.12-2
backintime-gnome=1.1.12-2
sudo apt-mark hold backintime-common backintime-qt4 backintime-gnome
After this, all is good, and BiT performs with its previous accuracy,
speed, and without flaws.
FYI: currently I'm on Xubuntu 18.04 LTS HWE (kernel 4.18.0-18).
Hope this helps...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Thanks @BelladonnavGFH for your tip! I haven't tried to do a backup with 1.2.0 and cannot say anything about the slowness, but the old backups wasn't, as you said, recognized any longer and I panicked. I'm also on Xubuntu 18.04 and it wasn't even possible to rezise the window, it was super big! Not possible to get it smaller. |
|
No thanks @pehlm. If it now works for you, perfect! |
|
I can confirm the slow backup, although all my backups were found and no full backup was done. Although looking at the logs I was able to see that apparently there were many files backuped, which actually didn't change. |
|
In the Ubuntu bionic ppa for bit "stable", when bit 1.2 was added, 1.1.24 was removed. Can it be put back, please? There were many bugs fixed between the version in the official Ubuntu repositories (1.1.12?) and 1.1.24, which is why I was using the ppa. I was fortunate enough to see the change coming before it was applied, and disabled the ppa in order to keep the version I've been using until I have time to figure out if 1.2 will work for me. Which will take a lot longer if I have no easy way to get back to 1.1.24 on a system that I try 1.2 on. Thank you for working on this very useful software. |
|
After updating to 1.2.0 BiT does a (nearly) full backup because file permissions are handled differently. Before 1.2.0 all destination file permissions were set to -rw-r--r--. In 1.2.0 rsync is executed with --perms option which tells rsync to preserve the source file permission. That's why so many files seem to be changed. |
|
@jns is it actually backing them up again (and consuming extra disc space) or is it just changing the file permissions, which takes time but does not use any extra disc space? |
|
I think it's backing up the files again. My last (and first 1.2.0) backup took roughly 1 hour and created a snapshot [WITH ERRORS]. However, no error messages were written to the log file. After deleting that snapshot disk usage dropped from 35% to 19% (~300G). Unfortunately I did not check file inodes before deleting that snapshot. Considering the amount of freed disk space, I assume BiT made a complete backup, except files where permissions matched -rw-r--r--. |
|
In 1.1.24 (and I don't know how many earlier versions), there is a per-profile setting for "Full rsync mode". If you turned it on, it copied permissions to the backup directory with rsync instead of saving them in a file. I think I saw a change log comment somewhere that said 1.2 dropped support for the previous default mode and only supports "Full rsync mode". I wonder if the transition is smoother for backups that were already using "Full rsync mode"? Is there any documentation anywhere on how to make the switch to the new version without tears? |
|
Hey all, Like @jns- already pointed out, BiT will now let rsync change the permissions directly in the snapshot. At the first snapshot with the new version this will make rsync calculate checksums for all source and all snapshot files to make sure, they are definitely the same. Which of course takes a long time. Normally rsync only compares modification-time, size and permission to tell if a file has changed. With the next snapshot rsync should be fast as before or even faster because of the optimized code. To confirm you could compare the inodes of files that didn't change. Take a look at this FAQ: How can I check if my snapshots are incremental (using hard-links)? Also you can check the infos If you have lots of files that didn't change but are marked with |
|
And what happens on a drive not supporting Linux file permissions? I also
use samba mount.
--
Sent from my Samsung Galaxy S7
…On 2019. május 2. 20:29:38 Germar Reitze ***@***.***> wrote:
Hey all,
sorry for the inconvenience with version 1.2Like @jns- already pointed out,
BiT will now let rsync change the permissions directly in the snapshot. At
the first snapshot with the new version this will make rsync calculate
checksums for all source and all snapshot files to make sure, they are
definitely the same. Which of course takes a long time. Normally rsync only
compares modification-time, size and permission to tell if a file has
changed. With the next snapshot rsync should be fast as before or even
faster because of the optimized code.
To confirm you could compare the inodes of files that didn't change. Take a
look at this FAQ: How can I check if my snapshots are incremental (using
hard-links)?
Also you can check the infos rsync provides for each file it processed.
From man rsync:
The attribute that is associated with each letter is as follows: o A c
means either that a regular file has a different checksum (requires
--checksum) or that a symlink, device, or special file has a changed value.
Note that if you are sending files to an rsync prior to 3.0.1, this change
flag will be present only for checksum-differing regular files. o A s means
the size of a regular file is different and will be updated by the file
transfer. o A t means the modification time is different and is being
updated to the sender’s value (requires --times). An alter‐ nate value of T
means that the modification time will be set to the transfer time, which
happens when a file/symlink/device is updated without --times and when a
symlink is changed and the receiver can’t set its time. (Note: when using
an rsync 3.0.0 client, you might see the s flag combined with t instead of
the proper T flag for this time-setting failure.) o A p means the
permissions are different and are being updated to the sender’s value
(requires --perms). o An o means the owner is different and is being
updated to the sender’s value (requires --owner and super-user privileges).
o A g means the group is different and is being updated to the sender’s
value (requires --group and the authority to set the group). o The u slot
is reserved for future use. o The a means that the ACL information changed.
o The x means that the extended attribute information changed.
If you have lots of files that didn't change but are marked with c this
could indicate a failing harddrive. You should run fsck and consider
replacing the drive.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
@Germar I just ran a test on Ubuntu 18.04 backing up to local drive. I created a new backup with version 1.1.24 on default settings then upgraded (via the bit team stable ppa) to 1.2.0 and ran another backup. In the log this showed every file with [C] cf...p..... and the size of the backup doubled (measured with ncdu and inode values checked just to be sure). I am not sure from your previous post whether that is expected or not. |
BiT still stores permissions inside the |
No, thats not expected. Please hang on, I need to test this. Will take a bit as I need to prepare a test VM first which is an Ubuntu 16.04 updated to 18.04. Otherwise I'm not able to install 1.1.24 in 18.04 |
|
Hmmm All fine so far. But this is stange: EDIT: removed the last part as it was wrong... |
|
No, enabling Investigating a random file that did not change since 2017: BiT last log entry Running results [truncated] Inode of that file is 83624689 for all past snapshots and changes to 83758208 in |
|
What brought me back to business is following: I do not yet know if this procedure creates any regrettable side effects! |
|
@jns- Whaa! You lost me there... What exactly did you do? If I take a look at |
Ja, I realized my mistake after laying my head on the pillow last night. So I turned out again and deleted the last part of my comment 😆 Today I'd say you should rather ignore my last post at all 🙈 I did some further testing which now shows the root of the problem: So, changing permissions of one hard-link will change permissions in all hard-links. Which, after knowing the fact, is totally logical because a hard-link is just a pointer to the inode. It doesn't store anything else. Not even permissions. |
|
In my previous posts I used the term 'pre 1.2.0' for BiT versions older than 1.2.0. From now on let's assume it was version 1.1.4 for the sake of better readability. My intention was to prevent BiT 1.2.0 to create a full backup on top of my 1.1.4 snapshot history. The idea was to modify the 1.1.4 snapshot history outside of BiT and make it look ok to BiT 1.2.0 next time it is started. Like @Germar explains, hard links are just pointers to files on the disk and do not store any additional information. Permissions are part of the file the hard link is pointing to. BiT crates a snapshot folder called For example in
33204 in octal representation is 100664. The last 3 digits represent the original file permission, which corresponds to What I did in detail:
I used
for step 4. (works in Z shell, probably not in bash). The last I recommend to test permission modification first on a subset of backup files by truncating Running the one liner may take some time (~350GB corresponding to 200k files, took ~15min on my laptop). Optimize it according to your needs and show us something way more elegand 😏 Note 1: As I mentioned in a post above, I do not know if there are any side effects. Note 2: This procedure does not clean up your complete snapshot history. It only affects the latest version of your backup files. If you restore a file from far in the past the permissions will most probably be wrong. |
|
@jns I can confirm that procedure works for me on Ubuntu, thanks. I had to first install zsh to run it (which is no big deal), it would be nice if someone more knowledgeable that myself could do an sh or bash version in order to avoid this necessity. I can suggest a slight enhancement which avoids manually unpacking the file. The procedure is: cd path/to/backups/last_snapshot
cat fileinfo.bz2 | bzip2 -d | while read line; do perm=`echo "${line}" | cut -d" " -f1`; path_=`echo ${line} | cut -d" " -f4-`; perm=$(([##8]perm)); perm="${perm:$((${#perm}-3)):3}"; echo chmod "$perm" "backup$path_"; doneAs with @jns's version that will just echo all the commands it will perform, once happy that looks correct then repeat the command but without the last cat fileinfo.bz2 | bzip2 -d | while read line; do perm=`echo "${line}" | cut -d" " -f1`; path_=`echo ${line} | cut -d" " -f4-`; perm=$(([##8]perm)); perm="${perm:$((${#perm}-3)):3}"; chmod "$perm" "backup$path_"; doneYou can do that by hitting up arrow then left arrow to the echo text and delete the word echo. |
|
Thanks @jns- and @colinl for sharing this! I'm sure I'll give it a try too (as soon as I can get these pesky little humans squatting my box to play SuperTuxKart to go away). |
|
Thanks @jns- and @colinl for the fix. I'm planing to implement this to run automatically. But as I'm on vacation for the next weeks this will take some time...
BiT will still restore permissions from |
|
Building on the work of @jns- and @colinl, I added user and group. For my last 1.1.4 backup, this was necessary to prevent another full backup. cat fileinfo.bz2 | bzip2 -d | while read line; do perm=`echo "${line}" | cut -d" " -f1`; path_=`echo ${line} | cut -d" " -f4-`; user_=`echo "${line}" | cut -d" " -f2`; group_=`echo "${line}" | cut -d" " -f3`; perm=$(([##8]perm)); perm="${perm:$((${#perm}-3)):3}"; chmod "$perm" "backup$path_"; chown "$user_":"$group_" "backup$path_"; done |
|
Upon further review, something is wonky beyond just correcting the permissions, owners, and groups in the last good 1.1.4 snapshot. I've gone through by hand and confirmed that the permissions, owners, and groups in my source files are identical to those files in last_snapshot, but nonetheless, some files continue to be backed up in full (files that haven't changed in years). Running the above command did prevent many of the files from being backed up in full again, but it was not a panacea. |
|
@wiregrasscoder Do the problematic files have unusual permissions in the source? I am seeing it with r--r--r--. If so perhaps you are falling over issue #994 |
|
@colinl I have not seen the r--r--r-- issue. The permissions on the files in my scenario do have setuid and setgid applied. Also I'm also seeing that after a certain point during the snapshot, the snapshot size begins to grow very quickly, and log entries stop, so I'm not sure what's happening. |
|
Hi, same problem here: updated from 1.1.24 to 1.2.0. In https://github.com/bit-team/backintime/blob/master/common/tools.py I found a no_perms flag, how can I use this, is this something I could write in my config? Would the behaviour of 1.2.0 with this config-entry be the same as with pre 1.2.0 (without groups, users, perms). It would be fine for me to have permissions not synced. Thanks! |
In my environment I never modified the rsync arguments and always used the defaults. |
|
Just a quick n dirty note: I realized that there is a |
|
I do read the whole issue thread and wonder if I got this right. We have a lot of comments here and there are multiple issues and problems addressed. Other relevant tickets are linked. But the problem described here happens only when migrating from |
To my knowledge, yes, that's correct. I can't remember if new installations of >=1.2 are also affected, so we might treat that as a "maybe" for the moment. |
buhtz
modified the milestones:
upcoming release (1.3.4),
after the upcoming release (1.3.5 or 1.4.0)
That might be the case. I've recently started using Ubuntu 20.04 with BIT 1.2.1-2 and I have this issue. Adding the PPA and upgrading to BIT 1.3.3-3 hasn't helped. I detected the issue because the same backup on the very same machine was taking wayyy longer, and then I noticed that the target drive was getting filled up way faster than it should. My old setup (that I can still boot into) is Ubuntu 14.04 with BIT 1.0.34 where the same backup profile still works fine, even with more included paths. I don't think this matters, but one of the source paths is an NTFS drive. However, the target is EXT4, so there should not be permission issues specific to my case. As @emtiu, I can also confirm that the workaround of adding |
But is I think we should test this to see the behavior (full recopy or not). |
Summary:I vote to undo the new permission handling (
ReasonsThe intentions of the new permission handling were
Effectively the new permissions handling led to problems like
Alternatives
We have PR #1086 for this (thanks to @b3nmore for preparing this PR!).
This is only a partial fix, requires a lot of scenario testings and would not solve the other issues (slow first snapshot; full file copy if permissions are changed). Impact of a non-fix
Next steps@Germar @buhtz @emtiu I think it is time now to take a decision here -> RfC 😄 |
|
Thank you for the deep analysis, @aryoda, I agree with every point of it. Your proposed solution also minimizes the necessary testing of the handling of existing backups, because there will only be a few cases:
Since this is an "existential" issue for BiT, I think @Germar's input is especially important. |
|
Awesome work! Thanks a lot for diving into this. ❤️ One question: Jürgen, did you contacted Wayne about our issues? And did you point him to his own upstream bug? Second question: |
It would definitely solve #994. We don't understand #988 and #1437 well enough yet to say. Maybe the upstream fix would resolve those, maybe not. In any case, it would potentially take a long time for the fixed version of rsync to appear in all distros. Changing/fixing the behavior of BiT is much more under our control. The only point that makes me a little nervous is the handling of existing backups. We need to be very thorough in testing that. But I think that's within our capabilities. |
Not yet, I have just sent a public request at the
As @emtiu wrote: Not reliably. Furthermore it does not fix things I have just added in my above analysis:
Yes, it needs testing, but it is a direct "downgrade" (= does no longer treat permissions changes as a change) so it seems much less risky than keeping the new permissions handling. |
How about someone create an experimental branch that implements this fix/revert? It would be very useful for testing, and we're going to need a lot of testing :) |
I think I can do this but I need some time (I guess until end of October - I am in the middle of another major roll-out)
We need a test plan for that with a matrix of backup and restore scenarios:
We then should automate these tests
Eg. my MRE bash script |
I would even go so far as to do some tests with real, large datasets on real hard-drives. Some problems you only notice in "real world" scenarios. I have plenty of large external drives lying around to do that :) |
|
I fully agree that we should do heave real-world testing here. |
This comment was marked as abuse.
This comment was marked as abuse.
|
I think we could automate testing of the snapshot source and target folders quite "easily" (not only for this issue) similar to how the It basically uses
|
|
F***ing awesome! 🥳 🎉 🪅 Never realized that rsync itself could have a test suite. This is a very good "documentation" of its behavior. |
|
Hi! I found this bug while trying to find a solution to an issue I am having with BiT 1.2.1. I upgraded my system from Ubuntu 18.04 to Ubuntu Server 22.04. I then installed BiT 1.2.1 (which was an upgrade for me). I ran the initial scan and it went well. Since then, it is running a full scan frequently (not every time, but most times). The previous version just updated what had changed. Is this happening because the bug wasn't fixed before this update? Or is there possibly something else going on? Thank you for any thoughts/suggestions. |
and others
I have just updated from 1.1.24 to 1.2.0 (common, gnome, qt4), and making a snapshot takes hours. When downgrading to 1.1.24, all is normal (5-6 minutes, as always).
Ubuntu 18.10 with kernel 4.18.0-18.
The text was updated successfully, but these errors were encountered: