booting with bcachefs compression on is broken #261
Comments
codebam
changed the title
|
I can't do anything without logs. |
|
How to get the logs, except capturing the boot screen's image? The system does not boot, so |
|
Capture the boot image's screen, use a camera if you have to |
|
Here is a video in HEVC format https://r2.seanbehan.ca/b8a1e Relevant frame: https://r2.seanbehan.ca/8208d Edit: in H264: https://r2.seanbehan.ca/a1953 |
|
Hi, I've tried to increase verbosity but with not much luck, here's my transcript: (nixos, flake update included new kernel 6.8.8->6.8.9 and bcachefs-tools 1.41->1.70; rootfs with enabled encryption) |
|
I have the same or at least a similar problem with two of my systems, also from the nixpkgs upgrade of bcachefs-tools from 1.4.1 to 1.7.0. This is the superblock from one system with a single NVMe, the other system has two NVMes with 1 data replica. However, I also had an added problem where I also could not boot anymore with bcachefs-tools 1.4.1 when I use any of the 6.9-rc kernels. Kernel 6.8.9 works fine still, but the 6.9 wouldn't mount with a message like this: That just magically fixed itself on one of the systems now after trying again to get the exact phrasing right on this error message with 6.9-rc7. Previously this had even caused an error to be recorded in the superblock, The boot problem with bcachefs-tools 1.7.0 persists however, even though I am now using linux 6.9-rc7 and the superblock has been upgraded to bcachefs 1.7. A weird thing is that I can not reproduce this anymore with a fresh installation where the rootfs was created with bcachefs-tools 1.4.1. So only the two systems I created around the middle of January have these issues. Sorry if my description is a bit chaotic. 😅 |
|
@chaosbiber @tmuehlbacher What are you using in your fstab? device node, UUID=? Also it sounds like mostly single block device file system, not multi-device right? Update: Also what version of blkid are you using (check util-linux) |
generated from nixos-config fileSystems."/" =
{ device = "UUID=...";
fsType = "bcachefs";
options = [ "compression=lz4" ];
};
correct
show-super |
|
This is my fstab line, generated by NixOS:
Actually, one of my systems is multi-device. But both have the same problem. The |
|
Can you try 'bcachefs mount' with -v? |
|
Here is the terminal output from a separate live system, using the same nixpkgs In the first command it didn't ask for a password because the key was still in It works here in a full system, just not in the initramfs, apparently. |
|
So that looks like a keyring issue |
|
I've created a nix installer based on nix-unstable so it should use most of the same package versions. Ran it on the system where booting fails. If I haven't misspelled my passphrase in all 6 or 7 tries, then yes, might look like that password doesn't reach its destination. Edit: using kernel 6.8.9 with bcachefs-tools 1.7.0
Sorry for the dust, it's on my laptop, not your displays... |
|
Deleted because I just reproduced the issue mentioned also in the nixos wiki. |
|
@chaosbiber I can confirm I have the same issue. If you mount with |
|
@codebam Ah, right, I just reproduced an older issue for the 23.11 installer. But using the 1.7.0 again: with both the
|
|
I can confirm hitting this issue trying to upgrade bcachefs-tools to 1.6.x a month ago on NixOS. bcachefs-tools have always worked perfectly fine for me when run interactively from a rescue USB. (NixOS skipped bcachefs-tools 1.6x entirely; so if anyone's looking for root causes they're probably further back in time.) |
|
If I were to be suspicious of a particular commit here in Could some kind of subtle "bug" have been introduced with the conversion of main to Rust? For example - something subtle like changing the return code when executed using the |
If you are interested can probably run |
|
If I do run a bisect - do y'all consider it safe to use intermediate versions? Or should I restrict to tagged releases? |
|
I'd recommend staying on the safe side and test tagged releases first, then we can move from there. @koverstreet can confirm better though if it's safe to use intermediate commits. |
|
Alright; ran a bisect on my machine (restricting to tagged releases). The last good release was v1.6.3 and the first bad release is v1.6.4.
I also added a little extra logging in stage-1-init.sh; and can tell you a little bit more precisely about what regressed:
If it's relevant - I'm not running compression but am running encryption; and have a multi-device setup. |
|
@marcin-github would it be possible for you to test if you can reproduce this, on a NixOS machine? Not sure if it would be relevant in Fedora or Debian. |
Meseems you want to mention someone else :) |
|
Having the same issue. NixOS with Linux 6.8.9, bcachefs-tools 1.7.0, configuration: fileSystems."/" = {
device = "UUID=5f910790-3f93-4e9e-baf4-13b69719dc6a";
fsType = "bcachefs";
options = [
"compression=lz4"
"fix_errors=yes"
"nojournal_transaction_names"
"relatime"
"discard"
"background_compression=lz4"
];
};After typing my password on boot, getting an error: Edit: using encryption together with compression. |
Well you seem to be quite active in this repo testing |
|
Well, I'm not using nixos, I'm not using bcachefs as root fs and I'm not using enryption. What I can say is that -tools on latest commits works on my host. |
|
Got confirmation from @koverstreet on IRC that he thinks it should be reasonably safe to bisect further between v1.6.3 and v1.6.4; will hopefully have time to do so later tonight. @marcin-github: Sadly, the tools mostly work fine for me as well - just not in initramfs. @JohnRTitor: What's the precise reason for wanting to ship bcachefs-tools-1.7.0 with NixOS 24.05? Is 24.05 shipping with a 6.9 kernel? If not - I'd actually suggest that bcachefs-tools-1.6.x (specifically 1.6.3 because I believe it works) would be more appropriate because it matches the disk format in the 6.8 kernel? |
|
6.9 kernel release is due in a week, and 24.05 NixOS release will definitely ship with it (release schedule). The ISO's, by default, however will stick to 6.6 LTS releases, so this issue will not affect most of the users, except those who are actively using 6.9 kernel uses |
|
That diff doesn't have any obvious issues in it, unless it's the reordering of this check from before the password prompt to after it and inside of the decryption logic? Won't be able to try myself for at least several hours - but would be curious if moving it back to the start of 86049a1#diff-0988d5d2259e1a401001675578c38d09c46290b877b66b07fb96a7e034ccdf70R95-R98 |
|
@reedriley Yes actually, moving it back does in fact boot 12164c9 |
|
Yeah, I believe it makes a lot of sense. |
|
After rebasing 12164c9 onto 477670f (https://github.com/codebam/bcachefs-tools/commits/try-fix/ , https://github.com/codebam/bcachefs-tools/commits/fix-nixos-stage1) The password prompt doesn't read in passwords properly. It prompts, but pressing Edit: tried with @tmuehlbacher's commit as well and no dice, at least not on master |
let's always first check if there is already a key in the keyring available before we try to get the key from some more involved means. Fixes: koverstreet#261 Signed-off-by: Thomas Mühlbacher <tmuehlbacher@posteo.net>
|
I didn't really test that commit yet. |
let's always first check if there is already a key in the keyring available before we try to get the key from some more involved means. Fixes: koverstreet#261 Signed-off-by: Thomas Mühlbacher <tmuehlbacher@posteo.net>
let's always first check if there is already a key in the keyring available before we try to get the key from some more involved means. Fixes: koverstreet#261 Signed-off-by: Thomas Mühlbacher <tmuehlbacher@posteo.net>
|
@codebam thanks, so still some problems on master? |
Yeah but it's a different problem on master where you can't enter the passphrase. I'm on this commit with your commit and it boots fine 5531acc also 6.9-rc7 |
let's always first check if there is already a key in the keyring available before we try to get the key from some more involved means. Fixes: koverstreet#261 Signed-off-by: Thomas Mühlbacher <tmuehlbacher@posteo.net>
|
I pushed another commit to #263 that makes mounting work on master now. :) |
|
Please test the latest changes so that it is fixed in upstream. NixOS users can easily do so like JohnRTitor/nix-conf@b60df8a |
|
Tested and working on rc7 |
Hey I'm using NixOS, and I can't boot after updating to 1.7.0.
I use both encryption and compression.
This is my configuration: https://github.com/codebam/nixos (I've pinned 1.4.0 for the time being).
What I've tried is here: NixOS/nixpkgs#309388
The text was updated successfully, but these errors were encountered: