Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pi3B+ : USB+ethernet file transfer problem #2449

Closed
toto8551 opened this issue Mar 18, 2018 · 204 comments
Closed

Pi3B+ : USB+ethernet file transfer problem #2449

toto8551 opened this issue Mar 18, 2018 · 204 comments

Comments

@toto8551
Copy link

toto8551 commented Mar 18, 2018

Hi,

I have problems while transferring files from an USB external HDD or USB flash drive connected on the new RPi 3B+ to another computer via SFTP or samba. After a random amount of data transfered, the transfer rate drop to 0 and stay like that for at least 30 min ( I canceled the transfer after that time).

I cannot see any error on the standard raspbian system logs. I am running raspbian stretch with the last updates, I also did a firmware upgrade with rpi-update.

The RPi 3B+ is powered with an official raspberry power supply (2.5A) and I tried with a spare one.
I tried to give more power with an external power supply for the USB HDD without any change, but like I got the same problem with an USB flash drive, I don't expect a power issue or ?

I tried to transfer files with thunar, pcmanfm, scp or rsync. rsync is the only one working, but with a strong fluctuation of the transfer rate (from 0 to 20Mo/s). scp tell me that the file is "stalled" and stop any progress.

If I start the same SD card with a RPi 3B (not plus) the file transfer is working like a charm ( I used it for almost 3 years like that).

I followed tips from this link without successI having no problems during ssh sessions or with vnc.

Any ideas ?

@mkreisl
Copy link

mkreisl commented Mar 18, 2018

Having same strange issues with Pi3B+ [1] [2]

IMO it's network driver (lan78xx) related or the usb host driver does not handle usb transfers correctly.

After switching from eth to wlan, everything is ok

[1] Was never able to make an 3GB image backup (root fs is on external usb disk) over network (kernel 4.9, 4.14 and 4.15 tested). Same configuration works on an older Pi3B without any issues

[2] When root fs is on iSCSI target, got massive kernel OOPS when using kernel 4.14.27+. After switching to 4.9.87+, ethernet connection was better (no kernel OOPS), but maybe still not stable (more testing needed). Kernel 4.15.10+ not tested

@toto8551
Copy link
Author

I did another test.

Now I am back with the official kernel of the raspbian stretch : 4.9.80-v7, and it is working up to now.
I transfered 10Go from an USB HDD with a rate of 20 Mo/s.

@pjgpetecodes
Copy link

I've got the same problem here... Trying to transfer large files over ethernet fails after a minute or so... Fine on Wifi... However, when I try to downgrade to 4.9.80-v7, I get four flashing green lights on the PI3B+... I can take the card out and plug it into a Pi2 and it boots up just fine... I then have to re-update the firmware and swap everything back to the Pi3B+... Frustrating...

@fieryo
Copy link

fieryo commented Mar 21, 2018

Is this bug related to #2446? If so, then the forthcoming kernel 4.14.28 should fix it. Reverting to an old kernel may be a temporary solution, but only for advanced users.

Are other distros affected: xbian, osmc, LibreELEC?

At least we know that OSMC used pre-production units, so may be they included a fix/workaround to this problem.

I have 10 days to decide to return the RPi 3B+, so I want to start with a working Ethernet or just return the unit unopened.

@mkreisl
Copy link

mkreisl commented Mar 21, 2018

Is this bug related to #2446? If so, then the forthcoming kernel 4.14.28 should fix it. Reverting to an old kernel may be a temporary solution, but only for advanced users.

IMO there is a good chance to solve some strange issues.

As already reported, I'm struggling with my Pi3B+ since I got that part. Raspbian seems to work in my testing scenario [1], but XBian was crashing always, sooner or later. The major difference between Raspbian and XBian is, that Rasbian is using ipv6, whereas XBian is using ipv4 (ipv6 is disabled per default), and those patches fixing the ipv4 stack. After enabling tcp6 on XBian, my test seems so finish successfully.

[1] Root fs on usb disk (60GB), mounting Samba share on my server to /mnt, and then run dd if=/dev/sda of=/mnt/test.img bs=1M status=process. Raspbian succeeded, XBian crashes always in different ways: kernel oops, kernel panic of just completely freezed.

@pelwell
Copy link
Contributor

pelwell commented Mar 22, 2018

If you have a Linux target machine, can you try running this one-liner (you'll need to change the login credentials for the target):

dd if=/dev/zero bs=1M status=progress | ssh user@target "cat >/dev/null"

I've had that running overnight and it's approaching 2TB transferred without issue. The channel is encrypted, so the 29.4MB/s (235Mb/s) it's achieving isn't bad.

If that's solid, switch to:

while true; do date 1>&2; sudo dd if=/dev/mmcblk0 bs=1M status=progress; done | ssh user@target "cat >/dev/null"

Then:

while true; do date 1>&2; sudo dd if=/dev/sda bs=1M status=progress; done | ssh user@target "cat >/dev/null"

@mkreisl
Copy link

mkreisl commented Mar 22, 2018

@pelwell
Sure, there are many many ways to run a test. Unfortunately, after my tests has been finished successfully, my real application making an image backup started from Kodi GUI still kills the Pi3B+ completely [1], in different ways (4.14.27+ mostly throws kernel Oops #5, so built kernel with netdev patch, Oops were gone, but now still getting VFS CIFS stuck for more than 15s message, and Kernel 4.14.29+ build tonight did not make any difference).

Conclusion:
Pi3B+ is at the moment absolutely unusable, crappiest Pi ever had. Extremely frustrating

[1] That backup is using btrfs send/receive, sends all local btrfs subvolumes (located on external USB disk) to an image mounted on a network share (cifs of nfs).

@pelwell
Copy link
Contributor

pelwell commented Mar 22, 2018

Pi3B+ is at the moment absolutely unusable, crappiest Pi ever had. Extremely frustrating

For you, apparently, but not for me - that's why I'm trying to be methodical and work out which step is causing the problem.

@Ferroin
Copy link
Contributor

Ferroin commented Mar 22, 2018

@mkreisl Out of curiosity, could you possibly try using sshfs instead of CIFS or NFS? It's a FUSE module instead of a native filesystem driver, and it uses a completely different (and much more efficient) protocol on-the-wire, so it may work even though CIFS and NFS aren't. sudo apt-get install sshfs should get it for you (I'm about 99% certain Raspbian has it). If that works, it should at least give you something you can use in the short term until this gets figured out.

@mkreisl
Copy link

mkreisl commented Mar 22, 2018

For you, apparently, but not for me - that's why I'm trying to be methodical and work out which step is causing the problem.

Those basic tests you are running does cause the issue. IMO many many data in and out of the USB is causing the problem. That's the main difference between Pi3B nd Pi3B+, Pi3B has one usb hub, and Pi3B+ has two:

Pi3B:

Bus 001 Device 004: ID 046d:c503 Logitech, Inc. Cordless Mouse+Keyboard Receiver
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. SMC9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Pi3B+:

Bus 001 Device 004: ID 046d:c503 Logitech, Inc. Cordless Mouse+Keyboard Receiver
Bus 001 Device 005: ID 0424:7800 Standard Microsystems Corp. 
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

@mkreisl
Copy link

mkreisl commented Mar 22, 2018

@Ferroin Good idea, I'll test it. Takes some time, have to modify the backup script

@mkreisl
Copy link

mkreisl commented Mar 22, 2018

@Ferroin sshfs doesn't make a difference, Pi3B+ dying with kernel Oops again (Pi3B works like a charm)

Mar 22 17:40:00 kmxbilr2 kernel: [  555.453472] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
Mar 22 17:40:00 kmxbilr2 kernel: [  555.453480] pgd = a12d8000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.453484] [0000000d] *pgd=00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.453493] Internal error: Oops: 5 [#1] SMP ARM
Mar 22 17:40:00 kmxbilr2 kernel: [  555.453578] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454081] CPU: 1 PID: 10626 Comm: sudo Tainted: G         C      4.14.29+ #1
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454170] Hardware name: BCM2835
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454216] task: 91ab1e00 task.stack: 93ea0000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454287] PC is at locks_remove_posix+0x30/0x14c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454364] LR is at filp_close+0x68/0x8c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454447] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 20000013
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454550] sp : 93ea1eb0  ip : 93ea1f58  fp : 93ea1f54
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454616] r10: 00000000  r9 : 93ea0000  r8 : 80108204
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454680] r7 : 00000006  r6 : acc63900  r5 : 9d640180  r4 : ae690440
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454754] r3 : 00000001  r2 : ad1bb940  r1 : acc63900  r0 : 9d640180
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454831] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454914] Control: 10c5383d  Table: 212d806a  DAC: 00000055
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454983] Process sudo (pid: 10626, stack limit = 0x93ea0210)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455058] Stack: (0x93ea1eb0 to 0x93ea2000)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455112] 1ea0:                                     802fe824 91ab1e00 00000044 802a8f5c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455210] 1ec0: 00000004 00000000 00000000 00000100 81f22e80 80d093bc 00000017 808ac294
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455316] 1ee0: 76ebfe2c 93ea1fb0 7ed36760 808a8780 93ea1f14 93ea1f00 808a8780 806fbae8
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455440] 1f00: add51c00 add51d74 93ea1f34 93ea1f18 806fbae8 808a7758 add52400 7f0220bc
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455561] 1f20: add52400 add52444 93ea1f54 00000000 9d640180 acc63900 00000006 80108204
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455679] 1f40: 93ea0000 00000000 93ea1f74 93ea1f58 80286ac4 802e088c 0000003c acc63900
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455787] 1f60: 9d640180 00000006 93ea1f94 93ea1f78 802aad38 80286a68 76fbd218 0000000b
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455892] 1f80: 00000000 00000006 93ea1fa4 93ea1f98 80286b18 802aac7c 00000000 93ea1fa8
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455993] 1fa0: 80108060 80286af4 76fbd218 0000000b 0000003c 7ed36760 76fbcb5c 76fbcb5c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456102] 1fc0: 76fbd218 0000000b 00000000 00000006 01d185b8 7ed36760 76fface8 0000003c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456205] 1fe0: 7ed36738 7ed36728 76fbcb6c 76e94fc2 60000030 0000003c 00000000 00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456325] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456435] [<80286ac4>] (filp_close) from [<802aad38>] (__close_fd+0xc8/0xec)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456529] [<802aad38>] (__close_fd) from [<80286b18>] (SyS_close+0x30/0x58)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456626] [<80286b18>] (SyS_close) from [<80108060>] (ret_fast_syscall+0x0/0x28)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456728] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456857] ---[ end trace 124dc21b94998788 ]---
Mar 22 17:40:00 kmxbilr2 kernel: [  555.477670] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
Mar 22 17:40:00 kmxbilr2 kernel: [  555.477836] pgd = 80004000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.477896] [0000000d] *pgd=00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.477964] Internal error: Oops: 5 [#2] SMP ARM
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478032] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478580] CPU: 3 PID: 10620 Comm: sudo Tainted: G      D  C      4.14.29+ #1
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478678] Hardware name: BCM2835
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478725] task: 82818f00 task.stack: 86ea8000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478800] PC is at locks_remove_posix+0x30/0x14c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478867] LR is at filp_close+0x68/0x8c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478933] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 20040013
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479016] sp : 86ea9d08  ip : 86ea9db0  fp : 86ea9dac
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479084] r10: 0000000b  r9 : ad41a980  r8 : 00000004
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479152] r7 : ade91500  r6 : ade91500  r5 : 9d640180  r4 : ae690440
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479234] r3 : 00000001  r2 : 9d640180  r1 : ade91500  r0 : 9d640180
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479316] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479407] Control: 10c5383d  Table: 109d806a  DAC: 00000055
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479486] Process sudo (pid: 10620, stack limit = 0x86ea8210)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479561] Stack: (0x86ea9d08 to 0x86eaa000)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479621] 9d00:                   80d87980 809024bc 2e486000 808a6840 af11ad40 86ea9d48
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479728] 9d20: 8c3da480 8025bc28 ae811a40 80e15480 60040013 60040013 0000008c 00033c4b
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479832] 9d40: 00000000 808a8780 86ea9d6c 86ea9d58 808a8780 806fbae8 add51c00 add51d74
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479956] 9d60: 86ea9d8c 86ea9d70 806fbae8 808a7758 add52400 7f0220bc add52400 add52444
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480075] 9d80: 86ea9dac 00000000 9d640180 ade91500 ade91500 00000004 ad41a980 0000000b
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480184] 9da0: 86ea9dcc 86ea9db0 80286ac4 802e088c 00000009 000000f0 00000000 ade91500
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480287] 9dc0: 86ea9df4 86ea9dd0 802aa864 80286a68 82818f00 82819440 ade91500 ae811a40
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480389] 9de0: 00000001 ae811a78 86ea9e14 86ea9df8 802aa974 802aa7bc 82818f00 00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480493] 9e00: 00000544 ae811a40 86ea9e54 86ea9e18 80121d6c 802aa928 86ea9e74 86ea9e28
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480595] 9e20: 86ea9edc 86ea9fb0 00000000 0000000b ac9a02c0 86ea9edc 86ea8000 00106001
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480698] 9e40: 86ea8000 0000000b 86ea9e74 86ea9e58 8012260c 801219e0 00000000 ad6aba88
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480800] 9e60: 86ea9edc 86ea8000 86ea9ec4 86ea9e78 8012d9f4 801225cc 80d02040 418004fc
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480903] 9e80: 80d03d68 86ea9ec8 ac9a02c0 ad6abec4 ad6ab9c0 000000a0 0000000b 76e27574
Mar 22 17:40:00 kmxbilr2 kernel: [  555.494505] 9ea0: 86ea9ec8 86ea9fb0 76e27576 00000000 86ea8000 00000000 86ea9f8c 86ea9ec8
Mar 22 17:40:00 kmxbilr2 kernel: [  555.506477] 9ec0: 8010b318 8012d6c8 86ea9ef4 86ea9ed8 8012d084 8012cf08 00000000 0000000b
Mar 22 17:40:00 kmxbilr2 kernel: [  555.518136] 9ee0: 00000000 00000000 0000297c 00000000 ad6ab9e8 00000001 82818f00 00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.531395] 9f00: 00000000 0000297c 00000000 ad6ab9e8 00000001 82818f00 00000001 86ea9f68
Mar 22 17:40:00 kmxbilr2 kernel: [  555.545698] 9f20: 86ea9f64 86ea9f30 8012f17c 801edc50 82819444 00000002 7ed36aa8 00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.558183] 9f40: 00000002 7ed36aa8 000000ae 80108204 86ea8000 00000000 86ea9fa4 86ea9f68
Mar 22 17:40:00 kmxbilr2 kernel: [  555.570528] 9f60: 8012f6f0 00000001 86ea8010 80108204 86ea9fb0 80108204 86ea8000 00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.582517] 9f80: 86ea9fac 86ea9f90 8010b820 8010b260 7ed36aa8 0000000b 0000000b 00000025
Mar 22 17:40:00 kmxbilr2 kernel: [  555.594335] 9fa0: 00000000 86ea9fb0 80108094 8010b774 00000000 0000000b 3a4d6900 3a4d6900
Mar 22 17:40:00 kmxbilr2 kernel: [  555.607730] 9fc0: 7ed36aa8 0000000b 0000000b 00000025 01d15c58 0049016c 00000000 7ed36a18
Mar 22 17:40:00 kmxbilr2 kernel: [  555.621658] 9fe0: 004a1dd8 7ed369c4 0047dd5b 76e27576 20040030 0000297c a302000d 64690063
Mar 22 17:40:00 kmxbilr2 kernel: [  555.635304] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.647889] [<80286ac4>] (filp_close) from [<802aa864>] (put_files_struct+0xb4/0x10c)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.660265] [<802aa864>] (put_files_struct) from [<802aa974>] (exit_files+0x58/0x5c)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.672651] [<802aa974>] (exit_files) from [<80121d6c>] (do_exit+0x398/0xba0)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.684677] [<80121d6c>] (do_exit) from [<8012260c>] (do_group_exit+0x4c/0xe4)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.696657] [<8012260c>] (do_group_exit) from [<8012d9f4>] (get_signal+0x338/0x6e0)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.708835] [<8012d9f4>] (get_signal) from [<8010b318>] (do_signal+0xc4/0x3e4)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.720791] [<8010b318>] (do_signal) from [<8010b820>] (do_work_pending+0xb8/0xd0)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.732744] [<8010b820>] (do_work_pending) from [<80108094>] (slow_work_pending+0xc/0x20)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.744414] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
Mar 22 17:40:00 kmxbilr2 kernel: [  555.755952] ---[ end trace 124dc21b94998789 ]---
Mar 22 17:40:00 kmxbilr2 kernel: [  555.772247] Fixing recursive fault but reboot is needed!
Mar 22 17:40:02 kmxbilr2 kernel: [  557.572209] Unable to handle kernel paging request at virtual address 04bc7ffc
Mar 22 17:40:02 kmxbilr2 kernel: [  557.583366] pgd = a0380000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.594624] [04bc7ffc] *pgd=00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.605878] Internal error: Oops: 5 [#3] SMP ARM
Mar 22 17:40:02 kmxbilr2 kernel: [  557.616992] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
Mar 22 17:40:02 kmxbilr2 kernel: [  557.639993] CPU: 3 PID: 10496 Comm: btrfs Tainted: G      D  C      4.14.29+ #1
Mar 22 17:40:02 kmxbilr2 kernel: [  557.651518] Hardware name: BCM2835
Mar 22 17:40:02 kmxbilr2 kernel: [  557.662917] task: 9851bc00 task.stack: a01e4000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.674480] PC is at __d_lookup_rcu+0x68/0x19c
Mar 22 17:40:02 kmxbilr2 kernel: [  557.685867] LR is at lookup_fast+0x4c/0x2c8
Mar 22 17:40:02 kmxbilr2 kernel: [  557.697094] pc : [<802a4da0>]    lr : [<80295cc4>]    psr: 20010013
Mar 22 17:40:02 kmxbilr2 kernel: [  557.708335] sp : a01e5d20  ip : 80d04590  fp : a01e5d5c
Mar 22 17:40:02 kmxbilr2 kernel: [  557.719619] r10: a01e5e50  r9 : 00000006  r8 : e6ad3f44
Mar 22 17:40:02 kmxbilr2 kernel: [  557.730983] r7 : a03d8550  r6 : a03d8550  r5 : 00000000  r4 : 04bc8000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.741942] r3 : 0001cd5a  r2 : 00004000  r1 : 00000000  r0 : a03d8550
Mar 22 17:40:02 kmxbilr2 kernel: [  557.752961] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Mar 22 17:40:02 kmxbilr2 kernel: [  557.763554] Control: 10c5383d  Table: 2038006a  DAC: 00000055
Mar 22 17:40:02 kmxbilr2 kernel: [  557.774189] Process btrfs (pid: 10496, stack limit = 0xa01e4210)
Mar 22 17:40:02 kmxbilr2 kernel: [  557.784760] Stack: (0xa01e5d20 to 0xa01e6000)
Mar 22 17:40:02 kmxbilr2 kernel: [  557.795359] 5d20: 8c6f2000 a01e5d6c 00000006 8c6f0038 a01e5d5c a01e5e48 00000000 a01e5da8
Mar 22 17:40:02 kmxbilr2 kernel: [  557.806204] 5d40: a03d8550 a01e5da0 ad639b10 a01e5da4 a01e5d9c a01e5d60 80295cc4 802a4d44
Mar 22 17:40:02 kmxbilr2 kernel: [  557.817041] 5d60: 8c6f2000 ae4085a0 00001455 00000006 00001542 a01e5e48 00000000 00000003
Mar 22 17:40:02 kmxbilr2 kernel: [  557.828472] 5d80: 8c6f003f a01e5e48 8cff9c59 a03d8550 a01e5dd4 a01e5da0 80298140 80295c84
Mar 22 17:40:02 kmxbilr2 kernel: [  557.843155] 5da0: a01e5dc4 a01e5db0 8029683c 8042ea20 c2d0f82e 47090a62 e6ad3f44 8c6f003f
Mar 22 17:40:02 kmxbilr2 kernel: [  557.855354] 5dc0: a01e5e48 8cff9c59 a01e5e24 a01e5dd8 8029856c 80298110 80295448 61c88647
Mar 22 17:40:02 kmxbilr2 kernel: [  557.866613] 5de0: 00000000 00000006 a01e5e48 8c6f0010 a01e5e24 a01e5e00 80294f74 8c6f0010
Mar 22 17:40:02 kmxbilr2 kernel: [  557.878046] 5e00: a01e5e48 a01e5f38 a01e5f38 a01e5f40 00000000 ffffff9c a01e5e44 a01e5e28
Mar 22 17:40:02 kmxbilr2 kernel: [  557.889410] 5e20: 802988c4 802983ec 8c6f0000 a01e5e48 00000000 a01e5f38 a01e5eec a01e5e48
Mar 22 17:40:02 kmxbilr2 kernel: [  557.900746] 5e40: 8029a4b4 8029889c ad639b10 a03d8550 e6ad3f44 00000006 8c6f0038 80276ee4
Mar 22 17:40:02 kmxbilr2 kernel: [  557.912237] 5e60: ae990c10 ae7af660 883d6660 00000050 00000006 000003a4 00000000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.923647] 5e80: 00000000 a01e5e88 80d0459c 8c6f0000 80d0459c 7ea7a81c 00000000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.934994] 5ea0: a01e5edc a01e5eb0 8029a30c 805d1904 a01e5e88 8c6f4000 8c6f0000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.946359] 5ec0: 00000026 00000002 ffffff9c ffffff9c 8c6f4000 7ea7981c a01e5f50 00000026
Mar 22 17:40:02 kmxbilr2 kernel: [  557.957715] 5ee0: a01e5f8c a01e5ef0 8029b984 8029a444 a01e5f50 a01e5f28 98785c08 a01e5ef0
Mar 22 17:40:02 kmxbilr2 kernel: [  557.969150] 5f00: 00000001 802ad51c 98785c00 00000800 00000000 00000000 7ea7a81c ffffff9c
Mar 22 17:40:02 kmxbilr2 kernel: [  557.980558] 5f20: 00000000 00000000 98785c08 00000000 ad639b10 962fd330 a01e4000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.992053] 5f40: aa64a66c 0000000a 8c6f402a 80276ee4 00000000 00000000 a01e5f7c 98785c00
Mar 22 17:40:02 kmxbilr2 kernel: [  558.003698] 5f60: 98785c00 00000000 00e9a010 00e9a050 00000026 80108204 a01e4000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  558.015188] 5f80: a01e5fa4 a01e5f90 8029bd9c 8029b8bc 00000000 00e9a050 00000000 a01e5fa8
Mar 22 17:40:02 kmxbilr2 kernel: [  558.026723] 5fa0: 80108060 8029bd74 00000000 00e9a010 7ea7981c 7ea7a81c ffffffff 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  558.038198] 5fc0: 00000000 00e9a010 00e9a050 00000026 004fd000 76f05ce8 7ea7981c 7ea7a81c
Mar 22 17:40:02 kmxbilr2 kernel: [  558.049811] 5fe0: 004fd198 7ea79814 0049cc9b 76cd7796 00010030 7ea7981c 00000000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  558.061386] [<802a4da0>] (__d_lookup_rcu) from [<80295cc4>] (lookup_fast+0x4c/0x2c8)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.072944] [<80295cc4>] (lookup_fast) from [<80298140>] (walk_component+0x3c/0x2dc)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.084330] [<80298140>] (walk_component) from [<8029856c>] (link_path_walk+0x18c/0x4b0)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.095582] [<8029856c>] (link_path_walk) from [<802988c4>] (path_parentat+0x34/0x6c)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.106851] [<802988c4>] (path_parentat) from [<8029a4b4>] (filename_parentat+0x7c/0x104)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.117923] [<8029a4b4>] (filename_parentat) from [<8029b984>] (SyS_renameat2+0xd4/0x48c)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.129156] [<8029b984>] (SyS_renameat2) from [<8029bd9c>] (SyS_rename+0x34/0x3c)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.140092] [<8029bd9c>] (SyS_rename) from [<80108060>] (ret_fast_syscall+0x0/0x28)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.150966] Code: ea000002 e5944000 e3540000 0a000028 (e5141004) 
Mar 22 17:40:02 kmxbilr2 kernel: [  558.161756] ---[ end trace 124dc21b9499878a ]---
Mar 22 17:40:05 kmxbilr2 kernel: [  560.624738] Alignment trap: not handling instruction e1b04f9f at [<80566d48>]
Mar 22 17:40:05 kmxbilr2 kernel: [  560.635458] Unhandled fault: alignment exception (0x001) at 0x9ac47754
Mar 22 17:40:05 kmxbilr2 kernel: [  560.646194] pgd = acecc000
Mar 22 17:40:05 kmxbilr2 kernel: [  560.656802] [9ac47754] *pgd=1ac1141e(bad)
Mar 22 17:40:05 kmxbilr2 kernel: [  560.667391] Internal error: : 1 [#4] SMP ARM
Mar 22 17:40:05 kmxbilr2 kernel: [  560.677894] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
Mar 22 17:40:05 kmxbilr2 kernel: [  560.700191] CPU: 3 PID: 4194 Comm: PeripBusUSBUdev Tainted: G      D  C      4.14.29+ #1
Mar 22 17:40:05 kmxbilr2 kernel: [  560.711520] Hardware name: BCM2835
Mar 22 17:40:05 kmxbilr2 kernel: [  560.724080] task: 81945a00 task.stack: 9e712000
Mar 22 17:40:05 kmxbilr2 kernel: [  560.737866] PC is at lockref_put_return+0x3c/0x94
Mar 22 17:40:05 kmxbilr2 kernel: [  560.750567] LR is at dput+0x40/0x2d0
Mar 22 17:40:05 kmxbilr2 kernel: [  560.761780] pc : [<80566d4c>]    lr : [<802a1abc>]    psr: 20060013
Mar 22 17:40:05 kmxbilr2 kernel: [  560.773122] sp : 9e713de0  ip : 9e713e00  fp : 9e713dfc
Mar 22 17:40:05 kmxbilr2 kernel: [  560.784351] r10: 00000000  r9 : 9e712000  r8 : 9e713f60
Mar 22 17:40:05 kmxbilr2 kernel: [  560.795460] r7 : 00000001  r6 : 00010001  r5 : 00000001  r4 : 00010001
Mar 22 17:40:05 kmxbilr2 kernel: [  560.806753] r3 : 00000000  r2 : 00010001  r1 : 00000000  r0 : 9ac47754
Mar 22 17:40:05 kmxbilr2 kernel: [  560.817978] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Mar 22 17:40:05 kmxbilr2 kernel: [  560.829007] Control: 10c5383d  Table: 2cecc06a  DAC: 00000055
Mar 22 17:40:05 kmxbilr2 kernel: [  560.839867] Process PeripBusUSBUdev (pid: 4194, stack limit = 0x9e712210)
Mar 22 17:40:05 kmxbilr2 kernel: [  560.851294] Stack: (0x9e713de0 to 0x9e714000)
Mar 22 17:40:05 kmxbilr2 kernel: [  560.863891] 3de0: 9ac47704 00080040 9ac47754 00000041 9e713e24 9e713e00 802a1abc 80566d1c
Mar 22 17:40:05 kmxbilr2 kernel: [  560.876801] 3e00: 00000000 9e713e78 9e713f60 00000041 9e713f60 9e712000 9e713e44 9e713e28
Mar 22 17:40:05 kmxbilr2 kernel: [  560.887838] 3e20: 80294b9c 802a1a88 9e713e78 fffffffe 9e713f60 00000041 9e713e74 9e713e48
Mar 22 17:40:05 kmxbilr2 kernel: [  560.900010] 3e40: 802989bc 80294b5c 9e6683f8 af57c6bc 00000000 00000000 00000001 8c6f5000
Mar 22 17:40:05 kmxbilr2 kernel: [  560.915033] 3e60: 9e713e78 00000001 9e713f24 9e713e78 8029a5d8 80298908 ae990310 9ac47704
Mar 22 17:40:05 kmxbilr2 kernel: [  560.929842] 3e80: e73d77c1 00000006 8c6f5025 00000000 ae990c10 ae7af660 ae7db2b0 00000001
Mar 22 17:40:05 kmxbilr2 kernel: [  560.942629] 3ea0: 9e713dd0 000003be 00000000 00000000 00000000 9e713eb8 00000000 00000000
Mar 22 17:40:05 kmxbilr2 kernel: [  560.955273] 3ec0: 00001000 8c6f6000 00000000 00000001 80d0459c 8c6f5000 80d0459c 6b2fea18
Mar 22 17:40:05 kmxbilr2 kernel: [  560.966619] 3ee0: 00000000 00000001 8c6f5000 00000000 8029a30c 00000002 ffffff9c 00000001
Mar 22 17:40:05 kmxbilr2 kernel: [  560.978523] 3f00: ffffff9c 00000001 ffffff9c 9e713f60 ffffff9c 6b2fea18 9e713f4c 9e713f28
Mar 22 17:40:05 kmxbilr2 kernel: [  560.992103] 3f20: 8029a720 8029a548 00000000 00000000 9e713f4c 00000001 00000000 acb3e980
Mar 22 17:40:05 kmxbilr2 kernel: [  561.005850] 3f40: 9e713f94 9e713f50 80287344 8029a6d8 00000000 00000000 aaa67500 00000000
Mar 22 17:40:05 kmxbilr2 kernel: [  561.017377] 3f60: 00000000 00000000 00000005 00000000 5c21ee28 76f78ce8 00000021 80108204
Mar 22 17:40:05 kmxbilr2 kernel: [  561.029029] 3f80: 9e712000 00000000 9e713fa4 9e713f98 802874c4 802872b4 00000000 9e713fa8
Mar 22 17:40:05 kmxbilr2 kernel: [  561.040579] 3fa0: 80108060 802874ac 00000000 5c21ee28 6b2fea18 00000000 00000000 6b2fea33
Mar 22 17:40:05 kmxbilr2 kernel: [  561.052969] 3fc0: 00000000 5c21ee28 76f78ce8 00000021 6b2fea18 6b2fea48 716d7b30 ffffffea
Mar 22 17:40:05 kmxbilr2 kernel: [  561.068944] 3fe0: 75eebf10 6b2fea04 75ee4b7b 759c1b66 20060030 6b2fea18 aa98abba faaaaaea
Mar 22 17:40:05 kmxbilr2 kernel: [  561.083256] [<80566d4c>] (lockref_put_return) from [<802a1abc>] (dput+0x40/0x2d0)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.094752] [<802a1abc>] (dput) from [<80294b9c>] (terminate_walk+0x4c/0xc0)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.106786] [<80294b9c>] (terminate_walk) from [<802989bc>] (path_lookupat+0xc0/0x204)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.120543] [<802989bc>] (path_lookupat) from [<8029a5d8>] (filename_lookup+0x9c/0xf8)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.133721] [<8029a5d8>] (filename_lookup) from [<8029a720>] (user_path_at_empty+0x54/0x5c)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.144836] [<8029a720>] (user_path_at_empty) from [<80287344>] (SyS_faccessat+0x9c/0x1f8)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.155787] [<80287344>] (SyS_faccessat) from [<802874c4>] (SyS_access+0x24/0x28)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.166755] [<802874c4>] (SyS_access) from [<80108060>] (ret_fast_syscall+0x0/0x28)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.177608] Code: e1a07005 da000015 f590f000 e1b04f9f (e1340006) 
Mar 22 17:40:05 kmxbilr2 kernel: [  561.188475] ---[ end trace 124dc21b9499878b ]---

... and dmesg reports

[  555.453472] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
[  555.453480] pgd = a12d8000
[  555.453484] [0000000d] *pgd=00000000
[  555.453493] Internal error: Oops: 5 [#1] SMP ARM
[  555.453578] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  555.454081] CPU: 1 PID: 10626 Comm: sudo Tainted: G         C      4.14.29+ #1
[  555.454170] Hardware name: BCM2835
[  555.454216] task: 91ab1e00 task.stack: 93ea0000
[  555.454287] PC is at locks_remove_posix+0x30/0x14c
[  555.454364] LR is at filp_close+0x68/0x8c
[  555.454447] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 20000013
[  555.454550] sp : 93ea1eb0  ip : 93ea1f58  fp : 93ea1f54
[  555.454616] r10: 00000000  r9 : 93ea0000  r8 : 80108204
[  555.454680] r7 : 00000006  r6 : acc63900  r5 : 9d640180  r4 : ae690440
[  555.454754] r3 : 00000001  r2 : ad1bb940  r1 : acc63900  r0 : 9d640180
[  555.454831] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  555.454914] Control: 10c5383d  Table: 212d806a  DAC: 00000055
[  555.454983] Process sudo (pid: 10626, stack limit = 0x93ea0210)
[  555.455058] Stack: (0x93ea1eb0 to 0x93ea2000)
[  555.455112] 1ea0:                                     802fe824 91ab1e00 00000044 802a8f5c
[  555.455210] 1ec0: 00000004 00000000 00000000 00000100 81f22e80 80d093bc 00000017 808ac294
[  555.455316] 1ee0: 76ebfe2c 93ea1fb0 7ed36760 808a8780 93ea1f14 93ea1f00 808a8780 806fbae8
[  555.455440] 1f00: add51c00 add51d74 93ea1f34 93ea1f18 806fbae8 808a7758 add52400 7f0220bc
[  555.455561] 1f20: add52400 add52444 93ea1f54 00000000 9d640180 acc63900 00000006 80108204
[  555.455679] 1f40: 93ea0000 00000000 93ea1f74 93ea1f58 80286ac4 802e088c 0000003c acc63900
[  555.455787] 1f60: 9d640180 00000006 93ea1f94 93ea1f78 802aad38 80286a68 76fbd218 0000000b
[  555.455892] 1f80: 00000000 00000006 93ea1fa4 93ea1f98 80286b18 802aac7c 00000000 93ea1fa8
[  555.455993] 1fa0: 80108060 80286af4 76fbd218 0000000b 0000003c 7ed36760 76fbcb5c 76fbcb5c
[  555.456102] 1fc0: 76fbd218 0000000b 00000000 00000006 01d185b8 7ed36760 76fface8 0000003c
[  555.456205] 1fe0: 7ed36738 7ed36728 76fbcb6c 76e94fc2 60000030 0000003c 00000000 00000000
[  555.456325] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
[  555.456435] [<80286ac4>] (filp_close) from [<802aad38>] (__close_fd+0xc8/0xec)
[  555.456529] [<802aad38>] (__close_fd) from [<80286b18>] (SyS_close+0x30/0x58)
[  555.456626] [<80286b18>] (SyS_close) from [<80108060>] (ret_fast_syscall+0x0/0x28)
[  555.456728] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
[  555.456857] ---[ end trace 124dc21b94998788 ]---
[  555.477670] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
[  555.477836] pgd = 80004000
[  555.477896] [0000000d] *pgd=00000000
[  555.477964] Internal error: Oops: 5 [#2] SMP ARM
[  555.478032] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  555.478580] CPU: 3 PID: 10620 Comm: sudo Tainted: G      D  C      4.14.29+ #1
[  555.478678] Hardware name: BCM2835
[  555.478725] task: 82818f00 task.stack: 86ea8000
[  555.478800] PC is at locks_remove_posix+0x30/0x14c
[  555.478867] LR is at filp_close+0x68/0x8c
[  555.478933] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 20040013
[  555.479016] sp : 86ea9d08  ip : 86ea9db0  fp : 86ea9dac
[  555.479084] r10: 0000000b  r9 : ad41a980  r8 : 00000004
[  555.479152] r7 : ade91500  r6 : ade91500  r5 : 9d640180  r4 : ae690440
[  555.479234] r3 : 00000001  r2 : 9d640180  r1 : ade91500  r0 : 9d640180
[  555.479316] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  555.479407] Control: 10c5383d  Table: 109d806a  DAC: 00000055
[  555.479486] Process sudo (pid: 10620, stack limit = 0x86ea8210)
[  555.479561] Stack: (0x86ea9d08 to 0x86eaa000)
[  555.479621] 9d00:                   80d87980 809024bc 2e486000 808a6840 af11ad40 86ea9d48
[  555.479728] 9d20: 8c3da480 8025bc28 ae811a40 80e15480 60040013 60040013 0000008c 00033c4b
[  555.479832] 9d40: 00000000 808a8780 86ea9d6c 86ea9d58 808a8780 806fbae8 add51c00 add51d74
[  555.479956] 9d60: 86ea9d8c 86ea9d70 806fbae8 808a7758 add52400 7f0220bc add52400 add52444
[  555.480075] 9d80: 86ea9dac 00000000 9d640180 ade91500 ade91500 00000004 ad41a980 0000000b
[  555.480184] 9da0: 86ea9dcc 86ea9db0 80286ac4 802e088c 00000009 000000f0 00000000 ade91500
[  555.480287] 9dc0: 86ea9df4 86ea9dd0 802aa864 80286a68 82818f00 82819440 ade91500 ae811a40
[  555.480389] 9de0: 00000001 ae811a78 86ea9e14 86ea9df8 802aa974 802aa7bc 82818f00 00000000
[  555.480493] 9e00: 00000544 ae811a40 86ea9e54 86ea9e18 80121d6c 802aa928 86ea9e74 86ea9e28
[  555.480595] 9e20: 86ea9edc 86ea9fb0 00000000 0000000b ac9a02c0 86ea9edc 86ea8000 00106001
[  555.480698] 9e40: 86ea8000 0000000b 86ea9e74 86ea9e58 8012260c 801219e0 00000000 ad6aba88
[  555.480800] 9e60: 86ea9edc 86ea8000 86ea9ec4 86ea9e78 8012d9f4 801225cc 80d02040 418004fc
[  555.480903] 9e80: 80d03d68 86ea9ec8 ac9a02c0 ad6abec4 ad6ab9c0 000000a0 0000000b 76e27574
[  555.494505] 9ea0: 86ea9ec8 86ea9fb0 76e27576 00000000 86ea8000 00000000 86ea9f8c 86ea9ec8
[  555.506477] 9ec0: 8010b318 8012d6c8 86ea9ef4 86ea9ed8 8012d084 8012cf08 00000000 0000000b
[  555.518136] 9ee0: 00000000 00000000 0000297c 00000000 ad6ab9e8 00000001 82818f00 00000000
[  555.531395] 9f00: 00000000 0000297c 00000000 ad6ab9e8 00000001 82818f00 00000001 86ea9f68
[  555.545698] 9f20: 86ea9f64 86ea9f30 8012f17c 801edc50 82819444 00000002 7ed36aa8 00000000
[  555.558183] 9f40: 00000002 7ed36aa8 000000ae 80108204 86ea8000 00000000 86ea9fa4 86ea9f68
[  555.570528] 9f60: 8012f6f0 00000001 86ea8010 80108204 86ea9fb0 80108204 86ea8000 00000000
[  555.582517] 9f80: 86ea9fac 86ea9f90 8010b820 8010b260 7ed36aa8 0000000b 0000000b 00000025
[  555.594335] 9fa0: 00000000 86ea9fb0 80108094 8010b774 00000000 0000000b 3a4d6900 3a4d6900
[  555.607730] 9fc0: 7ed36aa8 0000000b 0000000b 00000025 01d15c58 0049016c 00000000 7ed36a18
[  555.621658] 9fe0: 004a1dd8 7ed369c4 0047dd5b 76e27576 20040030 0000297c a302000d 64690063
[  555.635304] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
[  555.647889] [<80286ac4>] (filp_close) from [<802aa864>] (put_files_struct+0xb4/0x10c)
[  555.660265] [<802aa864>] (put_files_struct) from [<802aa974>] (exit_files+0x58/0x5c)
[  555.672651] [<802aa974>] (exit_files) from [<80121d6c>] (do_exit+0x398/0xba0)
[  555.684677] [<80121d6c>] (do_exit) from [<8012260c>] (do_group_exit+0x4c/0xe4)
[  555.696657] [<8012260c>] (do_group_exit) from [<8012d9f4>] (get_signal+0x338/0x6e0)
[  555.708835] [<8012d9f4>] (get_signal) from [<8010b318>] (do_signal+0xc4/0x3e4)
[  555.720791] [<8010b318>] (do_signal) from [<8010b820>] (do_work_pending+0xb8/0xd0)
[  555.732744] [<8010b820>] (do_work_pending) from [<80108094>] (slow_work_pending+0xc/0x20)
[  555.744414] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
[  555.755952] ---[ end trace 124dc21b94998789 ]---
[  555.772247] Fixing recursive fault but reboot is needed!
[  557.572209] Unable to handle kernel paging request at virtual address 04bc7ffc
[  557.583366] pgd = a0380000
[  557.594624] [04bc7ffc] *pgd=00000000
[  557.605878] Internal error: Oops: 5 [#3] SMP ARM
[  557.616992] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  557.639993] CPU: 3 PID: 10496 Comm: btrfs Tainted: G      D  C      4.14.29+ #1
[  557.651518] Hardware name: BCM2835
[  557.662917] task: 9851bc00 task.stack: a01e4000
[  557.674480] PC is at __d_lookup_rcu+0x68/0x19c
[  557.685867] LR is at lookup_fast+0x4c/0x2c8
[  557.697094] pc : [<802a4da0>]    lr : [<80295cc4>]    psr: 20010013
[  557.708335] sp : a01e5d20  ip : 80d04590  fp : a01e5d5c
[  557.719619] r10: a01e5e50  r9 : 00000006  r8 : e6ad3f44
[  557.730983] r7 : a03d8550  r6 : a03d8550  r5 : 00000000  r4 : 04bc8000
[  557.741942] r3 : 0001cd5a  r2 : 00004000  r1 : 00000000  r0 : a03d8550
[  557.752961] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  557.763554] Control: 10c5383d  Table: 2038006a  DAC: 00000055
[  557.774189] Process btrfs (pid: 10496, stack limit = 0xa01e4210)
[  557.784760] Stack: (0xa01e5d20 to 0xa01e6000)
[  557.795359] 5d20: 8c6f2000 a01e5d6c 00000006 8c6f0038 a01e5d5c a01e5e48 00000000 a01e5da8
[  557.806204] 5d40: a03d8550 a01e5da0 ad639b10 a01e5da4 a01e5d9c a01e5d60 80295cc4 802a4d44
[  557.817041] 5d60: 8c6f2000 ae4085a0 00001455 00000006 00001542 a01e5e48 00000000 00000003
[  557.828472] 5d80: 8c6f003f a01e5e48 8cff9c59 a03d8550 a01e5dd4 a01e5da0 80298140 80295c84
[  557.843155] 5da0: a01e5dc4 a01e5db0 8029683c 8042ea20 c2d0f82e 47090a62 e6ad3f44 8c6f003f
[  557.855354] 5dc0: a01e5e48 8cff9c59 a01e5e24 a01e5dd8 8029856c 80298110 80295448 61c88647
[  557.866613] 5de0: 00000000 00000006 a01e5e48 8c6f0010 a01e5e24 a01e5e00 80294f74 8c6f0010
[  557.878046] 5e00: a01e5e48 a01e5f38 a01e5f38 a01e5f40 00000000 ffffff9c a01e5e44 a01e5e28
[  557.889410] 5e20: 802988c4 802983ec 8c6f0000 a01e5e48 00000000 a01e5f38 a01e5eec a01e5e48
[  557.900746] 5e40: 8029a4b4 8029889c ad639b10 a03d8550 e6ad3f44 00000006 8c6f0038 80276ee4
[  557.912237] 5e60: ae990c10 ae7af660 883d6660 00000050 00000006 000003a4 00000000 00000000
[  557.923647] 5e80: 00000000 a01e5e88 80d0459c 8c6f0000 80d0459c 7ea7a81c 00000000 00000000
[  557.934994] 5ea0: a01e5edc a01e5eb0 8029a30c 805d1904 a01e5e88 8c6f4000 8c6f0000 00000000
[  557.946359] 5ec0: 00000026 00000002 ffffff9c ffffff9c 8c6f4000 7ea7981c a01e5f50 00000026
[  557.957715] 5ee0: a01e5f8c a01e5ef0 8029b984 8029a444 a01e5f50 a01e5f28 98785c08 a01e5ef0
[  557.969150] 5f00: 00000001 802ad51c 98785c00 00000800 00000000 00000000 7ea7a81c ffffff9c
[  557.980558] 5f20: 00000000 00000000 98785c08 00000000 ad639b10 962fd330 a01e4000 00000000
[  557.992053] 5f40: aa64a66c 0000000a 8c6f402a 80276ee4 00000000 00000000 a01e5f7c 98785c00
[  558.003698] 5f60: 98785c00 00000000 00e9a010 00e9a050 00000026 80108204 a01e4000 00000000
[  558.015188] 5f80: a01e5fa4 a01e5f90 8029bd9c 8029b8bc 00000000 00e9a050 00000000 a01e5fa8
[  558.026723] 5fa0: 80108060 8029bd74 00000000 00e9a010 7ea7981c 7ea7a81c ffffffff 00000000
[  558.038198] 5fc0: 00000000 00e9a010 00e9a050 00000026 004fd000 76f05ce8 7ea7981c 7ea7a81c
[  558.049811] 5fe0: 004fd198 7ea79814 0049cc9b 76cd7796 00010030 7ea7981c 00000000 00000000
[  558.061386] [<802a4da0>] (__d_lookup_rcu) from [<80295cc4>] (lookup_fast+0x4c/0x2c8)
[  558.072944] [<80295cc4>] (lookup_fast) from [<80298140>] (walk_component+0x3c/0x2dc)
[  558.084330] [<80298140>] (walk_component) from [<8029856c>] (link_path_walk+0x18c/0x4b0)
[  558.095582] [<8029856c>] (link_path_walk) from [<802988c4>] (path_parentat+0x34/0x6c)
[  558.106851] [<802988c4>] (path_parentat) from [<8029a4b4>] (filename_parentat+0x7c/0x104)
[  558.117923] [<8029a4b4>] (filename_parentat) from [<8029b984>] (SyS_renameat2+0xd4/0x48c)
[  558.129156] [<8029b984>] (SyS_renameat2) from [<8029bd9c>] (SyS_rename+0x34/0x3c)
[  558.140092] [<8029bd9c>] (SyS_rename) from [<80108060>] (ret_fast_syscall+0x0/0x28)
[  558.150966] Code: ea000002 e5944000 e3540000 0a000028 (e5141004) 
[  558.161756] ---[ end trace 124dc21b9499878a ]---
[  560.624738] Alignment trap: not handling instruction e1b04f9f at [<80566d48>]
[  560.635458] Unhandled fault: alignment exception (0x001) at 0x9ac47754
[  560.646194] pgd = acecc000
[  560.656802] [9ac47754] *pgd=1ac1141e(bad)
[  560.667391] Internal error: : 1 [#4] SMP ARM
[  560.677894] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  560.700191] CPU: 3 PID: 4194 Comm: PeripBusUSBUdev Tainted: G      D  C      4.14.29+ #1
[  560.711520] Hardware name: BCM2835
[  560.724080] task: 81945a00 task.stack: 9e712000
[  560.737866] PC is at lockref_put_return+0x3c/0x94
[  560.750567] LR is at dput+0x40/0x2d0
[  560.761780] pc : [<80566d4c>]    lr : [<802a1abc>]    psr: 20060013
[  560.773122] sp : 9e713de0  ip : 9e713e00  fp : 9e713dfc
[  560.784351] r10: 00000000  r9 : 9e712000  r8 : 9e713f60
[  560.795460] r7 : 00000001  r6 : 00010001  r5 : 00000001  r4 : 00010001
[  560.806753] r3 : 00000000  r2 : 00010001  r1 : 00000000  r0 : 9ac47754
[  560.817978] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  560.829007] Control: 10c5383d  Table: 2cecc06a  DAC: 00000055
[  560.839867] Process PeripBusUSBUdev (pid: 4194, stack limit = 0x9e712210)
[  560.851294] Stack: (0x9e713de0 to 0x9e714000)
[  560.863891] 3de0: 9ac47704 00080040 9ac47754 00000041 9e713e24 9e713e00 802a1abc 80566d1c
[  560.876801] 3e00: 00000000 9e713e78 9e713f60 00000041 9e713f60 9e712000 9e713e44 9e713e28
[  560.887838] 3e20: 80294b9c 802a1a88 9e713e78 fffffffe 9e713f60 00000041 9e713e74 9e713e48
[  560.900010] 3e40: 802989bc 80294b5c 9e6683f8 af57c6bc 00000000 00000000 00000001 8c6f5000
[  560.915033] 3e60: 9e713e78 00000001 9e713f24 9e713e78 8029a5d8 80298908 ae990310 9ac47704
[  560.929842] 3e80: e73d77c1 00000006 8c6f5025 00000000 ae990c10 ae7af660 ae7db2b0 00000001
[  560.942629] 3ea0: 9e713dd0 000003be 00000000 00000000 00000000 9e713eb8 00000000 00000000
[  560.955273] 3ec0: 00001000 8c6f6000 00000000 00000001 80d0459c 8c6f5000 80d0459c 6b2fea18
[  560.966619] 3ee0: 00000000 00000001 8c6f5000 00000000 8029a30c 00000002 ffffff9c 00000001
[  560.978523] 3f00: ffffff9c 00000001 ffffff9c 9e713f60 ffffff9c 6b2fea18 9e713f4c 9e713f28
[  560.992103] 3f20: 8029a720 8029a548 00000000 00000000 9e713f4c 00000001 00000000 acb3e980
[  561.005850] 3f40: 9e713f94 9e713f50 80287344 8029a6d8 00000000 00000000 aaa67500 00000000
[  561.017377] 3f60: 00000000 00000000 00000005 00000000 5c21ee28 76f78ce8 00000021 80108204
[  561.029029] 3f80: 9e712000 00000000 9e713fa4 9e713f98 802874c4 802872b4 00000000 9e713fa8
[  561.040579] 3fa0: 80108060 802874ac 00000000 5c21ee28 6b2fea18 00000000 00000000 6b2fea33
[  561.052969] 3fc0: 00000000 5c21ee28 76f78ce8 00000021 6b2fea18 6b2fea48 716d7b30 ffffffea
[  561.068944] 3fe0: 75eebf10 6b2fea04 75ee4b7b 759c1b66 20060030 6b2fea18 aa98abba faaaaaea
[  561.083256] [<80566d4c>] (lockref_put_return) from [<802a1abc>] (dput+0x40/0x2d0)
[  561.094752] [<802a1abc>] (dput) from [<80294b9c>] (terminate_walk+0x4c/0xc0)
[  561.106786] [<80294b9c>] (terminate_walk) from [<802989bc>] (path_lookupat+0xc0/0x204)
[  561.120543] [<802989bc>] (path_lookupat) from [<8029a5d8>] (filename_lookup+0x9c/0xf8)
[  561.133721] [<8029a5d8>] (filename_lookup) from [<8029a720>] (user_path_at_empty+0x54/0x5c)
[  561.144836] [<8029a720>] (user_path_at_empty) from [<80287344>] (SyS_faccessat+0x9c/0x1f8)
[  561.155787] [<80287344>] (SyS_faccessat) from [<802874c4>] (SyS_access+0x24/0x28)
[  561.166755] [<802874c4>] (SyS_access) from [<80108060>] (ret_fast_syscall+0x0/0x28)
[  561.177608] Code: e1a07005 da000015 f590f000 e1b04f9f (e1340006) 
[  561.188475] ---[ end trace 124dc21b9499878b ]---
[  627.422503] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
[  627.436018] pgd = 82af0000
[  627.449159] [0000000d] *pgd=00000000
[  627.460412] Internal error: Oops: 5 [#5] SMP ARM
[  627.470877] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  627.492654] CPU: 0 PID: 10887 Comm: sudo Tainted: G      D  C      4.14.29+ #1
[  627.503546] Hardware name: BCM2835
[  627.514368] task: 91ab3c00 task.stack: 8287a000
[  627.525158] PC is at locks_remove_posix+0x30/0x14c
[  627.535881] LR is at filp_close+0x68/0x8c
[  627.547086] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 20000013
[  627.557884] sp : 8287beb0  ip : 8287bf58  fp : 8287bf54
[  627.568690] r10: 00000000  r9 : 8287a000  r8 : 80108204
[  627.579439] r7 : 00000006  r6 : acc63300  r5 : 9d640180  r4 : ae690440
[  627.590265] r3 : 00000001  r2 : ace99fc0  r1 : acc63300  r0 : 9d640180
[  627.601025] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  627.611944] Control: 10c5383d  Table: 02af006a  DAC: 00000055
[  627.622831] Process sudo (pid: 10887, stack limit = 0x8287a210)
[  627.633779] Stack: (0x8287beb0 to 0x8287c000)
[  627.644595] bea0:                                     802fe824 91ab3c00 00000041 802a8f5c
[  627.655547] bec0: 00000004 00000000 00000000 00000100 81efc688 80d093bc 00000017 808ac294
[  627.666714] bee0: 76e3be2c 8287bfb0 7e94f750 808a8780 8287bf14 8287bf00 808a8780 806fbae8
[  627.677972] bf00: add51c00 add51d74 8287bf34 8287bf18 806fbae8 808a7758 add52400 7f0220bc
[  627.689092] bf20: add52400 add52444 8287bf54 00000000 9d640180 acc63300 00000006 80108204
[  627.699934] bf40: 8287a000 00000000 8287bf74 8287bf58 80286ac4 802e088c 0000003c acc63300
[  627.710825] bf60: 9d640180 00000006 8287bf94 8287bf78 802aad38 80286a68 76f39218 0000000b
[  627.721610] bf80: 00000000 00000006 8287bfa4 8287bf98 80286b18 802aac7c 00000000 8287bfa8
[  627.732532] bfa0: 80108060 80286af4 76f39218 0000000b 0000003c 7e94f750 76f38b5c 76f38b5c
[  627.743512] bfc0: 76f39218 0000000b 00000000 00000006 022b85d0 7e94f750 76f76ce8 0000003c
[  627.754665] bfe0: 7e94f728 7e94f718 76f38b6c 76e10fc2 60000030 0000003c 00000000 00000000
[  627.765898] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
[  627.777075] [<80286ac4>] (filp_close) from [<802aad38>] (__close_fd+0xc8/0xec)
[  627.788388] [<802aad38>] (__close_fd) from [<80286b18>] (SyS_close+0x30/0x58)
[  627.799587] [<80286b18>] (SyS_close) from [<80108060>] (ret_fast_syscall+0x0/0x28)
[  627.810901] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
[  627.823021] ---[ end trace 124dc21b9499878c ]---
[  627.851770] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
[  627.863130] pgd = 80004000
[  627.874478] [0000000d] *pgd=00000000
[  627.885738] Internal error: Oops: 5 [#6] SMP ARM
[  627.896909] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  627.920757] CPU: 3 PID: 10879 Comm: sudo Tainted: G      D  C      4.14.29+ #1
[  627.932515] Hardware name: BCM2835
[  627.944207] task: 91ab6900 task.stack: aeb86000
[  627.955689] PC is at locks_remove_posix+0x30/0x14c
[  627.967555] LR is at filp_close+0x68/0x8c
[  627.979366] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 28070013
[  627.994526] sp : aeb87d08  ip : aeb87db0  fp : aeb87dac
[  628.006388] r10: 0000000b  r9 : 8d5bba00  r8 : 00000004
[  628.017356] r7 : acc63a00  r6 : acc63a00  r5 : 9d640180  r4 : ae690440
[  628.028340] r3 : 00000001  r2 : 9d640180  r1 : acc63a00  r0 : 9d640180
[  628.039180] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  628.050082] Control: 10c5383d  Table: 2cecc06a  DAC: 00000055
[  628.060845] Process sudo (pid: 10879, stack limit = 0xaeb86210)
[  628.071621] Stack: (0xaeb87d08 to 0xaeb88000)
[  628.082270] 7d00:                   802777a0 801edc50 00000000 80d87c00 ac80efc0 af2bfff0
[  628.093150] 7d20: 8012000e 00000001 ac80efc0 00000001 48070013 000b21eb aeb87d84 8012000e
[  628.104069] 7d40: 80224d2c 808a8780 aeb87d6c aeb87d58 808a8780 806fbae8 add51c00 add51d74
[  628.115133] 7d60: aeb87d8c aeb87d70 806fbae8 808a7758 add52400 7f0220bc add52400 add52444
[  628.126354] 7d80: aeb87dac 00000000 9d640180 acc63a00 acc63a00 00000004 8d5bba00 0000000b
[  628.137591] 7da0: aeb87dcc aeb87db0 80286ac4 802e088c 00000009 000000f0 00000000 acc63a00
[  628.149003] 7dc0: aeb87df4 aeb87dd0 802aa864 80286a68 91ab6900 91ab6e40 acc63a00 ac80efc0
[  628.160356] 7de0: 00000001 ac80eff8 aeb87e14 aeb87df8 802aa974 802aa7bc 91ab6900 00000000
[  628.171934] 7e00: 00000544 ac80efc0 aeb87e54 aeb87e18 80121d6c 802aa928 aeb87e74 aeb87e28
[  628.183447] 7e20: aeb87edc aeb87fb0 00000000 0000000b 82e42100 aeb87edc aeb86000 00106001
[  628.194987] 7e40: aeb86000 0000000b aeb87e74 aeb87e58 8012260c 801219e0 00000000 8bf6da08
[  628.206554] 7e60: aeb87edc aeb86000 aeb87ec4 aeb87e78 8012d9f4 801225cc 80d02040 418004fc
[  628.218157] 7e80: 80d03d68 aeb87ec8 82e42100 8bf6de44 8bf6d940 000000a0 0000000b 76da3574
[  628.230003] 7ea0: aeb87ec8 aeb87fb0 76da3576 00000000 aeb86000 00000000 aeb87f8c aeb87ec8
[  628.241510] 7ec0: 8010b318 8012d6c8 aeb87ef4 aeb87ed8 8012d084 8012cf08 00000000 0000000b
[  628.252971] 7ee0: 00000000 00000000 00002a7f 00000000 8bf6d968 00000001 91ab6900 00000000
[  628.264251] 7f00: 00000000 00002a7f 00000000 8bf6d968 00000001 91ab6900 00000001 aeb87f68
[  628.275572] 7f20: aeb87f64 aeb87f30 8012f17c 801edc50 91ab6e44 00000002 7e94fa98 00000000
[  628.287125] 7f40: 00000002 7e94fa98 000000ae 80108204 aeb86000 00000000 aeb87fa4 aeb87f68
[  628.298685] 7f60: 8012f6f0 00000001 aeb86010 80108204 aeb87fb0 80108204 aeb86000 00000000
[  628.310431] 7f80: aeb87fac aeb87f90 8010b820 8010b260 7e94fa98 0000000b 0000000b 00000025
[  628.322157] 7fa0: 00000000 aeb87fb0 80108094 8010b774 00000000 0000000b 453b0700 453b0700
[  628.334017] 7fc0: 7e94fa98 0000000b 0000000b 00000025 022b5c58 004e516c 00000000 7e94fa08
[  628.345896] 7fe0: 004f6dd8 7e94f9b4 004d2d5b 76da3576 20040030 00002a7f 000001b8 00000000
[  628.357879] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
[  628.369957] [<80286ac4>] (filp_close) from [<802aa864>] (put_files_struct+0xb4/0x10c)
[  628.381868] [<802aa864>] (put_files_struct) from [<802aa974>] (exit_files+0x58/0x5c)
[  628.393828] [<802aa974>] (exit_files) from [<80121d6c>] (do_exit+0x398/0xba0)
[  628.405659] [<80121d6c>] (do_exit) from [<8012260c>] (do_group_exit+0x4c/0xe4)
[  628.417476] [<8012260c>] (do_group_exit) from [<8012d9f4>] (get_signal+0x338/0x6e0)
[  628.429281] [<8012d9f4>] (get_signal) from [<8010b318>] (do_signal+0xc4/0x3e4)
[  628.440899] [<8010b318>] (do_signal) from [<8010b820>] (do_work_pending+0xb8/0xd0)
[  628.452491] [<8010b820>] (do_work_pending) from [<80108094>] (slow_work_pending+0xc/0x20)
[  628.463887] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
[  628.475205] ---[ end trace 124dc21b9499878d ]---
[  628.488855] Fixing recursive fault but reboot is needed!

@pelwell
Copy link
Contributor

pelwell commented Mar 22, 2018

If you have vcgencmd, please report the output of vcgencmd get_throttled.

@Ferroin
Copy link
Contributor

Ferroin commented Mar 22, 2018

Hmm, that's interesting...

All of the stack traces in those kernel OOPS messages implicate absolutely nothing from the networking stack, they've just got generic core kernel stuff and VFS layer functions referenced.

The complaint about a kernel NULL pointer dereference at a very low virtual address is also somewhat suspicious. Stuff like that is usually indicative of either hardware failure, or a very poorly behaved kernel driver scribbling on memory locations it shouldn't be touching.

@mkreisl
Copy link

mkreisl commented Mar 22, 2018

All of the stack traces in those kernel OOPS messages implicate absolutely nothing from the networking stack, they've just got generic core kernel stuff and VFS layer functions referenced.

I know, could it be that corrupted data read from usb disk caused this?

@mkreisl
Copy link

mkreisl commented Mar 22, 2018

'Ported' the backup script to Raspbian, an run it there to backup one of my btrfs partition. Similar result, did not finished. btrfs send/receive stucks, nothing in logs. Tested with original 4.14.27-v7+ kernel and my 4.15.10+ kernel, and of course, backup was running successful on Pi3B

@pelwell

vcgencmd get_throttled
throttled=0x0

@pelwell
Copy link
Contributor

pelwell commented Mar 22, 2018

Thanks - that rules out a power supply issue.

@mkreisl
Copy link

mkreisl commented Mar 22, 2018

Thanks - that rules out a power supply issue.

Already tried different PSU's, usually using brand new 2.5A PSU for Pi3B+ and an older one (2A) for Pi3B

@pelwell
Copy link
Contributor

pelwell commented Mar 22, 2018

Yes, so you said, but I'd rather hear it from the Pi.

@mkreisl
Copy link

mkreisl commented Mar 22, 2018

@pelwell
Are there any tests I can run to make absolutely sure that hardware is ok?

Btw, I have similar issues if root fs is on iSCSI target and no usb drive is connected and used.

@mkreisl
Copy link

mkreisl commented Mar 22, 2018

@pelwell @Ferroin
Oh, and before I forget:
If I'm using onboard WLAN instead of onboard ethernet, issues are still there
and, already changed network cable and port on the network switch
and, switching root to sd-card does not make any difference, getting this on my final test for today:

[  515.046170] CIFS VFS: sends on sock a715d500 stuck for 15 seconds
[  515.046196] CIFS VFS: Error -11 sending data on socket to server
[  517.443871] systemd-logind[3010]: New session c2 of user root.
[  530.166192] CIFS VFS: sends on sock a715d500 stuck for 15 seconds
[  530.166217] CIFS VFS: Error -11 sending data on socket to server
[  540.648102] Status code returned 0xc0000008 NT_STATUS_INVALID_HANDLE
[  540.648134] CIFS VFS: Send error in read = -9
[  540.648161] print_req_error: I/O error, dev loop0, sector 231328
[  540.648195] BTRFS error (device dm-1): bdev /dev/mapper/loop0p2 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
[  735.206538] INFO: task kworker/u8:2:219 blocked for more than 120 seconds.
[  735.206553]       Tainted: G         C      4.14.29+ #1
[  735.206558] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  735.206567] kworker/u8:2    D    0   219      2 0x00000000
[  735.206596] Workqueue: writeback wb_workfn (flush-btrfs-3)
[  735.206633] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  735.206652] [<808a6544>] (schedule) from [<80476418>] (btrfs_tree_lock+0x15c/0x278)
[  735.206672] [<80476418>] (btrfs_tree_lock) from [<804578e0>] (lock_extent_buffer_for_io+0xec/0x270)
[  735.206694] [<804578e0>] (lock_extent_buffer_for_io) from [<8045b160>] (btree_write_cache_pages+0x248/0x344)
[  735.206716] [<8045b160>] (btree_write_cache_pages) from [<80422eb0>] (btree_writepages+0x84/0x8c)
[  735.206734] [<80422eb0>] (btree_writepages) from [<8022edb4>] (do_writepages+0x30/0x8c)
[  735.206750] [<8022edb4>] (do_writepages) from [<802bb208>] (__writeback_single_inode+0x44/0x434)
[  735.206766] [<802bb208>] (__writeback_single_inode) from [<802bbb08>] (writeback_sb_inodes+0x214/0x4c4)
[  735.206781] [<802bbb08>] (writeback_sb_inodes) from [<802bbe48>] (__writeback_inodes_wb+0x90/0xd0)
[  735.206797] [<802bbe48>] (__writeback_inodes_wb) from [<802bc120>] (wb_writeback+0x298/0x33c)
[  735.206811] [<802bc120>] (wb_writeback) from [<802bc9b8>] (wb_workfn+0xdc/0x4d8)
[  735.206831] [<802bc9b8>] (wb_workfn) from [<801374b8>] (process_one_work+0x158/0x454)
[  735.206850] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  735.206868] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  735.206886] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  735.206920] INFO: task kworker/u8:3:399 blocked for more than 120 seconds.
[  735.206926]       Tainted: G         C      4.14.29+ #1
[  735.206931] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  735.206936] kworker/u8:3    D    0   399      2 0x00000000
[  735.206954] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[  735.206972] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  735.206988] [<808a6544>] (schedule) from [<808aa12c>] (schedule_timeout+0x1d0/0x3e4)
[  735.207005] [<808aa12c>] (schedule_timeout) from [<808a71cc>] (wait_for_common+0xc0/0x184)
[  735.207021] [<808a71cc>] (wait_for_common) from [<808a72b0>] (wait_for_completion+0x20/0x24)
[  735.207036] [<808a72b0>] (wait_for_completion) from [<8040c478>] (btrfs_async_run_delayed_refs+0x12c/0x14c)
[  735.207053] [<8040c478>] (btrfs_async_run_delayed_refs) from [<8042d7ec>] (__btrfs_end_transaction+0x228/0x324)
[  735.207069] [<8042d7ec>] (__btrfs_end_transaction) from [<8042d904>] (btrfs_end_transaction+0x1c/0x20)
[  735.207085] [<8042d904>] (btrfs_end_transaction) from [<80438b40>] (btrfs_finish_ordered_io+0x230/0x848)
[  735.207102] [<80438b40>] (btrfs_finish_ordered_io) from [<8043955c>] (finish_ordered_fn+0x1c/0x20)
[  735.207118] [<8043955c>] (finish_ordered_fn) from [<8046a1d8>] (normal_work_helper+0xb0/0x3e0)
[  735.207135] [<8046a1d8>] (normal_work_helper) from [<8046a8f0>] (btrfs_endio_write_helper+0x1c/0x20)
[  735.207152] [<8046a8f0>] (btrfs_endio_write_helper) from [<801374b8>] (process_one_work+0x158/0x454)
[  735.207170] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  735.207187] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  735.207203] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  735.207217] INFO: task kworker/u8:6:759 blocked for more than 120 seconds.
[  735.207224]       Tainted: G         C      4.14.29+ #1
[  735.207229] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  735.207233] kworker/u8:6    D    0   759      2 0x00000000
[  735.207250] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper
[  735.207269] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  735.207284] [<808a6544>] (schedule) from [<80475e2c>] (btrfs_tree_read_lock+0x120/0x198)
[  735.207303] [<80475e2c>] (btrfs_tree_read_lock) from [<803f9660>] (btrfs_read_lock_root_node+0x38/0x50)
[  735.207321] [<803f9660>] (btrfs_read_lock_root_node) from [<803ff188>] (btrfs_search_slot+0x8b0/0xb10)
[  735.207339] [<803ff188>] (btrfs_search_slot) from [<80400f10>] (btrfs_insert_empty_items+0x7c/0xd4)
[  735.207357] [<80400f10>] (btrfs_insert_empty_items) from [<8040f01c>] (__btrfs_run_delayed_refs+0xe90/0x16fc)
[  735.207374] [<8040f01c>] (__btrfs_run_delayed_refs) from [<80412d9c>] (btrfs_run_delayed_refs+0x9c/0x310)
[  735.207390] [<80412d9c>] (btrfs_run_delayed_refs) from [<804130c4>] (delayed_ref_async_start+0xb4/0xc0)
[  735.207405] [<804130c4>] (delayed_ref_async_start) from [<8046a1d8>] (normal_work_helper+0xb0/0x3e0)
[  735.207419] [<8046a1d8>] (normal_work_helper) from [<8046a990>] (btrfs_extent_refs_helper+0x1c/0x20)
[  735.207437] [<8046a990>] (btrfs_extent_refs_helper) from [<801374b8>] (process_one_work+0x158/0x454)
[  735.207455] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  735.207471] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  735.207487] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  735.207625] INFO: task btrfs-transacti:8152 blocked for more than 120 seconds.
[  735.207632]       Tainted: G         C      4.14.29+ #1
[  735.207637] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  735.207641] btrfs-transacti D    0  8152      2 0x00000000
[  735.207663] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  735.207678] [<808a6544>] (schedule) from [<8014b128>] (io_schedule+0x20/0x40)
[  735.207693] [<8014b128>] (io_schedule) from [<8021cd60>] (wait_on_page_bit+0x120/0x140)
[  735.207711] [<8021cd60>] (wait_on_page_bit) from [<80459a00>] (read_extent_buffer_pages+0x258/0x33c)
[  735.207731] [<80459a00>] (read_extent_buffer_pages) from [<80422350>] (btree_read_extent_buffer_pages+0xb0/0x120)
[  735.207747] [<80422350>] (btree_read_extent_buffer_pages) from [<80423818>] (read_tree_block+0x38/0x54)
[  735.207763] [<80423818>] (read_tree_block) from [<803f7914>] (read_node_slot+0xc4/0x100)
[  735.207779] [<803f7914>] (read_node_slot) from [<803fd508>] (push_leaf_right+0xb8/0x1f0)
[  735.207796] [<803fd508>] (push_leaf_right) from [<803fe6c8>] (split_leaf+0x5f4/0x804)
[  735.207813] [<803fe6c8>] (split_leaf) from [<803ff270>] (btrfs_search_slot+0x998/0xb10)
[  735.207830] [<803ff270>] (btrfs_search_slot) from [<80400f10>] (btrfs_insert_empty_items+0x7c/0xd4)
[  735.207847] [<80400f10>] (btrfs_insert_empty_items) from [<8040f01c>] (__btrfs_run_delayed_refs+0xe90/0x16fc)
[  735.207863] [<8040f01c>] (__btrfs_run_delayed_refs) from [<80412d9c>] (btrfs_run_delayed_refs+0x9c/0x310)
[  735.207879] [<80412d9c>] (btrfs_run_delayed_refs) from [<8042c378>] (btrfs_commit_transaction+0x38/0xc2c)
[  735.207893] [<8042c378>] (btrfs_commit_transaction) from [<804274bc>] (transaction_kthread+0x1b4/0x1c8)
[  735.207908] [<804274bc>] (transaction_kthread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  735.207923] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  735.207947] INFO: task btrfs:9851 blocked for more than 120 seconds.
[  735.207953]       Tainted: G         C      4.14.29+ #1
[  735.207958] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  735.207963] btrfs           D    0  9851   9850 0x00000000
[  735.207984] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  735.207999] [<808a6544>] (schedule) from [<8044fdec>] (btrfs_start_ordered_extent+0x128/0x158)
[  735.208016] [<8044fdec>] (btrfs_start_ordered_extent) from [<80450394>] (btrfs_wait_ordered_range+0x140/0x18c)
[  735.208032] [<80450394>] (btrfs_wait_ordered_range) from [<8043b4f0>] (btrfs_truncate+0x54/0x2c8)
[  735.208048] [<8043b4f0>] (btrfs_truncate) from [<8043c074>] (btrfs_setattr+0x2f4/0x488)
[  735.208067] [<8043c074>] (btrfs_setattr) from [<802a9318>] (notify_change+0x1cc/0x408)
[  735.208084] [<802a9318>] (notify_change) from [<80286c98>] (do_truncate+0x90/0xc0)
[  735.208099] [<80286c98>] (do_truncate) from [<80286eac>] (vfs_truncate+0x1e4/0x25c)
[  735.208113] [<80286eac>] (vfs_truncate) from [<80286fa0>] (do_sys_truncate+0x7c/0xac)
[  735.208127] [<80286fa0>] (do_sys_truncate) from [<80287200>] (SyS_truncate64+0x18/0x1c)
[  735.208143] [<80287200>] (SyS_truncate64) from [<80108060>] (ret_fast_syscall+0x0/0x28)
[  858.087442] INFO: task kworker/u8:2:219 blocked for more than 120 seconds.
[  858.087457]       Tainted: G         C      4.14.29+ #1
[  858.087462] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.087469] kworker/u8:2    D    0   219      2 0x00000000
[  858.087503] Workqueue: writeback wb_workfn (flush-btrfs-3)
[  858.087538] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  858.087557] [<808a6544>] (schedule) from [<80476418>] (btrfs_tree_lock+0x15c/0x278)
[  858.087579] [<80476418>] (btrfs_tree_lock) from [<804578e0>] (lock_extent_buffer_for_io+0xec/0x270)
[  858.087601] [<804578e0>] (lock_extent_buffer_for_io) from [<8045b160>] (btree_write_cache_pages+0x248/0x344)
[  858.087624] [<8045b160>] (btree_write_cache_pages) from [<80422eb0>] (btree_writepages+0x84/0x8c)
[  858.087642] [<80422eb0>] (btree_writepages) from [<8022edb4>] (do_writepages+0x30/0x8c)
[  858.087659] [<8022edb4>] (do_writepages) from [<802bb208>] (__writeback_single_inode+0x44/0x434)
[  858.087675] [<802bb208>] (__writeback_single_inode) from [<802bbb08>] (writeback_sb_inodes+0x214/0x4c4)
[  858.087690] [<802bbb08>] (writeback_sb_inodes) from [<802bbe48>] (__writeback_inodes_wb+0x90/0xd0)
[  858.087705] [<802bbe48>] (__writeback_inodes_wb) from [<802bc120>] (wb_writeback+0x298/0x33c)
[  858.087719] [<802bc120>] (wb_writeback) from [<802bc9b8>] (wb_workfn+0xdc/0x4d8)
[  858.087739] [<802bc9b8>] (wb_workfn) from [<801374b8>] (process_one_work+0x158/0x454)
[  858.087758] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  858.087775] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  858.087793] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  858.087827] INFO: task kworker/u8:3:399 blocked for more than 120 seconds.
[  858.087833]       Tainted: G         C      4.14.29+ #1
[  858.087838] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.087843] kworker/u8:3    D    0   399      2 0x00000000
[  858.087861] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[  858.087879] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  858.087895] [<808a6544>] (schedule) from [<808aa12c>] (schedule_timeout+0x1d0/0x3e4)
[  858.087912] [<808aa12c>] (schedule_timeout) from [<808a71cc>] (wait_for_common+0xc0/0x184)
[  858.087929] [<808a71cc>] (wait_for_common) from [<808a72b0>] (wait_for_completion+0x20/0x24)
[  858.087944] [<808a72b0>] (wait_for_completion) from [<8040c478>] (btrfs_async_run_delayed_refs+0x12c/0x14c)
[  858.087961] [<8040c478>] (btrfs_async_run_delayed_refs) from [<8042d7ec>] (__btrfs_end_transaction+0x228/0x324)
[  858.087976] [<8042d7ec>] (__btrfs_end_transaction) from [<8042d904>] (btrfs_end_transaction+0x1c/0x20)
[  858.087994] [<8042d904>] (btrfs_end_transaction) from [<80438b40>] (btrfs_finish_ordered_io+0x230/0x848)
[  858.088011] [<80438b40>] (btrfs_finish_ordered_io) from [<8043955c>] (finish_ordered_fn+0x1c/0x20)
[  858.088027] [<8043955c>] (finish_ordered_fn) from [<8046a1d8>] (normal_work_helper+0xb0/0x3e0)
[  858.088042] [<8046a1d8>] (normal_work_helper) from [<8046a8f0>] (btrfs_endio_write_helper+0x1c/0x20)
[  858.088060] [<8046a8f0>] (btrfs_endio_write_helper) from [<801374b8>] (process_one_work+0x158/0x454)
[  858.088077] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  858.088094] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  858.088109] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  858.088124] INFO: task kworker/u8:6:759 blocked for more than 120 seconds.
[  858.088130]       Tainted: G         C      4.14.29+ #1
[  858.088135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.088140] kworker/u8:6    D    0   759      2 0x00000000
[  858.088157] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper
[  858.088176] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  858.088191] [<808a6544>] (schedule) from [<80475e2c>] (btrfs_tree_read_lock+0x120/0x198)
[  858.088210] [<80475e2c>] (btrfs_tree_read_lock) from [<803f9660>] (btrfs_read_lock_root_node+0x38/0x50)
[  858.088228] [<803f9660>] (btrfs_read_lock_root_node) from [<803ff188>] (btrfs_search_slot+0x8b0/0xb10)
[  858.088246] [<803ff188>] (btrfs_search_slot) from [<80400f10>] (btrfs_insert_empty_items+0x7c/0xd4)
[  858.088264] [<80400f10>] (btrfs_insert_empty_items) from [<8040f01c>] (__btrfs_run_delayed_refs+0xe90/0x16fc)
[  858.088281] [<8040f01c>] (__btrfs_run_delayed_refs) from [<80412d9c>] (btrfs_run_delayed_refs+0x9c/0x310)
[  858.088297] [<80412d9c>] (btrfs_run_delayed_refs) from [<804130c4>] (delayed_ref_async_start+0xb4/0xc0)
[  858.088312] [<804130c4>] (delayed_ref_async_start) from [<8046a1d8>] (normal_work_helper+0xb0/0x3e0)
[  858.088326] [<8046a1d8>] (normal_work_helper) from [<8046a990>] (btrfs_extent_refs_helper+0x1c/0x20)
[  858.088344] [<8046a990>] (btrfs_extent_refs_helper) from [<801374b8>] (process_one_work+0x158/0x454)
[  858.088362] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  858.088378] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  858.088394] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  858.088532] INFO: task btrfs-transacti:8152 blocked for more than 120 seconds.
[  858.088539]       Tainted: G         C      4.14.29+ #1
[  858.088544] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.088548] btrfs-transacti D    0  8152      2 0x00000000
[  858.088570] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  858.088585] [<808a6544>] (schedule) from [<8014b128>] (io_schedule+0x20/0x40)
[  858.088600] [<8014b128>] (io_schedule) from [<8021cd60>] (wait_on_page_bit+0x120/0x140)
[  858.088617] [<8021cd60>] (wait_on_page_bit) from [<80459a00>] (read_extent_buffer_pages+0x258/0x33c)
[  858.088637] [<80459a00>] (read_extent_buffer_pages) from [<80422350>] (btree_read_extent_buffer_pages+0xb0/0x120)
[  858.088653] [<80422350>] (btree_read_extent_buffer_pages) from [<80423818>] (read_tree_block+0x38/0x54)
[  858.088669] [<80423818>] (read_tree_block) from [<803f7914>] (read_node_slot+0xc4/0x100)
[  858.088686] [<803f7914>] (read_node_slot) from [<803fd508>] (push_leaf_right+0xb8/0x1f0)
[  858.088703] [<803fd508>] (push_leaf_right) from [<803fe6c8>] (split_leaf+0x5f4/0x804)
[  858.088720] [<803fe6c8>] (split_leaf) from [<803ff270>] (btrfs_search_slot+0x998/0xb10)
[  858.088737] [<803ff270>] (btrfs_search_slot) from [<80400f10>] (btrfs_insert_empty_items+0x7c/0xd4)
[  858.088754] [<80400f10>] (btrfs_insert_empty_items) from [<8040f01c>] (__btrfs_run_delayed_refs+0xe90/0x16fc)
[  858.088770] [<8040f01c>] (__btrfs_run_delayed_refs) from [<80412d9c>] (btrfs_run_delayed_refs+0x9c/0x310)
[  858.088786] [<80412d9c>] (btrfs_run_delayed_refs) from [<8042c378>] (btrfs_commit_transaction+0x38/0xc2c)
[  858.088800] [<8042c378>] (btrfs_commit_transaction) from [<804274bc>] (transaction_kthread+0x1b4/0x1c8)
[  858.088814] [<804274bc>] (transaction_kthread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  858.088830] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  858.088854] INFO: task btrfs:9851 blocked for more than 120 seconds.
[  858.088860]       Tainted: G         C      4.14.29+ #1
[  858.088865] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.088869] btrfs           D    0  9851   9850 0x00000000
[  858.088891] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  858.088906] [<808a6544>] (schedule) from [<8044fdec>] (btrfs_start_ordered_extent+0x128/0x158)
[  858.088924] [<8044fdec>] (btrfs_start_ordered_extent) from [<80450394>] (btrfs_wait_ordered_range+0x140/0x18c)
[  858.088941] [<80450394>] (btrfs_wait_ordered_range) from [<8043b4f0>] (btrfs_truncate+0x54/0x2c8)
[  858.088957] [<8043b4f0>] (btrfs_truncate) from [<8043c074>] (btrfs_setattr+0x2f4/0x488)
[  858.088975] [<8043c074>] (btrfs_setattr) from [<802a9318>] (notify_change+0x1cc/0x408)
[  858.088993] [<802a9318>] (notify_change) from [<80286c98>] (do_truncate+0x90/0xc0)
[  858.089008] [<80286c98>] (do_truncate) from [<80286eac>] (vfs_truncate+0x1e4/0x25c)
[  858.089022] [<80286eac>] (vfs_truncate) from [<80286fa0>] (do_sys_truncate+0x7c/0xac)
[  858.089036] [<80286fa0>] (do_sys_truncate) from [<80287200>] (SyS_truncate64+0x18/0x1c)
[  858.089052] [<80287200>] (SyS_truncate64) from [<80108060>] (ret_fast_syscall+0x0/0x28)

Btw, system (root partition) was cloned from usb partition to sd-card using same backup procedure and ran without any issues.

@JamesH65
Copy link
Contributor

Have you got another Pi3B+ you can try this on. It really is very bizarre, and bizarre always makes me think HW fault.

@mkreisl
Copy link

mkreisl commented Mar 22, 2018

Have you got another Pi3B+ you can try this on. It really is very bizarre, and bizarre always makes me think HW fault.

No, unfortunately not

@pelwell
Copy link
Contributor

pelwell commented Mar 22, 2018

It doesn't feel like a hardware fault to me - if so, you would expect to see it on a stock Raspbian kernel as well, and so far I don't think we have.

@mkreisl
Copy link

mkreisl commented Mar 22, 2018

if so, you would expect to see it on a stock Raspbian kernel as well

Already posted. It happens on Raspbian using standard kernel (4.14.27+) as well

@pelwell
Copy link
Contributor

pelwell commented Mar 23, 2018

Yes, reading your earlier comment again I can see how it means that.

Can you try limiting the ARM cores to 1.2GHz by adding the following to config.txt?:

arm_freq=1200

@mkreisl
Copy link

mkreisl commented Mar 23, 2018

@pelwell
I'll try this. Currently running another test to a different machine, with very strange result

getting periodically message like this:

[ 1416.176886] nfs: server kmxbmc not responding, still trying
[ 1417.217789] nfs: server kmxbmc not responding, still trying
[ 1417.575681] nfs: server kmxbmc OK
[ 1417.588839] nfs: server kmxbmc OK

The interval is exactly 239s. Any idea where this value comes from. NFS share is mounted as follows

kmxbmc:/srv on /mnt type nfs (rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.5,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=192.168.1.5)

@mkreisl
Copy link

mkreisl commented Mar 23, 2018

@pelwell
Doesn't help 😞

@mkreisl
Copy link

mkreisl commented Mar 23, 2018

... and it does not depend on kernel version. 4.9.xx producing same issues

popcornmix pushed a commit that referenced this issue Mar 27, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Mar 27, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Mar 27, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Apr 3, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Apr 5, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Apr 5, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Apr 8, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Apr 11, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Apr 16, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Apr 16, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Apr 18, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Apr 23, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Apr 29, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Apr 29, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
jai-raptee pushed a commit to jai-raptee/iliteck1 that referenced this issue Apr 30, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

raspberrypi/linux#2449
raspberrypi/linux#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
jai-raptee pushed a commit to jai-raptee/iliteck1 that referenced this issue Apr 30, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

raspberrypi/linux#2449
raspberrypi/linux#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
jai-raptee pushed a commit to jai-raptee/iliteck1 that referenced this issue Apr 30, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

raspberrypi/linux#2449
raspberrypi/linux#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
jai-raptee pushed a commit to jai-raptee/iliteck1 that referenced this issue Apr 30, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

raspberrypi/linux#2449
raspberrypi/linux#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
jai-raptee pushed a commit to jai-raptee/iliteck1 that referenced this issue Apr 30, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

raspberrypi/linux#2449
raspberrypi/linux#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
jai-raptee pushed a commit to jai-raptee/iliteck1 that referenced this issue Apr 30, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

raspberrypi/linux#2449
raspberrypi/linux#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
jai-raptee pushed a commit to jai-raptee/iliteck1 that referenced this issue Apr 30, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

raspberrypi/linux#2449
raspberrypi/linux#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue May 2, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue May 13, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue May 20, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue May 20, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue May 28, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue May 28, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Jun 3, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Jun 3, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
popcornmix pushed a commit that referenced this issue Jun 3, 2024
TSO seems to be having issues when packets are dropped and the
remote end uses Selective Acknowledge (SACK) to denote that
data is missing. The missing data is never resent, so the
connection eventually stalls.

There is a module parameter of enable_tso added to allow
further debugging without forcing a rebuild of the kernel.

#2449
#2482

Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests