[BUG] moosefs client: Wasting of cpu time

[shenziro]

Have you read through available documentation, open Github issues and Github Q&A Discussions?

Yes

System information

Your moosefs version and its origin (moosefs.com, packaged by distro, built from source, ...).

latest version from official repo

# dpkg-query --status moosefs-client
Package: moosefs-client
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 2476
Maintainer: MooseFS Team <contact@moosefs.com>
Architecture: amd64
Source: moosefs
Version: 3.0.116-1

Operating system (distribution) and kernel version.

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.6 LTS
Release:        18.04
Codename:       bionic
# uname -srvm
Linux 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64

Hardware / network configuration, and underlying filesystems on master, chunkservers, and clients.

Virtual machines (KVM on Proxmox VE) on recent Intel platform (12th gen).
Chunkservers with ext4 filesystem.

How much data is tracked by moosefs master (order of magnitude)?

• All fs objects: 230628
• Total space: 25 TiB
• Free space: 13 TiB
• RAM used: 540 MiB on mfsmaster
• last metadata save duration: ~0.2s

Describe the problem you observed.

The mfsmount process is wasting (?) cpu time with dozens of calls to nanosleep.

Can you reproduce it? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Reproducible on another Ubuntu 18.04 server on all mfsmount instances. (continuous load of 2-3% cpu per mountpoint with no access to the moosefs-filesystem at this moment.)

No (known) special configuration / mounting with the following options: mfsmaster=<ipaddr>,mfssubfolder=<foldername>

Include any warning/errors/backtraces from the system logs.

Output of strace (~0.5 seconds) for one thread of the process, that is using abount 2-3% cpu.

# time strace -p 9780
strace: Process 9780 attached
restart_syscall(<... resuming interrupted nanosleep ...>) = 0
nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7fa98641fea0) = 0
[... 102 identical lines ...]
nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7fa98641fea0) = 0
read(6, "\0\0\0\0\0\0\0\0", 8)          = 8
write(6, "\0\0\0\0\0\0\0\0", 8)         = 8
nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7fa98641fea0) = 0
[... 76 identical lines ...]
nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7fa98641fea0) = 0
nanosleep({tv_sec=0, tv_nsec=2500000}, ^Cstrace: Process 9780 detached
 <detached ...>

real    0m0.481s
user    0m0.004s
sys     0m0.005s

[chogata]

MooseFS client is a big, complicated process, in which many threads perform many different tasks. Some of those tasks need to be performed periodically, in order of the system to run smoothly, efficiently and correctly (most notably all the tasks that deal with various caches and their refreshing, invalidating etc.). What you see in strace - two or more nanosleep calls one after the other - is not what is happening in the process, there is a lot of work done in between those 2 calls, but since this work does not include any system calls, what you see with strace seems to be "just" the sleeps.