Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bad value display for "iostat -xdm", sysstat version:sysstat-12.5.5 #312

Open
huiwenshu opened this issue Feb 17, 2022 · 3 comments
Open

Comments

@huiwenshu
Copy link

bad value display for "iostat -xdm", sysstat version:sysstat-12.5.5
When I use iostat -xdm 2 (or user sar -dm 2)to monitor. It always display bad value for me. Besides the bad value for "r/s" is a same value :9223372036854775808.00

[18:01:12]Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
[18:01:12]dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[18:01:12]sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.01 0.00 0.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.10
[18:01:12]sde 0.50 0.01 0.00 0.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05
[18:01:12]sdf 9223372036854775808.00 4503599627370496.00 0.00 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 214748364.70
[18:01:12]sdg 1.00 0.02 0.00 0.00 0.50 16.25 0.50 0.00 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05
[18:01:12]sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[18:01:12]sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[18:01:12]sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[18:01:12]sdk 0.50 0.00 0.00 0.00 1.00 0.50 0.50 0.00 0.00 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.15
[18:01:12]sdl 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[18:01:12]sdm 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[18:01:12]sdn 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.01 0.00 0.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05
[18:01:12]sdo 1.00 0.01 0.00 0.00 0.50 8.25 0.50 0.00 0.00 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.15
[18:01:12]sdp 0.50 0.01 0.00 0.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05
[18:01:12]sdq 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[18:01:12]sdr 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[18:01:12]sds 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[18:01:12]sdt 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[18:01:12]sdu 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[18:01:12]sdv 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

@sysstat
Copy link
Owner

sysstat commented Feb 18, 2022

It seems that there is a kernel counter which is decreasing with time although this shouldn't be possible.
To verify this, you can create a datafile with sar (e.g. sar -o datafile -d 2 ; stop it with ^C once the problem appeared) and send me the resulting datafile so that I can inspect its contents.

@huiwenshu
Copy link
Author

sar.txt this is dump file,can you help me analyze the file ,thank you!

@sysstat
Copy link
Owner

sysstat commented Feb 22, 2022

I confirm: some kernel counters have decreased though this should never happen...
E.g. at 09:02:05 AM problem concerns /dev/sdc and /dev/sdn :

$ sar -f sar.txt -d -s 09:02:03 -e 09:02:05 --dev=sdc,sdn
Linux 5.4.17-2102.201.3.el7uek.x86_64 (cd-mesdb-rac01) 	02/21/2022 	_x86_64_	(104 CPU)

09:02:03 AM       DEV       tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util
09:02:05 AM       sdc 9223372036854775808.00 4611686018427387904.00    315.75      0.00      0.00      0.00      0.00      0.25
09:02:05 AM       sdn      1.50 4611686018427387904.00     43.25      0.00     26.17      0.00      1.33      0.35

Here are the counters values collected by sadc from the kernel:

$ sadf -T -r -O debug sar.txt -- -d -s 09:02:03 -e 09:02:05 --dev=sdc,sdn
# name; A_DISK; nr_curr; 32; nr_alloc; 78; nr_ini; 78
09:02:05; major; 8; minor; 32; DEV; sdc; tps [DEC]; 45009321; 45009319; rkB/s [DEC]; 992424950; 992424872; wkB/s; 57354580; 57355843; dkB/s; 0; 0; rd_ticks [DEC]; 61036794; 61036793; wr_ticks; 1698078; 1698083; dc_ticks; 0; 0; tot_ticks; 8645440; 8645445; aqu-sz; 45851636; 45851637;
09:02:05; major; 8; minor; 208; DEV; sdn; tps; 3426592; 3426595; rkB/s [DEC]; 7954166; 7954150; wkB/s; 233575494; 233575667; dkB/s; 0; 0; rd_ticks; 164315; 164315; wr_ticks; 2438140; 2438144; dc_ticks; 0; 0; tot_ticks; 5858295; 5858302; aqu-sz; 757046; 757047;
# uptime_cs; 155628921; ust_time; 1645430527; extra_next; 0; record_type; 1; HH:MM:SS; 16:02:07

For /dev/sdc device, the total number of I/O (calculated as the sum of reads, writes and discards) goes from 45009321 to 45009319, and the total number of sectors read goes from 992424950 to 992424872.
For /dev/sdn, we get a total number of sectors read which goes from 7954166 to 7954150.
All those counters should be monotonic and always increase. I cannot tell you why this happens, but it looks like the problem lies in the kernel rather than in sysstat...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants