New taskq stats #16171
New taskq stats #16171
Conversation
|
A counter for |
Hmm. I hadn't really thought much about counting what the task itself does; this was all about internal taskq behaviour. But yes, in general I think it might be interesting to know when a task did something like block, sleep, be pre-empted, or otherwise make the thread unavailable to other tasks. I probably won't try it in this PR though. (unless I'm completely misunderstanding you of course. If so, tell me, and say "well obviously you just tap your belt buckle thrice", and then I'll do the thing) |
|
Conceptually, it didn't strike me as that different a thing to track, since you're monitoring stats about how much time the tasks spent enqueued/dequeuing/waiting/delayed, and I spend a bunch of time sometimes in that question, so "how many times/how long did I spend inside/waiting on fpu sections" and the like, naturally interested me. But no, you got the idea. |
robn
force-pushed
the
taskq-stats
branch
3 times, most recently
from
c7c5184
to
92f2287
Compare
For spl-taskq to use the kstats infrastructure, it has to be available first. Sponsored-by: Klara, Inc. Sponsored-by: Wasabi Technology, Inc. Sponsored-by: Syneto Signed-off-by: Rob Norris <rob.norris@klarasystems.com>
This exposes a variety of per-taskq stats under /proc/spl/kstat/taskq, one file per taskq, named for the taskq name.instance. These include a small amount of info about the taskq config, the current state of the threads and queues, and various counters for thread and queue activity since the taskq was created. To assist with decrementing queue size counters, the list an entry is on is encoded in spare bits in the entry flags. Sponsored-by: Klara, Inc. Sponsored-by: Wasabi Technology, Inc. Sponsored-by: Syneto Signed-off-by: Rob Norris <rob.norris@klarasystems.com>
This adds /proc/spl/kstats/taskq/summary, which attempts to show a useful subset of stats for all taskqs in the system. Sponsored-by: Klara, Inc. Sponsored-by: Wasabi Technology, Inc. Sponsored-by: Syneto Signed-off-by: Rob Norris <rob.norris@klarasystems.com>
These had minimal useful information for the admin, didn't work properly in some places, and knew far too much about taskq internals. With the new stats available, these should never be needed anymore. Sponsored-by: Klara, Inc. Sponsored-by: Wasabi Technology, Inc. Sponsored-by: Syneto Signed-off-by: Rob Norris <rob.norris@klarasystems.com>
|
Last push removes the timing stats. I'll keep working on measuring throughput in a future PR; this one I think is good to go. |
Motivation and Context
I keep trying to improve performance by looking for bottlenecks with external tools, and I find something, and fix it, only to find that that thing goes away, but performance doesn't change, or changes in ways I don't expect. So I spend more and more time wondering what even is happening inside some of these black boxes. Then I decided I'd had enough, and made a start on bringing those mysteries out into the light.
So here's round 1: taskq stats. They're at the heart of so many things, and I want to know what they're up to.
Description
This gives every taskq three sets of stats:
Each taskq gets its own kstat node in
/proc/spl/kstat/taskq:Within, we find all sorts of useful things:
The meaning of most of these should be self-explanatory. The first set (down to
entries_free) are current state; the second set (down tothread_sleeps) are event counters since the beginning.The
_time_group is the most interesting, and probably the most controversial. Each entry (taskq_ent_t) now gets a timestamp at each "step" of its journey through the taskq:The delta between each step is then computed, and named: enqueue, wait, dequeue, execute.
For each type, the min/max delta since the epoch is recorded, and then the delta is mixed in to a moving average over the last 10, 100, 1000 and 10000 tasks. This makes it much easier to see the signs of, eg, queue lock contention (higher average enqueue/dequeue times), or too few worker threads (higher average wait times).
A "total" time from dispatch to completion is also recorded in the same way.
Finally, a summary list is included, showing all taskqs in the system, with a few key values for each. The idea is to give a "quick glance" state of the world at any given moment, and invite further digging:
Finally, this removes the old
/proc/spl/taskqandtaskq-allfiles.Design and implementation notes
I'm aware of how busy taskqs get, and I've tried my best to not introduce additional bottlenecks. Gauge and counter stats are all regular
wmsum_ttypes, and only read if the kstats are actually read from. The rolling averages are recorded once at the end of each task, and do not lock the taskq itself to do so, which can mean trampling but they are not intended to be accurate. I've thrown some of my performance test workloads at it (the ones that I'm trying improve) and these changes don't appear to move the needle, so I feel at least okay. about it. It'd be good to get more testing. Importantly, reading the stats does not take the taskq lock, since the stats are held separately.The time stats have been useful already (a couple of interesting quirks discovered which I'll chase down later). They still don't quite feel right to me. I'm not sure if min/max is ever going to be useful. The averages undoubtedly are useful, but "last N tasks" is only useful during sustained operation, which many taskqs don't see. I'm actually not very good at math (moving average is my only trick 😅), so I'm hoping someone will say "oh, this is a perfect use for Numberwang!"
I'm not super persuaded about the summary list. So far, the "threads" column has been unnecessary and the "tasks on queue" column has been great (though I can take or leave "delay"). The averages column should be the most interesting, but there's not a single granularity that works for all taskq types - high-throughput queues like z_wr and z_rd really need 10000, but rarely-used taskqs might be better at 10, and there's everything in between. I did think a bit about choosing the most likely range based on total tasks, but again, it's tricky without some notion of time, and taskqs don't have a heartbeat, and I would prefer to keep it that way.
Your ideas for other stats and presentation welcome, as I would like this to be actually good. Yes, I am inviting bikeshedding 😇
Removing the old stats might not be wanted. I personally don't see the point in keeping them; they show some different things but they're mostly about taskq configuration (flags etc) which aren't especially useful to the reader and can't be seen anyway. When they're busy, dumping tasks is quite pointless and doesn't seem to work well anyway. Finally, the implementation is pretty invasive, messing around inside the taskq itself. But if someone wants to keep them, it won't particularly hurt anything; just be mess.
Finally, I would like to do something similar for FreeBSD, but since we use a lot more of the native taskqueues there, it might be an effort to actually get anything useful. I wouldn't hold this PR just for that, but if you know more about this I would like to hear about it.
How Has This Been Tested?
A lot by hand, including performance tests. A full ZTS run has been completed, but since it doesn't use anything from
/proc/spl/taskqI wasn't expecting it to show much and indeed, it did not.Types of changes
Checklist:
Signed-off-by.