Performance analysis and bottlenecks
Finding the resource that binds: methodology, the tools, and what each counter proves.

A host that feels slow is not a mystery; it is a queue that has filled somewhere. Performance work begins by finding the resource that binds the rest — CPU, memory, a storage device, a network interface — and proving it with a counter that measures the right thing.
The resource comes before the tool
The Utilization, Saturation and Errors (USE) Method is a checklist, not a recipe. Its rule is one sentence: “For every resource, check utilization, saturation, and errors.” It is a method rather than a metric because it “begins by posing questions, and then seeks answers, instead of beginning with given metrics (partial answers) and trying to work backwards”: you enumerate the resources a server has, then ask the three questions of each. A tool-driven approach fails in a known way, because it “relies exclusively on available (or known) tools, which can provide an incomplete view of the system”.
The same body of work names the anti-methods this checklist replaces, beginning with the streetlight method, which picks tools because they are “familiar” or “found on the Internet”. Without a method, analysis “can become a fishing expedition, where metrics are examined ad hoc, until the issue is found”, and the first bottleneck it surfaces is a lead, not a verdict: it “may be a problem but not the problem”.
What the three questions prove
The three questions are not synonyms. Utilization is “the average time that the resource was busy servicing work”; saturation is “the degree to which the resource has extra work which it can’t service, often queued”; errors are “the count of error events”. “100% utilization is usually a sign of a bottleneck (check saturation and its effect to confirm)”; “any degree of saturation can be a problem (non-zero)”; and “non-zero error counters are worth investigating, especially if they are still increasing while performance is poor”.
The decisive subtlety is that utilization is an average while saturation is a queue. “A burst of high utilization can cause saturation and performance issues, even though utilization is low when averaged over a long interval.” In the author’s example a monitored CPU “was never higher than 80%” yet still suffered saturation-driven latency, because the tool “was reporting five minute averages, during which CPU utilization hit 100% for seconds at a time.”
The tools, and what each one proves
Each tool proves a different part of the checklist; naming which part keeps the list honest.
| Tool | Scope | What the counter proves |
|---|---|---|
vmstat |
the whole host | r (runnable) and b (blocked on I/O) are saturation; si/so are paging; wa separates I/O wait from st, time taken by the hypervisor |
iostat |
one device | await mixes queue time and service time; aqu-sz is the queue length; %util is utilization, and stops meaning saturation on parallel devices |
pidstat |
one process | %wait (waiting to run), iodelay (block I/O delay), majflt/s (pages fetched from disk) attribute the load to a task |
sar |
the past | the same counters, read back from the daily file after the problem has passed |
perf |
hardware and tracepoint events | counters the kernel does not otherwise export |
vmstat gives the quickest whole-host view, reporting “information about processes, memory, paging, block IO, traps, disks and cpu activity”, and “Time stolen from a virtual machine” is the reading a guest should treat as a signal about its neighbours. iostat is per device, and its central field mixes what the method separates: the await line reads “The average time (in milliseconds) for I/O requests issued to the device to be served”, a wait that “includes the time spent by the requests in queue and the time spent servicing them”. pidstat moves the questions down to one process, where %wait is “Percentage of CPU spent by the task while waiting to run”; sar alone looks backwards, and will “Collect, report, or save system activity information”.
iterate the resources, ask the same three questions, keep the lead
resource utilization saturation errors
----------------------------------------------------------------------
CPU vmstat us+sy vmstat r (run queue) perf
sar -u pidstat %wait
memory capacity vmstat free vmstat si / so malloc fails
storage device I/O iostat %util iostat aqu-sz, await device errors
network interface throughput / limit drops / overruns NIC errors
controllers and the counters the hardware exports, where it exports them
interconnects
----------------------------------------------------------------------
the first non-zero saturation, or the reading nearest its limit, is the
lead and not yet the answer: drill down, then continue down the list
the average hides the burst: a five-minute mean of 70% can be 100% for
seconds at a time, and the tail is where the saturation lives
perf: the events, and the price of looking
perf reaches counters the other tools never expose. The kernel’s perf_events subsystem will “provide a framework for all things performance analysis”, and it “covers hardware level (CPU/PMU, Performance Monitoring Unit) features and software features (software counters, tracepoints) as well”. That breadth is also the hazard: “Data that belong to the fourth category can potentially contain sensitive process data.” The intended capability is CAP_PERFMON, which “implements the principle of least privilege” for this monitoring; what an unprivileged process may then see is set by perf_event_paranoid, where the value 2 holds the scope to “per-process performance monitoring only”. The cost is paid in resources too: a session needs no fewer than “the number of monitored events multiplied by the number of monitored CPUs” open file descriptors, and its buffers are bounded by perf_event_mlock_kb.
A counter that does not say what it looks like it says
Kernel Samepage Merging is the case that catches a hasty reading. It is “a memory-saving de-duplication feature” whose daemon “periodically scans those areas of user memory which have been registered with it, looking for pages of identical content which can be replaced by a single write-protected page”. Free memory therefore rises for reasons that are real but not free, and the counters that report the saving sit beside counters that report the cost: “A high ratio of pages_sharing to pages_shared indicates good sharing, but a high ratio of pages_unshared to pages_sharing indicates wasted effort.” The saving has a latency side too: it “might help save memory, it’s a tradeoff by may suffering delay on KSM COW or on swapping in copy”, and the scans cost as well: “KSM’s scans may use a lot of processing power: some installations will disable KSM for that reason”. A falling memory figure on a deduplicating host is a claim to be checked against the copy-on-write and scan counters.
The three ways to waste the exercise
The first error is to watch the average and not the tail. Saturation is a queue, and a queue lives in the busy seconds, not in the interval mean; an average that never exceeds 80% can hide a resource saturated for a fraction of every minute, so read the maximum over a short interval, not the daily figure. The second error is to measure after changing something: a tunable altered first one way and then the other, with no baseline kept, leaves nothing to compare against, and the problem-statement method’s first question — “What has changed recently? (Software? Hardware? Load?)” — is also the rule for keeping a measurement honest. The third error is to optimise the resource easiest to see rather than the one that binds: storage utilisation is one visible number and an interconnect is not.
Level and prerequisites
L5 — specialist: diagnosing a busy platform by method, choosing the measurement that proves the bottleneck, not the one quickest to run. Two things the sheet assumes are settled elsewhere: the three numbers of a medium — IOPS, throughput and latency — and how the medium and its interface decide which binds belong to the L1 sheet on HDD, SSD and NVMe; and how full an environment really is, and what headroom must stay free so a host can leave, belong to the L4 sheet on capacity planning and host maintenance. What stays here is the method and what each counter proves.
Where to go next
- Server & Virtualization — the area this sheet belongs to.
- The storage-medium sheet (L1, HDD, SSD and NVMe) for the three numbers a device is judged by.
- The capacity-planning sheet (L4, Capacity planning and host maintenance) for committed capacity and the reserve.
References
- Brendan Gregg — The USE Method (
gregg-usemethod.txt). - Brendan Gregg — Performance Analysis Methodology (
gregg-method.txt). - Linux man-pages — perf(1), iostat(1), pidstat(1), vmstat(8), sar(1) (
man-perf.txt,man-iostat.txt,man-pidstat.txt,man-vmstat.txt,man-sar.txt). - The Linux Kernel documentation — Perf events and tool security (
k-perf-security.txt). - The Linux Kernel documentation — Kernel Samepage Merging (
k-ksm.txt).