Articles

Performance analysis and bottlenecks

Finding the resource that binds: methodology, the tools, and what each counter proves.

Reading: 5 minServer & Virtualization

Article cover: Performance analysis and bottlenecks

A host that feels slow is not a mystery; it is a queue that has filled somewhere. Performance work begins by finding the resource that binds the rest — CPU, memory, a storage device, a network interface — and proving it with a counter that measures the right thing.

The resource comes before the tool

The Utilization, Saturation and Errors (USE) Method is a checklist, not a recipe. Its rule is one sentence: “For every resource, check utilization, saturation, and errors.” It is a method rather than a metric because it “begins by posing questions, and then seeks answers, instead of beginning with given metrics (partial answers) and trying to work backwards”: you enumerate the resources a server has, then ask the three questions of each. A tool-driven approach fails in a known way, because it “relies exclusively on available (or known) tools, which can provide an incomplete view of the system”.

The same body of work names the anti-methods this checklist replaces, beginning with the streetlight method, which picks tools because they are “familiar” or “found on the Internet”. Without a method, analysis “can become a fishing expedition, where metrics are examined ad hoc, until the issue is found”, and the first bottleneck it surfaces is a lead, not a verdict: it “may be a problem but not the problem”.

What the three questions prove

The three questions are not synonyms. Utilization is “the average time that the resource was busy servicing work”; saturation is “the degree to which the resource has extra work which it can’t service, often queued”; errors are “the count of error events”. “100% utilization is usually a sign of a bottleneck (check saturation and its effect to confirm)”; “any degree of saturation can be a problem (non-zero)”; and “non-zero error counters are worth investigating, especially if they are still increasing while performance is poor”.

The decisive subtlety is that utilization is an average while saturation is a queue. “A burst of high utilization can cause saturation and performance issues, even though utilization is low when averaged over a long interval.” In the author’s example a monitored CPU “was never higher than 80%” yet still suffered saturation-driven latency, because the tool “was reporting five minute averages, during which CPU utilization hit 100% for seconds at a time.”

The tools, and what each one proves

Each tool proves a different part of the checklist; naming which part keeps the list honest.

Tool Scope What the counter proves
vmstat the whole host r (runnable) and b (blocked on I/O) are saturation; si/so are paging; wa separates I/O wait from st, time taken by the hypervisor
iostat one device await mixes queue time and service time; aqu-sz is the queue length; %util is utilization, and stops meaning saturation on parallel devices
pidstat one process %wait (waiting to run), iodelay (block I/O delay), majflt/s (pages fetched from disk) attribute the load to a task
sar the past the same counters, read back from the daily file after the problem has passed
perf hardware and tracepoint events counters the kernel does not otherwise export

vmstat gives the quickest whole-host view, reporting “information about processes, memory, paging, block IO, traps, disks and cpu activity”, and “Time stolen from a virtual machine” is the reading a guest should treat as a signal about its neighbours. iostat is per device, and its central field mixes what the method separates: the await line reads “The average time (in milliseconds) for I/O requests issued to the device to be served”, a wait that “includes the time spent by the requests in queue and the time spent servicing them”. pidstat moves the questions down to one process, where %wait is “Percentage of CPU spent by the task while waiting to run”; sar alone looks backwards, and will “Collect, report, or save system activity information”.

   iterate the resources, ask the same three questions, keep the lead

   resource            utilization          saturation              errors
   ----------------------------------------------------------------------
   CPU                 vmstat us+sy         vmstat r (run queue)    perf
                       sar -u               pidstat %wait
   memory capacity     vmstat free          vmstat si / so          malloc fails
   storage device I/O  iostat %util         iostat aqu-sz, await    device errors
   network interface   throughput / limit   drops / overruns        NIC errors
   controllers and     the counters the hardware exports, where it exports them
   interconnects
   ----------------------------------------------------------------------
   the first non-zero saturation, or the reading nearest its limit, is the
   lead and not yet the answer: drill down, then continue down the list

   the average hides the burst: a five-minute mean of 70% can be 100% for
   seconds at a time, and the tail is where the saturation lives

perf: the events, and the price of looking

perf reaches counters the other tools never expose. The kernel’s perf_events subsystem will “provide a framework for all things performance analysis”, and it “covers hardware level (CPU/PMU, Performance Monitoring Unit) features and software features (software counters, tracepoints) as well”. That breadth is also the hazard: “Data that belong to the fourth category can potentially contain sensitive process data.” The intended capability is CAP_PERFMON, which “implements the principle of least privilege” for this monitoring; what an unprivileged process may then see is set by perf_event_paranoid, where the value 2 holds the scope to “per-process performance monitoring only”. The cost is paid in resources too: a session needs no fewer than “the number of monitored events multiplied by the number of monitored CPUs” open file descriptors, and its buffers are bounded by perf_event_mlock_kb.

A counter that does not say what it looks like it says

Kernel Samepage Merging is the case that catches a hasty reading. It is “a memory-saving de-duplication feature” whose daemon “periodically scans those areas of user memory which have been registered with it, looking for pages of identical content which can be replaced by a single write-protected page”. Free memory therefore rises for reasons that are real but not free, and the counters that report the saving sit beside counters that report the cost: “A high ratio of pages_sharing to pages_shared indicates good sharing, but a high ratio of pages_unshared to pages_sharing indicates wasted effort.” The saving has a latency side too: it “might help save memory, it’s a tradeoff by may suffering delay on KSM COW or on swapping in copy”, and the scans cost as well: “KSM’s scans may use a lot of processing power: some installations will disable KSM for that reason”. A falling memory figure on a deduplicating host is a claim to be checked against the copy-on-write and scan counters.

The three ways to waste the exercise

The first error is to watch the average and not the tail. Saturation is a queue, and a queue lives in the busy seconds, not in the interval mean; an average that never exceeds 80% can hide a resource saturated for a fraction of every minute, so read the maximum over a short interval, not the daily figure. The second error is to measure after changing something: a tunable altered first one way and then the other, with no baseline kept, leaves nothing to compare against, and the problem-statement method’s first question — “What has changed recently? (Software? Hardware? Load?)” — is also the rule for keeping a measurement honest. The third error is to optimise the resource easiest to see rather than the one that binds: storage utilisation is one visible number and an interconnect is not.

Level and prerequisites

L5 — specialist: diagnosing a busy platform by method, choosing the measurement that proves the bottleneck, not the one quickest to run. Two things the sheet assumes are settled elsewhere: the three numbers of a medium — IOPS, throughput and latency — and how the medium and its interface decide which binds belong to the L1 sheet on HDD, SSD and NVMe; and how full an environment really is, and what headroom must stay free so a host can leave, belong to the L4 sheet on capacity planning and host maintenance. What stays here is the method and what each counter proves.

Where to go next

  • Server & Virtualization — the area this sheet belongs to.
  • The storage-medium sheet (L1, HDD, SSD and NVMe) for the three numbers a device is judged by.
  • The capacity-planning sheet (L4, Capacity planning and host maintenance) for committed capacity and the reserve.

References

  • Brendan Gregg — The USE Method (gregg-usemethod.txt).
  • Brendan Gregg — Performance Analysis Methodology (gregg-method.txt).
  • Linux man-pages — perf(1), iostat(1), pidstat(1), vmstat(8), sar(1) (man-perf.txt, man-iostat.txt, man-pidstat.txt, man-vmstat.txt, man-sar.txt).
  • The Linux Kernel documentation — Perf events and tool security (k-perf-security.txt).
  • The Linux Kernel documentation — Kernel Samepage Merging (k-ksm.txt).
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.