Articles

Capacity planning and host maintenance

Knowing how full an environment really is, and taking a host out without an outage.

Reading: 5 minServer & Virtualization

Article cover: Capacity planning and host maintenance

An environment is rarely as full as its allocation suggests, and never as empty as it looks the day a host must leave. Capacity planning separates two facts: measuring how much of each host is genuinely committed, and keeping enough free that one host — for maintenance or failure — can be removed without dropping a workload.

        how full is the cluster — and what one host's absence asks for

   host1  declared ███████████████   used ████████░░░   headroom ███
   host2  declared ██████████████    used ██████████░   headroom █
   host3  declared ███████████████   used ████████░░░   headroom ███   <- drained
   host4  declared █████████████     used ███████░░░░   headroom ████

   to take host3 out with no outage, its declared load must fit, at once,
   in the headroom of hosts 1, 2 and 4. If it does not, the maintenance
   window is an outage waiting for a date.

   the loop:  measure -> plan -> drain -> update -> verify -+
                 ^                                          |
                 +------------------------------------------+

Allocated, committed, and the number that matters

The hypervisor answers two different questions with what looks like one figure. A machine is promised a memory size and a number of processors — allocated capacity, what the interface sums as a host’s capacity. The host must back only the pages and threads its guests actually touch: committed capacity, closer to the truth. The gap is oversubscription, developed in its own sheet; here it is only why a total of declared sizes is not a measurement.

Memory fills first; a load balancer is built around that: its first heuristic is “Current memory pressure: Memory is the most common resource constraint on a Hyper-V host.” Commitment is not fixed either: “The buffer isn’t maintained when there’s not enough physical memory available in the computer to give every virtual machine its requested memory buffer”. The Dynamic Memory Balancer’s “Available Memory” counter reads the real figure; a plan built on allocated sizes is built on the most optimistic number the platform will show.

The reserve a host’s absence requires

The figure to protect is not the total but the reserve: the capacity that must stay unused so a host can leave. Proxmox VE states the rule for its hyper-converged storage: usage “may be less under normal conditions”, but the storage layer “will consume more memory during critical operations, such as recovery, rebalancing, or backfilling”, so “you should avoid maxing out your available memory already on regular operation, but rather leave some headroom to cope with outages.” The same reasoning sizes the compute reserve: a node running a monitor, a manager and six storage daemons “should reserve 8 CPU cores purely for Ceph”.

The result is called n+1 — enough spare capacity that losing any one host still fits its guests. Proxmox VE’s scheduler weighs memory more heavily than CPU, “because it’s a truly limited resource”; a reserve is not spare disk on a datastore but memory and CPU threads no guest has been promised.

Taking a host out: maintenance mode and evacuation

Removing a host on purpose must be planned and reversible. Proxmox VE provides it as maintenance mode, whose purpose is “to mark the node as unavailable for HA operation, prompting all services managed by HA to migrate to other nodes.” Destinations are “selected from the other currently available nodes, and determined by the HA rules configuration and the configured cluster resource scheduler (CRS) mode”, and the original node is remembered so “the service can be moved back again automatically once the maintenance mode is disabled and the node is back online.”

Two properties make the drain safe: the node keeps its watchdog while it empties — it “will stay active until all active services got moved, and all running workers finished” — and the move can be done in bulk, since “pvenode also offers the migrateall subcommand for bulk migration”. The mechanics of each move belong to the live migration sheet, and a placement rule that could forbid one belongs to the scheduling sheet.

Updating a cluster without stopping it

The drained host then has to be updated, and that is the cycle Cluster-Aware Updating (CAU) automates on a Windows Server failover cluster: “an automated feature that enables you to update servers in a failover cluster with little or no loss in availability during the update process.” A run performs a fixed sequence: it “Puts each node of the cluster into node maintenance mode”, “Moves the clustered roles off the node”, installs the updates, “Performs a restart if necessary”, “Brings the node out of maintenance mode”, “Moves to update the next node” — one host at a time, the n+1 reserve spent and repaid.

The no-outage promise is conditional: for many clustered roles the update “triggers a planned failover”, which “can cause a transient service interruption for connected clients”; zero impact is claimed only “in the case of continuously available workloads, such as Hyper-V with live migration or file server with SMB Transparent Failover”.

Rebalancing: putting the load back afterwards

After maintenance the load is uneven: the drained host returns empty while the others carry its guests. Both platforms answer with a balancer. The Windows one “identifies over-committed machines and live migrates VMs from those machines to under-committed machines”, evaluating load by memory pressure and by “CPU utilization averaged over a five-minute window: Mitigates any machine in the system from becoming over-committed.” It is opt-in: “The default setting is Never”.

Proxmox VE rebalances through the same HA manager that ran the drain. When “resources on one node become bottlenecked due to resource contention, while another node is barely utilized”, the load balancer lowers the imbalance “by automatically issuing rebalancing migrations” that “always obey the current HA affinity rules”.

The operating cycle, and the trap it closes

Put together, the practice is a loop: measure how much each host really carries in memory and CPU; plan against the reserve, not the total; drain a host through maintenance mode and a bulk migration; update it with a cluster-aware run that moves one node at a time; verify the workload is healthy and the reserve is back.

The recurring error is to measure the wrong thing. Counting virtual machines per host — six here, nine there — says nothing about their weight: a host with fewer, larger guests can be the fuller one. Building the plan on allocated sizes repeats the mistake: allocated memory is what the guests were promised, not what they consume. The shortfall stays invisible until the day a host is drained, when the missing reserve turns a maintenance window into an outage. The number to watch is not how many workloads a host holds, but how much memory and how many CPU threads remain unused once the other hosts are as busy as planned.

Level and prerequisites

L4 — operating a full environment: measuring real headroom, keeping a reserve large enough that a host can leave, and removing a host without a service interruption. The quantity rules it assumes belong to the resource allocation and oversubscription sheet; the placement rules to the affinity, anti-affinity and scheduling sheet; the live move to the live migration sheet. It sits above L3 because it reasons about the fleet rather than one machine’s declaration, and below platform design because it configures an existing cluster.

Where to go next

  • Server & Virtualization — the area this sheet belongs to.
  • The sheets that fix the quantities and the places this sheet assumes: resource allocation and oversubscription, and affinity, anti-affinity and scheduling.
  • The sheet that moves a running guest: live migration and its prerequisites.

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.