Articles

High availability and failure handling

Restarting workloads after a failure: fencing, timeouts, and how it differs from a backup.

Reading: 5 minServer & Virtualization

Article cover: High availability and failure handling

High availability automates the decision an operator would otherwise make by hand: a workload has stopped answering, so start it somewhere else. It is safe only if one condition holds — before the workload is restarted, the failed node must be guaranteed unable to run it again. This sheet follows that arc: the restart policy, the fencing and watchdog that make a failed node provably dead, the timeouts that separate a real fault from a slow one, and the maintenance that turns failure into a plan. It closes on the limit that matters most: restarting is not restoring.

What the manager automates

The Proxmox VE HA manager states its job precisely: it “automatically detects node failures using watchdog-based fencing and restarts affected services on healthy nodes”. Its unit is an HA resource — a virtual machine or a Linux container, addressed by a service ID such as vm:100 — and every resource carries a requested state (started, stopped, disabled, ignored); a resource requested as started is restarted when it crashes.

The automation has a ceiling and an entry cost: its “typical error detection and failover times of about 2 minutes” cap a cluster at 99.999%, and the environment already needs three nodes, shared storage and redundant hardware. Membership and quorum are the neighbouring sheet’s subject; here they are only the precondition.

   a node stops answering
            |
            v
   +--------------------+   not a real fault
   | is the fault real? |------------------>  rejoin; no restart
   | timeouts/thresholds|
   +---------+----------+
             | real fault
             v
   +--------------------+   node not fenced
   | fence the node     |------------------>  STOP: two owners must
   | watchdog / STONITH |                      never write the same disk
   +---------+----------+
             | node guaranteed offline
             v
   restart the resource elsewhere
   restart first, then relocate

The restart policy: restart, then relocate

A workload that will not start and a node that has died are answered by different counters. The start failure policy “comes into effect if a service failed to start on a node one or more times” and holds two limits: max_restart, the “Maximum number of attempts to restart a failed service on the actual node”, one by default, and max_relocate. The manager restarts first and relocates only “after the max_restart value is exceeded on the actual node”; the relocate counter resets only after a successful start.

The policy rides out a shared resource briefly unavailable on one node without declaring the node lost. If both counters are exhausted the resource enters the error state, where it “won’t get touched by the HA stack anymore”; the only way out is disabling it:

# ha-manager set vm:100 --state disabled

Fencing and the watchdog: the precondition for restart

Before anything is restarted, the cluster must prove the old owner is dead: “fencing ensures that the erroneous node is guaranteed to be offline”. A resource must run in exactly one place, and without fencing “it would not be possible to recover a resource on another node”. A node that has lost every network except storage is unreachable yet still running, “in an unknown state where it may have still access to shared resources”; start its virtual machine elsewhere and “we would get a dangerous race condition, because we write from both nodes”; such “conditions can destroy all VM data and the whole VM could be rendered unusable”. The Windows failover-cluster documentation names the same hazard: replicated or DFS-backed storage can open “a partition-in-space or partition-in-time, where the cluster nodes are running independently of each other and could cause data loss”.

Proxmox VE fences either by isolating the node “by disabling complete network traffic on the switch” or by “self fencing using watchdog timers”; the family’s wider name is STONITH, which “requires STONITH to deal with various scenarios”. The watchdog is independent of the software it supervises: the manager resets it on a schedule, and if the resets stop it elapses and reboots the node, because “the computer fails to reset the watchdog, the timer will elapse and trigger a reset of the whole server”. On lost quorum or a daemon failure, “the respective watchdog triggers a node reset to ensure safe failover”. Without a hardware watchdog enabled on the motherboard the stack falls back to the kernel’s softdog, which “While still reliable, it is not independent of the servers hardware, and thus has a lower reliability than a hardware watchdog”.

Timeouts and thresholds: deciding whether the fault is real

A heartbeat that is merely slow must not read as a death. The token and consensus timeouts give the time to re-establish membership, and “Keeping the sum of token and consensus timeouts below 30 seconds reduces the time needed for reestablishing a new cluster membership after a node failure”. With HA enabled it must stay “below 45 seconds to ensure that a new cluster membership is formed before the watchdog timeout of 60 seconds expires, which would trigger a node fence”. Membership must re-form before the watchdog fires, or a slow link becomes a false fence: on default settings “nodes with HA resources will fence themselves already after roughly one minute without a stable quorum”.

Maintenance: a failure you schedule

Planned downtime uses the same machinery. Maintenance mode lets an operator “mark the node as unavailable for HA operation, prompting all services managed by HA to migrate to other nodes”, and records where each service was, so that “the service can be moved back again automatically once the maintenance mode is disabled and the node is back online”.

For maintenance that would itself disturb the cluster the manager can be disarmed: it releases every watchdog cluster-wide, and “While the HA stack is disarmed, no automatic recovery, failover, or fencing takes place”, so “Keep the disarm window as short as possible”.

Windows does the same as a feature: Cluster-Aware Updating “Puts each node of the cluster into node maintenance mode”, “Moves the clustered roles off the node”, updates it, and “Moves to update the next node”, at the cost of “a transient service interruption for connected clients”.

High availability is not a backup

Everything above keeps a service answering; none of it keeps the data. A restart moves the workload’s current state — wrong writes included — onto another host. Deletion, silent corruption, encryption by ransomware and the loss of the site are not node failures, so none is inside what the manager does: it will faithfully restart the very workload that is encrypting the shared storage. The line between keeping a workload running and keeping its data recoverable is drawn elsewhere in this area, by the L3 sheets “RAID and backup: availability is not protection” and “Retention, immutability and separate copies”. High availability restarts; a backup restores.

Level and prerequisites

L4 — advanced reliability: what happens after a failure and what makes an automatic restart safe. The reader should know what a cluster resource is and that membership and quorum decide which nodes may act; that belongs to the neighbouring sheet and is not restated here. Only the manager’s vocabulary (resource, fence, watchdog, relocate) is assumed.

Where to go next

  • Server & Virtualization — the area this sheet belongs to.
  • Sister sheets in preparation: Clusters and quorum (the precondition assumed here); Live migration and its prerequisites; Affinity, anti-affinity and scheduling (where a recovered service may land); Capacity planning and host maintenance (the headroom an evacuation needs).

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.