High availability and failure handling
Restarting workloads after a failure: fencing, timeouts, and how it differs from a backup.

High availability automates the decision an operator would otherwise make by hand: a workload has stopped answering, so start it somewhere else. It is safe only if one condition holds — before the workload is restarted, the failed node must be guaranteed unable to run it again. This sheet follows that arc: the restart policy, the fencing and watchdog that make a failed node provably dead, the timeouts that separate a real fault from a slow one, and the maintenance that turns failure into a plan. It closes on the limit that matters most: restarting is not restoring.
What the manager automates
The Proxmox VE HA manager states its job precisely: it “automatically detects node failures using watchdog-based fencing and restarts affected services on healthy nodes”. Its unit is an HA resource — a virtual machine or a Linux container, addressed by a service ID such as vm:100 — and every resource carries a requested state (started, stopped, disabled, ignored); a resource requested as started is restarted when it crashes.
The automation has a ceiling and an entry cost: its “typical error detection and failover times of about 2 minutes” cap a cluster at 99.999%, and the environment already needs three nodes, shared storage and redundant hardware. Membership and quorum are the neighbouring sheet’s subject; here they are only the precondition.
a node stops answering
|
v
+--------------------+ not a real fault
| is the fault real? |------------------> rejoin; no restart
| timeouts/thresholds|
+---------+----------+
| real fault
v
+--------------------+ node not fenced
| fence the node |------------------> STOP: two owners must
| watchdog / STONITH | never write the same disk
+---------+----------+
| node guaranteed offline
v
restart the resource elsewhere
restart first, then relocate
The restart policy: restart, then relocate
A workload that will not start and a node that has died are answered by different counters. The start failure policy “comes into effect if a service failed to start on a node one or more times” and holds two limits: max_restart, the “Maximum number of attempts to restart a failed service on the actual node”, one by default, and max_relocate. The manager restarts first and relocates only “after the max_restart value is exceeded on the actual node”; the relocate counter resets only after a successful start.
The policy rides out a shared resource briefly unavailable on one node without declaring the node lost. If both counters are exhausted the resource enters the error state, where it “won’t get touched by the HA stack anymore”; the only way out is disabling it:
# ha-manager set vm:100 --state disabled
Fencing and the watchdog: the precondition for restart
Before anything is restarted, the cluster must prove the old owner is dead: “fencing ensures that the erroneous node is guaranteed to be offline”. A resource must run in exactly one place, and without fencing “it would not be possible to recover a resource on another node”. A node that has lost every network except storage is unreachable yet still running, “in an unknown state where it may have still access to shared resources”; start its virtual machine elsewhere and “we would get a dangerous race condition, because we write from both nodes”; such “conditions can destroy all VM data and the whole VM could be rendered unusable”. The Windows failover-cluster documentation names the same hazard: replicated or DFS-backed storage can open “a partition-in-space or partition-in-time, where the cluster nodes are running independently of each other and could cause data loss”.
Proxmox VE fences either by isolating the node “by disabling complete network traffic on the switch” or by “self fencing using watchdog timers”; the family’s wider name is STONITH, which “requires STONITH to deal with various scenarios”. The watchdog is independent of the software it supervises: the manager resets it on a schedule, and if the resets stop it elapses and reboots the node, because “the computer fails to reset the watchdog, the timer will elapse and trigger a reset of the whole server”. On lost quorum or a daemon failure, “the respective watchdog triggers a node reset to ensure safe failover”. Without a hardware watchdog enabled on the motherboard the stack falls back to the kernel’s softdog, which “While still reliable, it is not independent of the servers hardware, and thus has a lower reliability than a hardware watchdog”.
Timeouts and thresholds: deciding whether the fault is real
A heartbeat that is merely slow must not read as a death. The token and consensus timeouts give the time to re-establish membership, and “Keeping the sum of token and consensus timeouts below 30 seconds reduces the time needed for reestablishing a new cluster membership after a node failure”. With HA enabled it must stay “below 45 seconds to ensure that a new cluster membership is formed before the watchdog timeout of 60 seconds expires, which would trigger a node fence”. Membership must re-form before the watchdog fires, or a slow link becomes a false fence: on default settings “nodes with HA resources will fence themselves already after roughly one minute without a stable quorum”.
Maintenance: a failure you schedule
Planned downtime uses the same machinery. Maintenance mode lets an operator “mark the node as unavailable for HA operation, prompting all services managed by HA to migrate to other nodes”, and records where each service was, so that “the service can be moved back again automatically once the maintenance mode is disabled and the node is back online”.
For maintenance that would itself disturb the cluster the manager can be disarmed: it releases every watchdog cluster-wide, and “While the HA stack is disarmed, no automatic recovery, failover, or fencing takes place”, so “Keep the disarm window as short as possible”.
Windows does the same as a feature: Cluster-Aware Updating “Puts each node of the cluster into node maintenance mode”, “Moves the clustered roles off the node”, updates it, and “Moves to update the next node”, at the cost of “a transient service interruption for connected clients”.
High availability is not a backup
Everything above keeps a service answering; none of it keeps the data. A restart moves the workload’s current state — wrong writes included — onto another host. Deletion, silent corruption, encryption by ransomware and the loss of the site are not node failures, so none is inside what the manager does: it will faithfully restart the very workload that is encrypting the shared storage. The line between keeping a workload running and keeping its data recoverable is drawn elsewhere in this area, by the L3 sheets “RAID and backup: availability is not protection” and “Retention, immutability and separate copies”. High availability restarts; a backup restores.
Level and prerequisites
L4 — advanced reliability: what happens after a failure and what makes an automatic restart safe. The reader should know what a cluster resource is and that membership and quorum decide which nodes may act; that belongs to the neighbouring sheet and is not restated here. Only the manager’s vocabulary (resource, fence, watchdog, relocate) is assumed.
Where to go next
- Server & Virtualization — the area this sheet belongs to.
- Sister sheets in preparation: Clusters and quorum (the precondition assumed here); Live migration and its prerequisites; Affinity, anti-affinity and scheduling (where a recovered service may land); Capacity planning and host maintenance (the headroom an evacuation needs).
References
- Proxmox Server Solutions GmbH, High Availability (Proxmox VE wiki) — https://pve.proxmox.com/wiki/High_Availability — fencing, watchdog timers and the rationale for self-fencing.
- Proxmox Server Solutions GmbH, Proxmox VE Administration Guide, Chapter 15, High Availability — https://pve.proxmox.com/pve-docs/pve-admin-guide.html — the HA manager, the start failure policy, fencing and watchdog status, maintenance and disarm, and the Corosync timeout guidance.
- Microsoft, Deploy a quorum witness for a failover cluster in Windows Server — https://learn.microsoft.com/en-us/windows-server/failover-clustering/manage-cluster-quorum — the partition-in-space / partition-in-time warning for replicated storage.
- Microsoft, Cluster-Aware Updating overview — https://learn.microsoft.com/en-us/windows-server/cluster-aware-updating/cluster-aware-updating-overview — the node-by-node maintenance sequence and its transient interruption.
- LINBIT, DRBD User’s Guide 9.0 — https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/ — the meaning and cost of STONITH in a cluster resource manager.