Articles

Clusters and quorum: who declares a failure

How a cluster agrees who is alive, what quorum protects, and what its loss costs.

Reading: 5 minServer & Virtualization

Article cover: Clusters and quorum: who declares a failure

A virtualization cluster is a set of hosts that manage workloads as one domain, and the first thing they must agree on is who is alive. The answer is not a roll call — a silent host is not voting no, it is absent — and not unanimity either. A cluster counts votes and acts only while it holds a quorum, “the minimum number of votes that a distributed transaction has to obtain in order to be allowed to perform an operation in a distributed system”. That count is not about peak availability: it stops two halves of the same cluster from both believing they are the cluster. This sheet is about that decision: who holds a vote, what a witness adds, and what losing quorum costs.

   five nodes, one vote each, quorum = three

      [n1]---[n2]---[n3]---[n4]---[n5]
       1      1      1      1      1       five votes -> quorate

      the link between n3 and n4 fails

      { n1  n2  n3 }   |   { n4  n5 }
          3 votes       |    2 votes
        >= quorum       |   < quorum
        keeps serving   |   stops; it cannot form a second cluster

A vote, not a roll call

A cluster manager groups hosts and runs a distributed protocol underneath. Proxmox VE uses the Corosync Cluster Engine for group communication and replicates its configuration through a database-backed file system; Windows uses Failover Clustering for the same role. What matters is the arithmetic. “Proxmox VE assigns a single vote to each node by default”, and the status tool exposes the sum: a four-node cluster reports four expected votes, four total votes and a quorum of three. Three votes are a majority of four; one or two are not. The difference from a roll call is direction. Unanimity asks whether every host answered; quorum asks whether a majority did, and counts a silent host as a missing vote rather than a veto.

The majority the cluster acts on

The property a majority buys is exclusivity: two subsets cannot each hold more than half of a set, so at most one partition can be quorate at a time. So the cluster stays a single decision-making body even when its network does not. Proxmox states the consequence — “In case of network partitioning, state changes requires that a majority of nodes are online. The cluster switches to read-only mode if it loses quorum.” Read-only is a decision, not a fault: the cluster does not guess which side is right or let both act; it withdraws state changes until the votes are counted again. Availability is what it gives up to keep its members from diverging.

A witness: a vote where there is no host

Another failure is arithmetical: an even number of nodes has no majority without all of them, so a two-node cluster whose link is down has no quorate side. The remedy is a vote that is not a full member — a witness. Windows Failover Clustering offers “three types of quorum witnesses” — disk, file share or cloud — and notes that modern clusters “automatically manage the witness vote and their nodes vote with Dynamic Quorum”. A disk witness is “The disk used for the witness must be a shared disk”; a file share witness is an SMB share, and Microsoft asks that “The file share should be physically separate from the cluster nodes or cluster sites”, because a witness on the same power or network can vanish with the nodes. A cloud witness is the same idea off-site: a blob holding “the blob file required for voting arbitration”. Proxmox’s QDevice is the same, a daemon that supplies votes on an external arbitrator’s verdict, which “can see all nodes and thus choose only one set of nodes to give its vote”.

What a lost quorum costs

When a node stops taking part, the cluster does not merely shrink; it decides. A node that leaves the quorum becomes unknown, and the cluster stops accepting the state changes a lone member might otherwise forge. The requirement of “at least three cluster nodes (to get reliable quorum)” says the same from the other side: two nodes run workloads but cannot guarantee a majority when one disappears. Losing quorum is paid for in service, on purpose. What happens to the workloads afterwards — the restart, the isolation of the failed node, the timeouts — belongs to the companion sheet on high availability and failure handling; the claim here is that a cluster without quorum first refuses to act.

Split brain: the failure the vote exists to prevent

A partition is a network split; a split brain is both sides keeping on serving. If two partitions each ran the same virtual machine against the same shared storage, both would write and the copies would diverge beyond reconciliation — Microsoft warns that a replicated file system “can cause a partition-in-space or partition-in-time, where the cluster nodes are running independently of each other and could cause data loss”. The vote is what makes a partition decidable: only the side that keeps a majority may continue, and the minority must stop rather than become a second cluster. The term also covers a storage-level failure, where replicas on two hosts both take the primary role; that belongs with the sheets on shared and distributed storage, not with the membership vote counted here.

Sizing votes and the witness is a design decision

The rule to keep: quorum protects the integrity of the cluster, not the availability of the service. Every vote and every witness buys a way to survive a failure and costs something — hardware, a network path, a dependency; there is no neutral default. “For smaller 2-node clusters, the QDevice can be used to provide a 3rd vote”, but a three-host cluster already reaches “at least three nodes for reliable quorum” with no witness, and an even cluster with a witness scores no better than an odd one without. The witness must sit where it can outlive the failure it arbitrates — a separate power feed, switch or rack — and it holds no data: a file share witness needs a few megabytes, a cloud witness a single blob, because its only job is to break a tie. Sizing the votes and placing the witness decide which side of a future partition keeps serving — a design decision, not a default.

Level and prerequisites

L4 — membership and the decision, not the recovery: how a cluster counts votes, what a witness adds, and why losing quorum stops the cluster from serving, without developing what happens to workloads afterwards. Prerequisites are the L3 sheet on hosts, virtual machines and their virtual devices, and the L3 storage sheets, on why shared storage lets two hosts run the same workload; the restart after a failure, node isolation and the timeouts are the sibling sheet on high availability and failure handling.

Where to go next

  • Knowledge — the index.
  • Server and virtualization — the area.
  • Virtualization and networking — this sheet’s node.
  • Companion sheets, still in draft: high availability and failure handling, which owns the restart and the node isolation deliberately left out; the storage sheets on shared and distributed storage, which own the storage-level split brain named only here; and affinity and scheduling, which owns placement once the cluster is quorate again.

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.