Distributed systems and failure domains
Failure domains, consensus and partial failure: why a set of hosts is not a big host.

A single server fails whole: it is up or it is down. A distributed system is built from several servers so that no single one is the system, and that promise has a price — it introduces partial failure, where some machines work and some do not. This sheet is about the concepts that make such a system designable: the failure domain as a blast radius, why a set of hosts is not a bigger host, how a group agrees through consensus and quorum, and how the cloud formalises the same boundaries. It does not operate a cluster.
failure domains, nested three copies, three domains
region
+-- zone A copy 1 -> host rack A1 zone A
| +-- rack A1 -- host A1-1 copy 2 -> host rack A2 zone A
| +-- rack A2 -- host A2-1 copy 3 -> host rack B1 zone B
+-- zone B
+-- rack B1 -- host B1-1 lose a host -> two copies answer
lose zone B -> copies 1 and 2 answer
lose the region -> no copy answers
The failure domain is a blast radius
A failure domain is the set of components that fail together — the blast radius of one fault. Independent failures are the exception: a rack shares a power feed and a switch, so it fails as a unit. AWS names the boundaries directly, listing “Availability Zones (AZ), Regions, control planes, and data planes” and saying its infrastructure uses “multiple fault isolation constructs” so customers rely on “the predictable scope of impact containment they provide”. Ceph says the same about data: CRUSH places objects “distributing the data across the cluster in accordance with configured replication policy and failure domains”, so that “the failure of a single host or other kinds of failures will not affect availability”. The domain buys containment; it costs any promise you cannot keep outside it.
A set of hosts is not a bigger host
A host gives you one clock, one memory and one order of events. A group gives you none of them. Each machine measures time on its own clock, so two hosts disagree about when a thing happened. State lives in pieces on separate machines, and copying it takes time during which the copies differ. And there is no natural order to events across hosts; you obtain one only by building it, as Ceph builds a shared picture of the cluster as a hierarchy of “buckets” before it can place anything. A distributed system is not a larger computer but many computers plus the protocols that try to make them behave like one — and the protocols, not the hardware, are most of the difficulty.
Partial failure is the normal condition
A single server is up or it is down. In a group the important case is the ambiguous middle: some nodes running, some stopped, and the unreachable indistinguishable from the merely slow. A partition splits a group while both halves keep running, each certain it is the whole. Because the system cannot always tell what happened, it must choose what to guarantee — a decision, not a default. etcd is “a general substrate for large scale distributed systems. These are systems that will never tolerate split-brain operation and are willing to sacrifice availability to achieve this end”: consistency is kept, availability surrendered. Designing for partial failure means deciding, in advance, which side of that trade each service takes.
Consensus: elect one leader, replicate the log
The oldest way to make many machines act as one is to let one decide. Raft, “an algorithm for managing a replicated log”, gives one server leadership: “Log replication: the leader must accept log entries” and “replicates them on other servers”, and every follower applies them in the same order to an identical state machine. The leader is elected, not assigned — “Raft uses randomized timers to elect” — and safety holds because “Election Safety: at most one leader can be elected in a” term. Agreement is where copies become truth: “Raft guarantees that committed entries are durable” once an entry is replicated on a majority. etcd runs on this idea: a change “only needs to go through the raft protocol to commit”, and each modification gets a revision “from a monotonically increasing counter for reasoning over ordering”.
Quorum: a property, not a product
Quorum is the arithmetic that makes a leader’s decisions decidable: a majority is unique, so two divided halves cannot both hold it, and a group requiring a majority never tolerates “split-brain operation”. That is a property a system is designed to hold, not a box to buy and install, and the same property whether it guards a configuration store or a storage pool. The membership and the votes of a running cluster are the companion sheet on clusters and quorum; here the question is whether the systems you build keep the property under the failures you chose to survive.
The cloud model and its boundaries
The cloud makes the same separation a product. NIST defines cloud computing so that “five essential characteristics, three service models, and four deployment models” compose one model, built on pooling — the provider’s resources “pooled to serve multiple consumers using a multi-tenant model” — and on location independence, in that “the customer generally has no control or knowledge over the exact location of the provided resources”. That is a failure-domain statement: the boundary is real but hidden, so the design must not depend on which side of it a machine sits. AWS answers by exposing its boundaries as an explicit choice, so a workload can be spread across zones and regions on purpose.
How many copies, in how many domains
The design question has three parts. How many copies: enough that losing the number you accept losing still leaves one answering, with the copy count setting the cost of every write. In how many domains: one host survives a disk, one rack survives a host, one zone survives a rack, and CRUSH states it plainly — “each replica of the data is stored on a unique host”, and “distributing replicas across racks is typical for mid- to large-sized clusters”. What you are willing to lose: copies and domains cost money and latency, so the honest answer names a fault — a disk, a host, a rack, a whole site — you accept will take the service down. “Your selection of failure domain should be guided by the size and its CRUSH” topology, and the same holds for any platform. Copies, domains and accepted loss are one decision, better made before the failure than discovered during it.
Level and prerequisites
L5 — the distributed-systems concepts applied to infrastructure: the failure domain, why a set of hosts is not a bigger host, consensus through a leader and a replicated log, quorum as a majority property, and how to size copies against domains. It assumes the L3 sheets on hosts, virtual machines and storage. The membership of a running cluster is the L4 sheet on clusters and quorum, which owns the votes. Distributed storage consistency and the host fleet are sister sheets.
Where to go next
- Knowledge — the index.
- Server and virtualization — the area.
- Virtualization and networking — this sheet’s node.
- Companion sheets, still in draft: clusters and quorum, which owns membership and the votes of a running cluster; distributed storage consistency and availability trade-offs; and multi-host infrastructure, which owns operating a fleet of hosts.
References
- Amazon Web Services — AWS Fault Isolation Boundaries (AWS Whitepaper, 16 November 2022) — https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/abstract-and-introduction.html — the isolation boundaries (Availability Zone, Region, control plane, data plane) and the language of predictable impact containment.
- Ceph Foundation — CRUSH Maps (Ceph documentation) — https://docs.ceph.com/en/latest/rados/operations/crush-map/ — the CRUSH map, the failure-domain concept, and the failure-domain choice for a replicated pool.
- etcd Authors — etcd versus other key-value stores (etcd documentation) — https://etcd.io/docs/latest/learning/why/ — etcd as a consistent key-value substrate, its refusal of split brain, the raft path and the monotonic revision.
- Diego Ongaro and John Ousterhout — In Search of an Understandable Consensus Algorithm (Raft) (USENIX ATC 2014) — https://www.usenix.org/system/files/conference/atc14/atc14-paper-ongaro.pdf — leader election, log replication, and the election-safety and commit rules.
- Peter Mell and Timothy Grance — The NIST Definition of Cloud Computing (NIST SP 800-145, September 2011) — https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf — the cloud model, resource pooling and location independence.