Articles

Distributed systems and failure domains

Failure domains, consensus and partial failure: why a set of hosts is not a big host.

Reading: 5 minServer & Virtualization

Article cover: Distributed systems and failure domains

A single server fails whole: it is up or it is down. A distributed system is built from several servers so that no single one is the system, and that promise has a price — it introduces partial failure, where some machines work and some do not. This sheet is about the concepts that make such a system designable: the failure domain as a blast radius, why a set of hosts is not a bigger host, how a group agrees through consensus and quorum, and how the cloud formalises the same boundaries. It does not operate a cluster.

  failure domains, nested                    three copies, three domains

  region
   +-- zone A                                copy 1  ->  host  rack A1  zone A
   |    +-- rack A1 -- host A1-1             copy 2  ->  host  rack A2  zone A
   |    +-- rack A2 -- host A2-1             copy 3  ->  host  rack B1  zone B
   +-- zone B
        +-- rack B1 -- host B1-1        lose a host     -> two copies answer
                                        lose zone B     -> copies 1 and 2 answer
                                        lose the region -> no copy answers

The failure domain is a blast radius

A failure domain is the set of components that fail together — the blast radius of one fault. Independent failures are the exception: a rack shares a power feed and a switch, so it fails as a unit. AWS names the boundaries directly, listing “Availability Zones (AZ), Regions, control planes, and data planes” and saying its infrastructure uses “multiple fault isolation constructs” so customers rely on “the predictable scope of impact containment they provide”. Ceph says the same about data: CRUSH places objects “distributing the data across the cluster in accordance with configured replication policy and failure domains”, so that “the failure of a single host or other kinds of failures will not affect availability”. The domain buys containment; it costs any promise you cannot keep outside it.

A set of hosts is not a bigger host

A host gives you one clock, one memory and one order of events. A group gives you none of them. Each machine measures time on its own clock, so two hosts disagree about when a thing happened. State lives in pieces on separate machines, and copying it takes time during which the copies differ. And there is no natural order to events across hosts; you obtain one only by building it, as Ceph builds a shared picture of the cluster as a hierarchy of “buckets” before it can place anything. A distributed system is not a larger computer but many computers plus the protocols that try to make them behave like one — and the protocols, not the hardware, are most of the difficulty.

Partial failure is the normal condition

A single server is up or it is down. In a group the important case is the ambiguous middle: some nodes running, some stopped, and the unreachable indistinguishable from the merely slow. A partition splits a group while both halves keep running, each certain it is the whole. Because the system cannot always tell what happened, it must choose what to guarantee — a decision, not a default. etcd is “a general substrate for large scale distributed systems. These are systems that will never tolerate split-brain operation and are willing to sacrifice availability to achieve this end”: consistency is kept, availability surrendered. Designing for partial failure means deciding, in advance, which side of that trade each service takes.

Consensus: elect one leader, replicate the log

The oldest way to make many machines act as one is to let one decide. Raft, “an algorithm for managing a replicated log”, gives one server leadership: “Log replication: the leader must accept log entries” and “replicates them on other servers”, and every follower applies them in the same order to an identical state machine. The leader is elected, not assigned — “Raft uses randomized timers to elect” — and safety holds because “Election Safety: at most one leader can be elected in a” term. Agreement is where copies become truth: “Raft guarantees that committed entries are durable” once an entry is replicated on a majority. etcd runs on this idea: a change “only needs to go through the raft protocol to commit”, and each modification gets a revision “from a monotonically increasing counter for reasoning over ordering”.

Quorum: a property, not a product

Quorum is the arithmetic that makes a leader’s decisions decidable: a majority is unique, so two divided halves cannot both hold it, and a group requiring a majority never tolerates “split-brain operation”. That is a property a system is designed to hold, not a box to buy and install, and the same property whether it guards a configuration store or a storage pool. The membership and the votes of a running cluster are the companion sheet on clusters and quorum; here the question is whether the systems you build keep the property under the failures you chose to survive.

The cloud model and its boundaries

The cloud makes the same separation a product. NIST defines cloud computing so that “five essential characteristics, three service models, and four deployment models” compose one model, built on pooling — the provider’s resources “pooled to serve multiple consumers using a multi-tenant model” — and on location independence, in that “the customer generally has no control or knowledge over the exact location of the provided resources”. That is a failure-domain statement: the boundary is real but hidden, so the design must not depend on which side of it a machine sits. AWS answers by exposing its boundaries as an explicit choice, so a workload can be spread across zones and regions on purpose.

How many copies, in how many domains

The design question has three parts. How many copies: enough that losing the number you accept losing still leaves one answering, with the copy count setting the cost of every write. In how many domains: one host survives a disk, one rack survives a host, one zone survives a rack, and CRUSH states it plainly — “each replica of the data is stored on a unique host”, and “distributing replicas across racks is typical for mid- to large-sized clusters”. What you are willing to lose: copies and domains cost money and latency, so the honest answer names a fault — a disk, a host, a rack, a whole site — you accept will take the service down. “Your selection of failure domain should be guided by the size and its CRUSH” topology, and the same holds for any platform. Copies, domains and accepted loss are one decision, better made before the failure than discovered during it.

Level and prerequisites

L5 — the distributed-systems concepts applied to infrastructure: the failure domain, why a set of hosts is not a bigger host, consensus through a leader and a replicated log, quorum as a majority property, and how to size copies against domains. It assumes the L3 sheets on hosts, virtual machines and storage. The membership of a running cluster is the L4 sheet on clusters and quorum, which owns the votes. Distributed storage consistency and the host fleet are sister sheets.

Where to go next

  • Knowledge — the index.
  • Server and virtualization — the area.
  • Virtualization and networking — this sheet’s node.
  • Companion sheets, still in draft: clusters and quorum, which owns membership and the votes of a running cluster; distributed storage consistency and availability trade-offs; and multi-host infrastructure, which owns operating a fleet of hosts.

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.