Distributed storage: consistency and availability trade-offs
Replication, quorum reads and writes, and the trade-offs a distributed pool makes.

A distributed pool makes two promises it cannot keep whole at once. Durability: a copy of every object outlives a disk, a host, or a site. Availability: the data answers now. When a link breaks the pool must weaken one. The dials are replica count, minimum write quorum and replication mode, and this sheet is about what each costs.
WRITE ON A HEALTHY POOL WRITE ACROSS A BROKEN LINK
─────────────────────────── ─────────────────────────────
client ─► primary ─► replica zone A zone B
│ ─► replica │ │
│ ─► replica primary replica
▼ (serves) (isolated)
ack after: all replicas (synchronous) │
one replica (asynchronous) ack locally, then
diverge ── or ──
refuse writes until
the link comes back
Consistency and availability are promises about an answer
SNIA names the conflict precisely: the CAP theorem is “A hypothesis that it is impossible for a distributed system to provide Consistency, Availability, and Partition Tolerance guarantees at the same time.” A partition — two halves that can no longer talk — is the failure the pool exists to survive: consistency gives every reader the same answer, availability gives every reader an answer.
The choice is made in the write path. Synchronous replication commits data “at both the primary site and the secondary site before the write is acknowledged to the host”; asynchronous replication commits “at only the primary site”. One buys a current copy and charges a round trip to the farthest replica; the other keeps writes fast and leaves the surviving copy behind. Eventual consistency weakens the promise instead: “over time but does not provide immediate consistency guarantees.”
Ceph: an object store and the map that places it
Under Ceph’s block, file and object services sits RADOS, “the object store under every Ceph service”, which keeps data available when hardware fails. Placement belongs to no controller: CRUSH “computes storage locations in order to determine how to store and retrieve data”, letting clients “communicate with OSDs directly rather than through a centralized server or broker.”
The client computes the target from a shared map. “CRUSH uses a map of the cluster (the CRUSH map) to map data to OSDs, distributing the data across the cluster in accordance with configured replication policy and failure domains.” The word carrying the trade-off is failure domain: choose host and “each replica of the data is stored on a unique host”; choose rack and the copies spread across racks, so a larger domain lets two copies fall together sooner.
Placement groups and the arithmetic of replicas
“Pools are logical partitions that are used to store RADOS objects”, and the unit actually placed is the placement group (PG). Durability is stated as arithmetic: “the number of OSDs that can fail in parallel without data loss is one less than the number of replicas,” with “three replicas (copies) of each RADOS object” as the usual layout.
That number also fixes the floor at which the pool stops serving. The min_size setting “Sets the minimum number of active replicas (or shards) required for PGs to be active and thus for I/O operations to proceed”; below it the write is refused. The same dial governs both ends of the promise: the pool survives more failures but also goes read-only sooner.
Erasure coding and what it costs
Replication is not the only way to buy durability. In erasure coding, “data is broken into fragments of two kinds: data blocks and parity blocks.” The default profile “can sustain the overlapping loss of two OSDs without losing data”, is “equivalent to a replicated pool of size three,” and stores that data in less space: “instead of requiring 3TB to store 1TB, it requires only 2TB to store 1TB.”
What the pool consumes is “space amplification”, the factor “(k+m) / k”: a 4+2 profile uses 1.5 GiB for every 1 GiB of user data, against 3.0 for three replicas. But “Do not mistake erasure coding for a free lunch: there is a significant performance tradeoff, especially when using HDDs and when performing cluster recovery or backfill.” The saving is paid on the write path and during recovery, and most profiles need at least k+m failure domains.
DRBD: primary and secondary, and the quorum that guards them
The other classic answer builds the mirror from two hosts. DRBD is a “shared-nothing, replicated storage solution mirroring the content of block devices” between nodes. Its roles are strict: the primary “can be used unrestrictedly for read and write operations”, the secondary “receives all updates from the peer node’s device, but otherwise disallows access completely.”
In “single-primary mode, a resource is, at any given time, in the primary role on only one cluster member”, so any conventional filesystem fits. Dual-primary mode puts that “primary role on two nodes at a time.”
Dual-primary must “replicate synchronously (protocol C)”, so it is “latency sensitive, and ill-suited for WAN environments”, and with both nodes primary “any interruption in the network between nodes will result in a split-brain” — data that “cannot be trivially merged.” The guard is quorum: “DRBD quorum requires at least three nodes but a third node which acts as an arbitrator can be diskless.”
Stretch clusters: availability in a site versus survival of a site
“A stretch cluster spreads hosts across geographically separated data centers”. In the two-zone layout each zone holds “a copy of the data”, and “A third zone houses a tiebreaker (arbiter, witness) Monitor” that “picks a winner when a network connection between zones fails”.
Stretch mode is deliberately asymmetric. Pools “increase in size from the default 3 to 4”, with “two replicas will be placed at each zone”, so one site alone holds enough to serve; when a data center is lost, “pools’ min_size will be reduced to 1” and the cluster goes active on the survivors. The pool keeps answering from what is left, holding only two copies. Erasure coding is out here: “Erasure-coded pools cannot be used with stretch mode.”
The Windows path, and what a pool charges
Storage Spaces Direct does the same on Windows: it pools a cluster’s internal drives and protects the volumes with “mirroring, erasure coding, or both” — “distributed, software-defined RAID using the drives in the pool.” The volumes “typically have resiliency to two simultaneous drive or server failures”, and a Cluster Shared Volume exposes them through “a single namespace accessible through any server.”
The bill is the same everywhere. Write latency: every synchronous write waits for its slowest required copy. Space: replication multiplies capacity by the replica count, while erasure coding trades some of it back for processor time and slower recovery. And the last cost is not a number — a copy that exists but that no one can locate, mount or rebuild is not a copy at all.
Level and prerequisites
L5 — specialist: the consistency and availability trade-offs a distributed pool makes once it exists. Prerequisites are the L4 sheet on shared and distributed storage, and the L3 sheet on replication and its mistakes.
Where to go next
- Knowledge — the index.
- Server and virtualization — the area this sheet belongs to.
- Storage — this sheet’s node.
Companion sheets: the L4 storage sheet, and the L5 sheets on multi-host infrastructure, on multi-site disaster recovery and on distributed systems and failure domains.
References
- Ceph — Architecture — https://docs.ceph.com/en/latest/architecture/ — RADOS as the object store under every Ceph service.
- Ceph — CRUSH Maps — https://docs.ceph.com/en/latest/rados/operations/crush-map/ — the placement algorithm, the cluster map, the buckets and the failure-domain choice, the primary OSD in the acting set.
- Ceph — Pools — https://docs.ceph.com/en/latest/rados/operations/pools/ — pools as partitions of RADOS objects, the replica arithmetic, the min_size floor.
- Ceph — Erasure code — https://docs.ceph.com/en/latest/rados/operations/erasure-code/ — data and coding chunks, the default profile, the (k+m)/k overhead and the failure-domain requirement.
- Ceph — Stretch Clusters and Stretch Mode — https://docs.ceph.com/en/latest/rados/operations/stretch-mode/ — the two-zone layout, the tiebreaker Monitor, the size-3-to-4 change and degraded stretch mode.
- LINBIT — DRBD User Guide 9.0 — https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/ — primary and secondary roles, single- and dual-primary mode, protocol C, split brain and quorum.
- Microsoft — Storage Spaces Direct overview — https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview — the software-defined pool, resiliency and the Cluster Shared Volume.
- SNIA — SNIA Dictionary — https://www.snia.org/tech_activities/standards/curr_standards/dictionary — the CAP theorem, consistency group, eventual consistency, synchronous and asynchronous replication.