Articles

Distributed storage: consistency and availability trade-offs

Replication, quorum reads and writes, and the trade-offs a distributed pool makes.

Reading: 5 minServer & Virtualization

Article cover: Distributed storage: consistency and availability trade-offs

A distributed pool makes two promises it cannot keep whole at once. Durability: a copy of every object outlives a disk, a host, or a site. Availability: the data answers now. When a link breaks the pool must weaken one. The dials are replica count, minimum write quorum and replication mode, and this sheet is about what each costs.

   WRITE ON A HEALTHY POOL                    WRITE ACROSS A BROKEN LINK
  ───────────────────────────                ─────────────────────────────
   client ─► primary ─► replica                     zone A        zone B
                │   ─► replica                        │              │
                │   ─► replica                     primary        replica
                ▼                                  (serves)      (isolated)
   ack after:  all replicas   (synchronous)           │
               one replica    (asynchronous)     ack locally, then
                                                 diverge  ── or ──
                                                 refuse writes until
                                                 the link comes back

Consistency and availability are promises about an answer

SNIA names the conflict precisely: the CAP theorem is “A hypothesis that it is impossible for a distributed system to provide Consistency, Availability, and Partition Tolerance guarantees at the same time.” A partition — two halves that can no longer talk — is the failure the pool exists to survive: consistency gives every reader the same answer, availability gives every reader an answer.

The choice is made in the write path. Synchronous replication commits data “at both the primary site and the secondary site before the write is acknowledged to the host”; asynchronous replication commits “at only the primary site”. One buys a current copy and charges a round trip to the farthest replica; the other keeps writes fast and leaves the surviving copy behind. Eventual consistency weakens the promise instead: “over time but does not provide immediate consistency guarantees.”

Ceph: an object store and the map that places it

Under Ceph’s block, file and object services sits RADOS, “the object store under every Ceph service”, which keeps data available when hardware fails. Placement belongs to no controller: CRUSH “computes storage locations in order to determine how to store and retrieve data”, letting clients “communicate with OSDs directly rather than through a centralized server or broker.”

The client computes the target from a shared map. “CRUSH uses a map of the cluster (the CRUSH map) to map data to OSDs, distributing the data across the cluster in accordance with configured replication policy and failure domains.” The word carrying the trade-off is failure domain: choose host and “each replica of the data is stored on a unique host”; choose rack and the copies spread across racks, so a larger domain lets two copies fall together sooner.

Placement groups and the arithmetic of replicas

“Pools are logical partitions that are used to store RADOS objects”, and the unit actually placed is the placement group (PG). Durability is stated as arithmetic: “the number of OSDs that can fail in parallel without data loss is one less than the number of replicas,” with “three replicas (copies) of each RADOS object” as the usual layout.

That number also fixes the floor at which the pool stops serving. The min_size setting “Sets the minimum number of active replicas (or shards) required for PGs to be active and thus for I/O operations to proceed”; below it the write is refused. The same dial governs both ends of the promise: the pool survives more failures but also goes read-only sooner.

Erasure coding and what it costs

Replication is not the only way to buy durability. In erasure coding, “data is broken into fragments of two kinds: data blocks and parity blocks.” The default profile “can sustain the overlapping loss of two OSDs without losing data”, is “equivalent to a replicated pool of size three,” and stores that data in less space: “instead of requiring 3TB to store 1TB, it requires only 2TB to store 1TB.”

What the pool consumes is “space amplification”, the factor “(k+m) / k”: a 4+2 profile uses 1.5 GiB for every 1 GiB of user data, against 3.0 for three replicas. But “Do not mistake erasure coding for a free lunch: there is a significant performance tradeoff, especially when using HDDs and when performing cluster recovery or backfill.” The saving is paid on the write path and during recovery, and most profiles need at least k+m failure domains.

DRBD: primary and secondary, and the quorum that guards them

The other classic answer builds the mirror from two hosts. DRBD is a “shared-nothing, replicated storage solution mirroring the content of block devices” between nodes. Its roles are strict: the primary “can be used unrestrictedly for read and write operations”, the secondary “receives all updates from the peer node’s device, but otherwise disallows access completely.”

In “single-primary mode, a resource is, at any given time, in the primary role on only one cluster member”, so any conventional filesystem fits. Dual-primary mode puts that “primary role on two nodes at a time.”

Dual-primary must “replicate synchronously (protocol C)”, so it is “latency sensitive, and ill-suited for WAN environments”, and with both nodes primary “any interruption in the network between nodes will result in a split-brain” — data that “cannot be trivially merged.” The guard is quorum: “DRBD quorum requires at least three nodes but a third node which acts as an arbitrator can be diskless.”

Stretch clusters: availability in a site versus survival of a site

“A stretch cluster spreads hosts across geographically separated data centers”. In the two-zone layout each zone holds “a copy of the data”, and “A third zone houses a tiebreaker (arbiter, witness) Monitor” that “picks a winner when a network connection between zones fails”.

Stretch mode is deliberately asymmetric. Pools “increase in size from the default 3 to 4”, with “two replicas will be placed at each zone”, so one site alone holds enough to serve; when a data center is lost, “pools’ min_size will be reduced to 1” and the cluster goes active on the survivors. The pool keeps answering from what is left, holding only two copies. Erasure coding is out here: “Erasure-coded pools cannot be used with stretch mode.”

The Windows path, and what a pool charges

Storage Spaces Direct does the same on Windows: it pools a cluster’s internal drives and protects the volumes with “mirroring, erasure coding, or both” — “distributed, software-defined RAID using the drives in the pool.” The volumes “typically have resiliency to two simultaneous drive or server failures”, and a Cluster Shared Volume exposes them through “a single namespace accessible through any server.”

The bill is the same everywhere. Write latency: every synchronous write waits for its slowest required copy. Space: replication multiplies capacity by the replica count, while erasure coding trades some of it back for processor time and slower recovery. And the last cost is not a number — a copy that exists but that no one can locate, mount or rebuild is not a copy at all.

Level and prerequisites

L5 — specialist: the consistency and availability trade-offs a distributed pool makes once it exists. Prerequisites are the L4 sheet on shared and distributed storage, and the L3 sheet on replication and its mistakes.

Where to go next

Companion sheets: the L4 storage sheet, and the L5 sheets on multi-host infrastructure, on multi-site disaster recovery and on distributed systems and failure domains.

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.