Articles

Shared and distributed storage for virtualization

Block and file storage shared between hosts — and the distributed systems replacing a LUN.

Reading: 5 minServer & Virtualization

Article cover: Shared and distributed storage for virtualization

A shared LUN is one block device that several hosts can see, and the host that uses it puts its own filesystem on it; the array and the fabric behind it are common to every client, so their failure is everyone’s failure. A distributed store reverses that: the disks already inside the hosts become one pool, and software on every host decides where each copy lands. The move buys node-level fault tolerance and removes the array as a shared point of failure; it charges for that in placement rules, in a consistency choice and in latency on the write path.

        ONE SHARED LUN                          A DISTRIBUTED POOL
  ─────────────────────────────           ───────────────────────────────
   hostA      hostB      hostC               hostA      hostB      hostC
     │          │          │                   │          │          │
     └──────────┼──────────┘                   └──────────┼──────────┘
                │  iSCSI / Fibre Channel                 │  the client asks
                │  (multipath: two paths,                │  the algorithm; no
                │   one LUN)                             │  broker sits in
                ▼                                        ▼  the write path
        ┌────────────────┐                    CRUSH map: host │ rack │ row
        │  array = 1 LUN │                       o    o    o    o    o    o
        │  one filesystem│                      A1   A2   B1   B2   C1   C2
        │  one writer    │                    copies land in different
        └────────────────┘                    failure domains; a dead
        the array and the fabric              node re-places its copies
        fail for every host at once           (rebalance and backfill)

Two ways to stop depending on one LUN

A virtualization cluster needs storage every host can see, and Proxmox VE gives the reason plainly: “One major benefit of storing VMs on shared storage is the ability to live-migrate running machines without any downtime, as all nodes in the cluster have direct access to VM disk images.” A LUN serves that and stays simple for the host, which owns the block device and the filesystem on it; the cost is structural: the array and the fabric are shared by every client, and one volume admits one writer, so a shared volume manager or cluster filesystem must sit on top. A distributed store meets the same need without an array, pooling the hosts’ own disks and computing placement on each host instead of in a controller. The protocol layer is the companion sheet’s ground.

Ceph: an algorithm instead of an array

Ceph is the reference example of the second model. Under every Ceph service is RADOS, “the object store under every Ceph service”, which keeps data available when hardware fails. Proxmox VE consumes it as a block device: “Ceph RADOS is a distributed systems, replicating storage data to different nodes that can be accessed as RBD (RADOS Block Device)”, and that RADOS block device, or RBD, is the direct replacement for the shared LUN. What is different is that no controller decides placement: “CRUSH allows Ceph clients to communicate with OSDs directly rather than through a centralized server or broker”, because each client works out the target from the cluster map and speaks to the object storage daemons (OSDs) that hold the copies.

CRUSH as a map of failure domains

The map is the trick. “The CRUSH algorithm computes storage locations in order to determine how to store and retrieve data”, and it computes them from a description of the cluster: “CRUSH uses a map of the cluster (the CRUSH map) to map data to OSDs, distributing the data across the cluster in accordance with configured replication policy and failure domains.” That map is a hierarchy of buckets — host, rack, row — with rules attached, and the choice that matters is the failure domain: pick host and “each replica of the data is stored on a unique host”, pick rack and the copies land in different racks, so that a failure in one place does not take every copy with it.

Placement groups, replicas and erasure coding

Objects are not placed one by one: a pool holds RADOS objects, and each is assigned to a placement group (PG), the unit CRUSH actually places. On a replicated pool the arithmetic is stated plainly: “the number of OSDs that can fail in parallel without data loss is one less than the number of replicas”, with three copies the usual default and a min_size floor below which I/O stops. Erasure coding buys the same durability for less space: “data is broken into fragments of two kinds: data blocks and parity blocks”. The default profile “can sustain the overlapping loss of two OSDs without losing data” — as durable as three replicas — yet “instead of requiring 3TB to store 1TB, it requires only 2TB to store 1TB”. The price is topology and performance; erasure coding is not “a free lunch”.

DRBD: mirroring two nodes

The other classic answer builds the block device from two hosts instead of an array. DRBD is “a software-based, shared-nothing, replicated storage solution mirroring the content of block devices” between hosts. Its roles are strict: “A DRBD device in the primary role can be used unrestrictedly for read and write operations”, while a secondary “receives all updates from the peer node’s device, but otherwise disallows access completely”. In single-primary mode “a resource is, at any given time, in the primary role on only one cluster member” — the one-writer rule of a LUN, so “this mode can be used with any conventional file system”. Dual-primary mode lets two nodes be primary at once, which “usually requires the use of a shared cluster file system that uses a distributed lock manager”, for live migration.

Split-brain and quorum

Dual-primary is honest about its price. It “requires that the resource is configured to replicate synchronously (protocol C)”, so it is “latency sensitive, and ill-suited for WAN environments”; and because both nodes stay primary, “any interruption in the network between nodes will result in a split-brain”. A split brain is what happens when “both nodes switched to the Primary role while disconnected”, and “two diverging sets of data have been created, which cannot be trivially merged”. The defence is quorum: “DRBD quorum requires at least three nodes”, one of which may be a diskless arbitrator. Membership belongs to the companion sheet on clusters and quorum; without it, the remaining recourses are fencing and manual repair.

Storage Spaces Direct and CSV: the Windows path

Windows reaches the same place differently. Storage Spaces Direct lets a cluster “combine internal storage drives on a cluster of physical servers (2 and up to 16) into a software-defined pool of storage”. Resiliency comes from mirroring and erasure coding: the volumes “typically have resiliency to two simultaneous drive or server failures”, and the data “stays online and accessible even if a drive fails or the entire node goes offline”. The last piece is the Cluster Shared Volume, or CSV: “The CSV file system unifies all the ReFS volumes into a single namespace accessible through any server”, and to each server “every volume looks and acts like it’s mounted locally” — a distributed pool wearing the shape of local storage.

Level and prerequisites

L4 — design-level: which storage model a virtualization cluster should stand on, and what each demands in placement, consistency and topology, rather than how to operate one host’s disks. Prerequisites are the L3 sheets on NAS and SAN and on replication, the earlier sheets on local storage and volumes, and the first sheet of this batch.

Where to go next

  • Knowledge — the index.
  • Server and virtualization — the area this sheet belongs to.
  • Storage — this sheet’s node.
  • Companion sheets: local and shared storage for virtualized workloads; clusters and quorum, which owns membership; live migration and its prerequisites, which consume the shared volume; and the L3 sheets on NAS and SAN and on replication.

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.