Shared and distributed storage for virtualization
Block and file storage shared between hosts — and the distributed systems replacing a LUN.

A shared LUN is one block device that several hosts can see, and the host that uses it puts its own filesystem on it; the array and the fabric behind it are common to every client, so their failure is everyone’s failure. A distributed store reverses that: the disks already inside the hosts become one pool, and software on every host decides where each copy lands. The move buys node-level fault tolerance and removes the array as a shared point of failure; it charges for that in placement rules, in a consistency choice and in latency on the write path.
ONE SHARED LUN A DISTRIBUTED POOL
───────────────────────────── ───────────────────────────────
hostA hostB hostC hostA hostB hostC
│ │ │ │ │ │
└──────────┼──────────┘ └──────────┼──────────┘
│ iSCSI / Fibre Channel │ the client asks
│ (multipath: two paths, │ the algorithm; no
│ one LUN) │ broker sits in
▼ ▼ the write path
┌────────────────┐ CRUSH map: host │ rack │ row
│ array = 1 LUN │ o o o o o o
│ one filesystem│ A1 A2 B1 B2 C1 C2
│ one writer │ copies land in different
└────────────────┘ failure domains; a dead
the array and the fabric node re-places its copies
fail for every host at once (rebalance and backfill)
Two ways to stop depending on one LUN
A virtualization cluster needs storage every host can see, and Proxmox VE gives the reason plainly: “One major benefit of storing VMs on shared storage is the ability to live-migrate running machines without any downtime, as all nodes in the cluster have direct access to VM disk images.” A LUN serves that and stays simple for the host, which owns the block device and the filesystem on it; the cost is structural: the array and the fabric are shared by every client, and one volume admits one writer, so a shared volume manager or cluster filesystem must sit on top. A distributed store meets the same need without an array, pooling the hosts’ own disks and computing placement on each host instead of in a controller. The protocol layer is the companion sheet’s ground.
Ceph: an algorithm instead of an array
Ceph is the reference example of the second model. Under every Ceph service is RADOS, “the object store under every Ceph service”, which keeps data available when hardware fails. Proxmox VE consumes it as a block device: “Ceph RADOS is a distributed systems, replicating storage data to different nodes that can be accessed as RBD (RADOS Block Device)”, and that RADOS block device, or RBD, is the direct replacement for the shared LUN. What is different is that no controller decides placement: “CRUSH allows Ceph clients to communicate with OSDs directly rather than through a centralized server or broker”, because each client works out the target from the cluster map and speaks to the object storage daemons (OSDs) that hold the copies.
CRUSH as a map of failure domains
The map is the trick. “The CRUSH algorithm computes storage locations in order to determine how to store and retrieve data”, and it computes them from a description of the cluster: “CRUSH uses a map of the cluster (the CRUSH map) to map data to OSDs, distributing the data across the cluster in accordance with configured replication policy and failure domains.” That map is a hierarchy of buckets — host, rack, row — with rules attached, and the choice that matters is the failure domain: pick host and “each replica of the data is stored on a unique host”, pick rack and the copies land in different racks, so that a failure in one place does not take every copy with it.
Placement groups, replicas and erasure coding
Objects are not placed one by one: a pool holds RADOS objects, and each is assigned to a placement group (PG), the unit CRUSH actually places. On a replicated pool the arithmetic is stated plainly: “the number of OSDs that can fail in parallel without data loss is one less than the number of replicas”, with three copies the usual default and a min_size floor below which I/O stops. Erasure coding buys the same durability for less space: “data is broken into fragments of two kinds: data blocks and parity blocks”. The default profile “can sustain the overlapping loss of two OSDs without losing data” — as durable as three replicas — yet “instead of requiring 3TB to store 1TB, it requires only 2TB to store 1TB”. The price is topology and performance; erasure coding is not “a free lunch”.
DRBD: mirroring two nodes
The other classic answer builds the block device from two hosts instead of an array. DRBD is “a software-based, shared-nothing, replicated storage solution mirroring the content of block devices” between hosts. Its roles are strict: “A DRBD device in the primary role can be used unrestrictedly for read and write operations”, while a secondary “receives all updates from the peer node’s device, but otherwise disallows access completely”. In single-primary mode “a resource is, at any given time, in the primary role on only one cluster member” — the one-writer rule of a LUN, so “this mode can be used with any conventional file system”. Dual-primary mode lets two nodes be primary at once, which “usually requires the use of a shared cluster file system that uses a distributed lock manager”, for live migration.
Split-brain and quorum
Dual-primary is honest about its price. It “requires that the resource is configured to replicate synchronously (protocol C)”, so it is “latency sensitive, and ill-suited for WAN environments”; and because both nodes stay primary, “any interruption in the network between nodes will result in a split-brain”. A split brain is what happens when “both nodes switched to the Primary role while disconnected”, and “two diverging sets of data have been created, which cannot be trivially merged”. The defence is quorum: “DRBD quorum requires at least three nodes”, one of which may be a diskless arbitrator. Membership belongs to the companion sheet on clusters and quorum; without it, the remaining recourses are fencing and manual repair.
Storage Spaces Direct and CSV: the Windows path
Windows reaches the same place differently. Storage Spaces Direct lets a cluster “combine internal storage drives on a cluster of physical servers (2 and up to 16) into a software-defined pool of storage”. Resiliency comes from mirroring and erasure coding: the volumes “typically have resiliency to two simultaneous drive or server failures”, and the data “stays online and accessible even if a drive fails or the entire node goes offline”. The last piece is the Cluster Shared Volume, or CSV: “The CSV file system unifies all the ReFS volumes into a single namespace accessible through any server”, and to each server “every volume looks and acts like it’s mounted locally” — a distributed pool wearing the shape of local storage.
Level and prerequisites
L4 — design-level: which storage model a virtualization cluster should stand on, and what each demands in placement, consistency and topology, rather than how to operate one host’s disks. Prerequisites are the L3 sheets on NAS and SAN and on replication, the earlier sheets on local storage and volumes, and the first sheet of this batch.
Where to go next
- Knowledge — the index.
- Server and virtualization — the area this sheet belongs to.
- Storage — this sheet’s node.
- Companion sheets: local and shared storage for virtualized workloads; clusters and quorum, which owns membership; live migration and its prerequisites, which consume the shared volume; and the L3 sheets on NAS and SAN and on replication.
References
- Ceph — Architecture — https://docs.ceph.com/en/latest/architecture/ — RADOS as the object store under every Ceph service and the cluster’s stated purpose.
- Ceph — Pools — https://docs.ceph.com/en/latest/rados/operations/pools/ — pools, the replica arithmetic, placement groups and the min_size floor.
- Ceph — CRUSH Maps — https://docs.ceph.com/en/latest/rados/operations/crush-map/ — the placement algorithm, the cluster map, buckets, failure domains and CRUSH rules.
- Ceph — Erasure code — https://docs.ceph.com/en/latest/rados/operations/erasure-code/ — data and parity chunks, the default profile, space amplification and failure-domain planning.
- LINBIT — DRBD User Guide 9.0 — https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/ — primary and secondary roles, single- and dual-primary mode, protocol C, split brain and quorum.
- Microsoft — Storage Spaces Direct overview — https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview — the software-defined pool, the software storage bus, resiliency and the Cluster Shared Volume.
- Proxmox — Proxmox VE Administration Guide — https://pve.proxmox.com/pve-docs/pve-admin-guide.html — the reason for shared storage, live migration without downtime, and Ceph RBD as a block backend.