Articles

Local and shared storage for virtualized workloads

Local disks and shared storage seen from the hypervisor: what each gives a workload.

Reading: 5 minServer & Virtualization

Article cover: Local and shared storage for virtualized workloads

Every virtual machine has at least one disk, and the guest sees only a block device. Where the bytes live is the datastore, and on a cluster the datastore decides which cluster functions are even possible. A hypervisor stores a VM disk in one of several shapes, and libvirt’s domain format names them: “Representation of the disk source depends on the disk type attribute value as follows” — a file (“The file attribute specifies the fully-qualified path to the file”), a host device (“The dev attribute specifies the fully-qualified path to the host”), or a network address, where “The protocol attribute specifies the protocol to access to the” image. The guest sees a SCSI or a virtio device either way. Proxmox VE states the split plainly: VM images “can either be stored on one or several local storages, or on shared storage like NFS or iSCSI (NAS, SAN)”. This sheet is about the guarantees a datastore must give a virtualized workload — and why a local disk cannot give the shared ones.

What a datastore has to guarantee

Five properties decide what a datastore can do for a cluster: who can see the bytes, who may write them, what the write path costs, how large the allocation unit is, and where capacity comes from. They are easiest to read side by side.

GUEST            one block device — the guest never knows where the bytes are
                      │
HYPERVISOR   <disk type='file'|'block'|'network'>  ──▶  the datastore
                      │
                   LOCAL                        SHARED
  visible to       one host                     every host in the cluster
  locking          a pointer to that host       cluster-wide lock (shared=1)
  host fails       the disk goes with it        bytes stay; HA can restart
  HA / migration   no / copy the whole image    yes / no copy, so faster
  I/O path         the host's own disk          host stack + network
  capacity         that one device              a pool several hosts consume

  A local disk is not a worse datastore — it is a datastore whose scope is
  one host, and that scope is exactly what forbids HA, migration and
  maintenance without an outage.

Shared visibility: one set of bytes

The first guarantee is that every host sees the same content. A cluster filesystem or a distributed pool provides it: in Storage Spaces Direct “all the servers can see all of each other’s local drives”, and the Cluster Shared Volume layer “unifies all the ReFS volumes into a single namespace accessible through any server. To each server, every volume looks and acts like it’s mounted locally.” Proxmox VE keeps one configuration for every node, because “Sharing storage configuration makes perfect sense for shared storage, because the same “shared” storage is accessible from all nodes.”

But the configuration is not the storage: the shared flag only “marks an already shared storage as such”, and a local type kept in a shared configuration “is available on all nodes, but it is physically different and can have totally different content.” Two nodes can then carry the same storage id and different bytes — the trap behind a guest that starts on the wrong host.

Locking: one writer, and who arbitrates

Visibility is not coordination. When several hosts can write the same blocks, something must decide who may. A file-level datastore arbitrates on the server — “Some storage operation need exclusive access to the storage, so proper locking is required” — and that locking “does not work between different clusters.” A block datastore must be told it is shared: “The backend implements proper cluster-wide locking if the storage is marked as shared in the configuration.” Without a shared locking layer the hypervisor falls back on the guest’s owner, keeping its configuration on one node so that “This concept enables the usage of local locks instead of expensive cluster-wide locks for preventing concurrent guest configuration changes.” The lock is cheap because it is local — and that is why it fails together with the node.

The write path and its latency

The third guarantee is where the bytes travel. On the local path a virtual disk is served by the host’s own storage stack, close to the media. On the shared path every I/O crosses the network to the node that owns the media, which is why the fabric can be ordinary Ethernet, replacing, in Microsoft’s words, “costly and restrictive Fibre Channel or Shared SAS cabling”. Latency is then the sum of more hops — the same media can be slower to a guest — and a datastore that stops answering stalls a guest whose I/O is synchronous with its disk.

The allocation unit and the declared size

A datastore hands out space in units. On block media “Disk devices typically store data in fixed size blocks (e.g., 512 byte blocks, or 4,096 byte blocks)”. The hypervisor adds its own unit: a modern virtual hard disk is created with “the optimal 4 KB alignment”, and the guidance is to “match block size to the allocation patterns of the workloads using the disk”, because a bigger block “increases the amount of space the VHD uses on the host”. The declared size of a disk is not the space the datastore commits; that gap is the sibling sheet Virtual disks: thin and thick provisioning. What matters is that the datastore sizes its unit and its capacity for many guests at once, because when it runs out the failure is shared: “If a storage runs full, all guests using volumes on that storage receive IO errors.”

Why a local disk rules out HA, migration and maintenance

A local disk is visible to one host only, so nothing that needs the disk to outlive that host is available. Microsoft states it directly: “Standalone Hyper-V with local disks doesn’t support automatic failover of virtual machine (VM) workloads due to the lack of shared storage”. With shared storage the same workload “can move VMs to any node in the cluster using Live Migrate or automatically restart after a failure (failover)”. Proxmox VE gives the reason to share: it allows one to “live-migrate running machines without any downtime, as all nodes in the cluster have direct access to VM disk images”, and “There is no need to copy VM image data, so live migration is very fast in that case.” Maintenance is the same property: draining a host before a reboot is live migration repeated, and with a local disk the honest options are an outage or a copy of the whole image. Even with shared disks, after the owning node fails a regular migration is impossible “because such a local lock on the (offline) owning node is unobtainable”, and guests “with local disks (or other local resources which are only available on the offline node) are not recoverable like this.” Restoring it needs a distributed lock service and an HA stack, which belong to the cluster sheets.

Level and prerequisites

L4 — infrastructure and reliability: what a datastore must guarantee a virtualized workload, and which cluster functions a local disk forbids. Prerequisites: the L3 sheets NAS and SAN and Virtual disks: thin and thick provisioning, and the L2 sheet Local storage. The distributed systems that replace a shared LUN are the sibling sheet Shared and distributed storage.

Where to go next

  • Server & Virtualization — the area this sheet belongs to, whose Storage node is storage.
  • The sibling sheet Shared and distributed storage.
  • The cluster sheets Clusters and quorum, High availability and failure handling, and Live migration and its prerequisites.
  • Capacity planning and host maintenance, where shared storage turns a host reboot into a migration.

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.