Articles

Resource allocation and oversubscription

What the hypervisor promises, what it only time-shares, and where overcommit hurts.

Reading: 5 minServer & Virtualization

Article cover: Resource allocation and oversubscription

A virtual machine is promised resources it does not own. The hypervisor keeps the CPUs and the memory and hands each guest a share; the question that decides whether a host is safe is which shares are guarantees and which are only time. One of them cannot be promised twice.

Guarantees, limits and time

The control-group documentation names four ways to divide a resource. A weight distributes by ratio and is “used for stateless resources”; a limit caps consumption, and “Limits can be over-committed - the sum of the limits of children can exceed the amount of resource available to the parent”; a protection promises a floor, and “Protections can also be over-committed”. Only an allocation is exclusive: “Allocations can’t be over-committed - the sum of the allocations of children can not exceed the amount of resource available to the parent”. CPU and memory fit the first three; storage fits the last.

model what it gives over-committable
weight a proportional share, work-conserving yes
limit a ceiling on consumption yes
protection a floor, hard or best-effort yes
allocation exclusive use of a finite amount no

Memory: the size the guest is told, the pages the host backs

Memory is declared as a size; the size is a target. libvirt distinguishes <memory>, the “maximum allocation of memory for the guest at boot time”, from <currentMemory>, “The actual allocation of memory for the guest”, a value that “can be less than the maximum allocation” so ballooning has room. Nothing in that pair forces the host to hold the pages, and reading the declared size as a reservation rather than a target is the common error.

Memory tuning states the promise: a hard_limit, “the maximum memory the guest can use”; a soft_limit, “the memory limit to enforce during memory contention”; and a min_guarantee, “the guaranteed minimum memory allocation for the guest”. Only the third is a reservation, and it “is only supported by VMware ESX and OpenVZ drivers”. A guest that locks all its memory “could cause a denial-of-service attack on the host”, which is why hard_limit is recommended alongside it.

Ballooning trades the declared size against the real one. Proxmox VE offers “a fixed size memory or asking Proxmox VE to dynamically allocate memory based on the current RAM usage of the host”; with the minimum below the maximum it “will make sure that the minimum amount you specified is always available to the VM”, using a driver that will “grab or release memory pages from the host”. Hyper-V’s Dynamic Memory adds startup, minimum and maximum sizes — the minimum being “the minimum amount allocated after startup and allows Hyper-V to reclaim the unused memory” — a “Memory buffer is specified as a percentage”, and a weight that decides how memory is distributed on shortfall.

CPU: weight, cap, and the cores nobody reserved

CPU is shared more casually because it is a schedule, not a store. The control-group cpu.weight “proportionally distributes CPU cycles to active children”, defaulting to 100; Proxmox’s cpuunits is the same dial, “a relative weight which defaults to 100” that decides who waits when demand exceeds supply, and reserves nothing.

The cap is separate: cpulimit sets the VM’s total “Host CPU Time”, a percentage in which 1.0 is one core fully used, and cpu.max enforces the same ceiling. Because both are over-committable, a host may run more virtual CPUs than cores: “It is perfectly safe if the overall number of cores of all your VMs is greater than the number of cores on the server”, because “the host system will balance the QEMU execution threads between your server cores”. The only guard is at boot — the platform “will prevent you from starting VMs with more virtual CPU cores than physically available” — so a busy neighbour turns a vCPU into latency the guest cannot even see.

Deduplication: KSM makes the same promise cheaper

Oversubscribing memory is cheaper when identical pages collapse. Kernel Samepage Merging is “a memory-saving de-duplication feature” that “was originally developed for use with KVM (where it was known as Kernel Shared Memory), to fit more virtual machines into physical memory” by scanning for “pages of identical content which can be replaced by a single write-protected page”.

The savings have edges: KSM “only merges anonymous (private) pages, never pagecache (file) pages”, so a file-cache-heavy guest recovers little; the scan “may use a lot of processing power: some installations will disable KSM for that reason”. The sharpest edge is on return: cancelling the advice on a range “may suddenly require more memory than is available”, and it will more probably arouse the out-of-memory (OOM) killer.

Where oversubscription becomes an incident

Memory. A control group’s memory.max is the hard limit — “If a cgroup’s memory usage reaches this limit and can’t be reduced, the OOM killer is invoked in the cgroup” — while memory.high throttles: “If a cgroup’s usage goes over the high boundary, the processes of the cgroup are throttled and put under heavy reclaim pressure”. Because “Going over the high limit never invokes the OOM killer”, high limits are the safe ones to over-commit, whereas over-protecting with memory.min “may lead to constant OOMs”. Under pressure the layers retreat: the balloon inflates, the guest swaps, and then “When the host is running low on RAM, the VM will then release some memory back to the host, swapping running processes if needed and starting the oom killer in last resort”. A Hyper-V restart falls back on Smart Paging, which “can degrade virtual machine performance because disk access speeds are much slower than memory access speeds”; guest-side paging is preferred — “When you oversubscribe host memory, Hyper-V continues to rely on the paging operation in the guest operating system” — and the Windows balloon driver is not recommended “on critical systems”.

CPU. Nothing faults; a low-weight VM just runs less.

Storage. Storage cannot rehearse oversubscription: a written block cannot be reclaimed like a page. Thin provisioning lets images exceed the space on hand precisely because unused blocks are never written, but the pool is finite and the failure is not graceful: “If a storage runs full, all guests using volumes on that storage receive IO errors”. Memory pressure degrades; storage pressure corrupts. That is why the memory rule — “leave 1GB of RAM available to the host” — becomes, for storage, a rule against over-provisioning at all.

  declared to the guest        backed by the host         under pressure
  -------------------------------------------------------------------------------
  vCPU N            ------>   N host threads     ------>  a weight decides who waits;
                               (cpu.weight)                no core is reserved

  vRAM size         ------>   real pages         ------>  balloon inflates, guest
  (memory)                     (currentMemory,             swaps, OOM killer fires
                                min_guarantee)

  disk / LUN        ------>   fixed storage      ------>  no reclaim and no second
                               (cpu.max caps a             promise: the bytes stay lost
                                share only)

Level and prerequisites

L4 — the policy of handing a machine more than it owns: which resources the hypervisor can promise twice, and how each fails. Prerequisites are the L3 sheet on hosts, VMs, vCPU (virtual CPU), vRAM (virtual memory) and virtual devices, then CPU cores, threads and memory pressure at L1. This sheet answers how much may be handed out; where it lands — pinning, NUMA, memory policy that “determines from which node the kernel will allocate memory in a NUMA system” — is the affinity and scheduling sheet.

Where to go next

  • Server & Virtualization — the area this sheet belongs to.
  • Affinity, anti-affinity and scheduling is the where to this sheet’s how much; capacity planning and host maintenance measures the headroom left; local and shared storage for virtualization covers thin and thick provisioning.

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.