GPU virtualization and AI infrastructure
Sharing accelerators: passthrough, vGPU, MIG and MPS, and what inference serving needs.

An accelerator is not a vCPU: a GPU is bought for its memory capacity, its bandwidth and its matrix throughput, and a card that looks half idle is often unusable by a second job because the first has already claimed the memory. The platform’s choice is narrow: give the whole device to one owner, cut a real partition into it, or let tenants share and accept the consequences. NVIDIA’s own summary is that MIG allows GPUs “to be securely partitioned into up to seven separate GPU Instances for CUDA applications”.
one physical GPU - four ways the platform can hand it out
========================================================
(a) WHOLE DEVICE - passthrough: one guest owns all of it
guest A --------------------------> GPU (entire)
driver + CUDA every SM, every byte of memory
no sharing: guest B waits, or gets a second card
(b) vGPU - the frame buffer is split, the engines are time-sliced
guest A guest B guest C
2 GB fb 2 GB fb 4 GB fb one profile each; their
| | | contexts take turns
+-----------+-----------+
v
+-----------------------------------+
| GPU SMs and engines shared |
+-----------------------------------+
(c) MIG - the hardware itself is cut into instances
+----------+----------+----------+
| GI: 1g | GI: 2g | GI: 3g | each instance has its own
| 1/8 mem | 2/8 mem | 3/8 mem | memory slice and SM slice,
+----------+----------+----------+ with isolated paths
| GPU (memory controllers, L2, DRAM split) |
+---------------------------------------------+
(d) MPS - many processes, one shared context
+----------+----------+----------+
| proc A | proc B | proc C | SM share can be capped,
| 30% SM | 30% SM | 40% SM | but memory, bandwidth and
+----------+----------+----------+ errors are shared
| GPU (one context, one scheduling domain) |
+---------------------------------------------+
The four ways a GPU is handed out
The four mechanisms differ in what gets partitioned. Passthrough gives one guest the whole device. vGPU splits the frame buffer into profiles and schedules their contexts onto shared engines in turns. MIG cuts the hardware into instances with dedicated memory and compute slices. MPS does the opposite: many CUDA processes, one shared context.
Whole-device passthrough: one owner, no sharing
Passthrough is the baseline, and on Windows it has a name: “Discrete Device Assignment allows physical Peripheral Component Interconnect Express (PCIe) hardware to be directly accessible from within a virtual machine (VM)”. The device leaves the host and belongs to the guest; the security boundary is the IOMMU, the kernel’s “VFIO driver is an IOMMU/device agnostic framework for exposing direct device access to userspace, in a secure, IOMMU protected environment”. The requirements are hardware-level: “the PCIe Root Complex has to support Access Control Services (ACS)”, and passthrough is impossible “without interrupt remapping”. Mobility goes with the sharing: DDA lists “Live migration of a VM” among the features it removes, and the Proxmox VE notes agree that “VMs with passed-through devices cannot be migrated”. Exposure follows, so “You should only use Discrete Device Assignment where the tenants of the VMs are trusted”.
vGPU: a partitioned framebuffer, time-sliced
Virtual GPU software keeps a managed middle ground: a hypervisor-side manager that will “configure NVIDIA Virtual GPU manager” and “configure virtual GPU software in pass-through mode”, with scheduling tuned per profile, including for “Time-Sliced vGPUs and MIG-Backed, Time-Sliced vGPUs”. A classic vGPU profile gives the guest a share of the frame buffer and of the engines’ time; a profile too small is a known failure, since “Memory exhaustion can occur with vGPU profiles that have 512 Mbytes or less of frame buffer”. The two compose: MIG lets “multiple vGPUs (and thereby VMs) to run in parallel on a single GPU” “while preserving the isolation guarantees that vGPU provides”.
MIG: a hardware partition with instances and profiles
Multi-Instance GPU is the only mechanism that partitions the hardware rather than the software’s view of it. “A GPU Instance (GI) is a combination of GPU slices and GPU engines (DMAs, NVDECs, and so on)”, built from “A GPU memory slice is the smallest fraction of the GPU’s memory” — roughly one eighth of capacity and bandwidth — and “A GPU SM slice is the smallest fraction of the SMs on the GPU”, roughly one seventh. Because an instance’s paths through the memory system are separate, its owner gets what sharing cannot: a workload that will “run with predictable throughput and latency”, and “a defined quality of service (QoS) with fault isolation”. Sizes are not free — “The number of slices that a GI can be created with is not arbitrary” — so the driver offers fixed profiles (1g.5gb, 2g.10gb, 3g.20gb, and so on). The limits are worth naming: “Toggling MIG mode requires the CAP_SYS_ADMIN capability”; “No graphics APIs are supported”; “NCCL is currently not supported with MIG”; and reconfiguration waits for idle. MPS may still run inside an instance — “CUDA MPS is supported on top of MIG”.
MPS: sharing a context, and the cost of isolation
MPS is not a partition at all: a lightweight runtime service whose payoff is “improved GPU utilization and reduced GPU context switching”. It can cap a client’s share — “MPS also provides memory and SM partitioning capabilities” — but those caps are fractions of one scheduling domain, not walls: “MPS currently does not offer error isolation between clients”, and “the scheduling hardware is still shared”. Memory bandwidth, caches and capacity are shared too. The vendor’s comparison makes the trade-off legible:
| Mechanism | Partition type | Max partitions | Error isolation | Memory-bandwidth QoS |
|---|---|---|---|---|
| CUDA streams | single process | unlimited | no | no |
| MPS | logical | 48 | no | no |
| MIG | physical | 7 | yes | yes |
A neighbour that saturates its DRAM interface slows every MPS client; under MIG the guide’s point is that this cannot happen.
What an inference service asks of the platform
An inference server turns the sharing question concrete, because it consumes exactly what a GPU partitions. Device memory is the binding constraint: an LLM’s attention key and value tensors live in GPU memory, and a serving engine’s central trick is “Efficient management of attention key and value memory with PagedAttention”. Throughput and latency are the two figures an operator trades; one engine’s selling point is “State-of-the-art serving throughput”, obtained partly by “Continuous batching of incoming requests, chunked prefill, prefix caching”. A model often will not fit on one device, so the server spans several: “Tensor, pipeline, data, expert, and context parallelism for distributed inference”, or splits the phases so that “Disaggregated prefill, decode, and encode” land on different hardware. The consequence: tensor parallelism wants GPU-to-GPU memory bandwidth, and a request-level scheduler is happier with many small, well-isolated slices than with one large card a tenant has saturated. The models themselves are the AI and LLM macro-area’s subject, not this sheet’s.
Rigid partition or opportunistic sharing
The question this sheet has been building toward is single: given that the GPU is the scarcest resource, is it better to partition it rigidly or to share it opportunistically? MIG is the rigid answer, buying predictable throughput, memory-bandwidth QoS and fault isolation at the cost of fixed shapes, a fragmentation tax as instances are created and destroyed, an idle-instance problem, and reconfiguration that waits for idle. MPS and time-slicing are the opportunistic answer, buying utilisation and simplicity at the cost of isolation: a crash or a bandwidth hog in one client is felt by the others. The two failure modes are genuinely different, which is why this is a design decision, not a tuning parameter: a rigid partition fails by waste (the wrong profile, capacity stranded between shapes), an opportunistic share fails by interference (one tenant’s burst becoming every tenant’s latency spike, with no fault domain to blame).
Level and prerequisites
L5 — design of a shared, specialised infrastructure: it compares four ways to hand one accelerator to several consumers and closes on the choice between a rigid partition and an opportunistic share, without repeating the passthrough and SR-IOV mechanics. Prerequisites: the L4 sheets on PCIe passthrough and IOMMU and on SR-IOV, the L3 sheet on hosts, VMs, vCPU, vRAM and virtual devices, and the L4 sheet on container images.
Where to go next
- Knowledge — the index.
- Server and virtualization — the area.
- Virtualization and networking — this node.
- Companion sheets, still in draft: PCIe passthrough and IOMMU; SR-IOV; resource allocation and oversubscription; and platform design: VMs, containers and specialised workloads.
References
- NVIDIA — Multi-Instance GPU User Guide (Release 615) —
nv-mig-guide.txt— MIG as hardware partitioning into GPU Instances and Compute Instances, the slice model, the instance profiles, the CUDA concurrency comparison, and the deployment limits. - NVIDIA — Virtual GPU Software User Guide — https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html — vGPU as a managed middle ground, time-sliced and MIG-backed vGPUs, and the frame-buffer profile limits.
- NVIDIA — Multi-Process Service — https://docs.nvidia.com/deploy/mps/index.html — MPS as a lightweight runtime service, its utilisation benefit, and its optional memory and SM partitioning.
- Linux kernel — VFIO — “Virtual Function I/O” —
k-vfio.txt— VFIO as an IOMMU-protected framework for direct device access, and the IOMMU group as the unit of ownership. - Microsoft — Deploy devices by using Discrete Device Assignment —
ms-dda.txt— whole-device passthrough, the ACS and IOMMU requirement, the VM features it removes, and the trusted-tenant condition. - Proxmox VE — PCI Passthrough — https://pve.proxmox.com/wiki/PCI_Passthrough — interrupt remapping as a hard requirement and the statement that passed-through VMs cannot be migrated.
- vLLM — Welcome to vLLM — https://docs.vllm.ai/en/stable/ — the serving engine’s framing of the demand: memory as the constraint, throughput and latency, and parallelism and disaggregation across GPUs.