Articles

PCIe passthrough and IOMMU

Giving a guest a real device: IOMMU groups, isolation limits and what they cost.

Reading: 5 minServer & Virtualization

Article cover: PCIe passthrough and IOMMU

Passthrough hands one guest a real PCIe device instead of a virtual stand-in, so that guest runs the device’s own driver against real hardware and gets its latency, bandwidth and features. It is not a sharing mechanism and not a copy: the device leaves the host and belongs to that one guest. Nor can it simply be handed over, because a PCIe device performs DMA — it reaches host memory on its own — so the host must first constrain where the device may reach. That constraint is the IOMMU, and the unit it isolates is not the device but the IOMMU group.

The model: a real device on the far side of the IOMMU

The chain changes from a virtual device to a real one, with one extra piece of hardware in the middle.

   host side                                        guest side
   ─────────                                        ──────────
   ┌──────────────── IOMMU group 26 ──────────────────┐
   │  ┌───────────────┐        ┌───────────────┐       │
   │  │ GPU 06:0d.0    │        │ HDMI audio    │       │   both functions share
   │  │ (device)      │        │ 06:0d.1       │       │   one isolation boundary
   │  └───────┬───────┘        └───────┬───────┘       │
   └──────────┼────────────────────────┼───────────────┘
              │   DMA + interrupts
      ┌───────▼────────────────────────▼───────┐
      │                 IOMMU                   │   translate I/O address -> physical,
      │   I/O virtual address -> physical       │   and isolate: the group may reach
      └─────────────────┬───────────────────────┘   only the memory it was granted
                        │
                ┌───────▼────────┐
                │   guest VM      │   owns the whole group; the guest's own driver
                │  (real device)  │   talks to the real hardware, not to virtio
                └────────────────┘

The device performs DMA through the IOMMU, which translates the addresses the device issues and walls it off from memory outside its grant. The guest owns the device only as a member of a group: the host checks whether the whole group can be released, not whether that one function can.

Why a device needs the IOMMU before a guest may own it

The kernel is explicit about the risk: “DMA is by far the most critical aspect for maintaining a secure environment as allowing a device read-write access to system memory imposes the greatest risk to the overall system integrity.” The vehicle is VFIO, “an IOMMU/device agnostic framework for exposing direct device access to userspace, in a secure, IOMMU protected environment” — the mechanism behind the I/O performance a guest gains. Proxmox states the outcome directly: “PCI passthrough allows you to use a physical PCI device (graphics card, network card) inside a VM (KVM virtualization only).”

The requirement is hardware: “For PCI(e) passthrough, the CPU and motherboard need to support an input/output memory management unit (IOMMU): Intel VT-d or AMD-Vi on x86, or a System Memory Management Unit (SMMU) on arm64.” Interrupt remapping is part of it — “It is not possible to use PCI passthrough without interrupt remapping.” — and some platforms need unsafe interrupts allowed, which “can make your system unstable”.

The group, not the device, is the unit of isolation

Isolation is meant to be per device, but the kernel admits it cannot always be: “This isolation is not always at the granularity of a single device though.” A bridge without the feature can let traffic bypass the IOMMU — “a non-PCI-ACS (Access Control Services) capable bridge allowing redirection without reaching the IOMMU” — so the kernel works in groups: “The IOMMU API therefore supports a notion of IOMMU groups. A group is a set of devices which is isolatable from all other devices in the system. Groups are therefore the unit of ownership used by VFIO.”

The consequence is that the whole group, not one function, must be freed: each device in the group “will need to be bound to a VFIO driver before operations on the VFIO group are allowed”. The Proxmox guide names the goal — a dedicated IOMMU group for every device assigned to a VM — and the acceptable sharing: “It is okay if the device is in an IOMMU group together with its functions (e.g. a GPU with the HDMI Audio device) or with its root port or PCI(e) bridge.” Anything else the group holds must go to the guest too, which is why the wiki points at ACS: “To have separate IOMMU groups, your processor needs to have support for a feature called ACS (Access Control Services).”

The Windows case: Discrete Device Assignment

Windows Server implements the same idea, with the same dependency, under a different name. “Discrete Device Assignment allows physical Peripheral Component Interconnect Express (PCIe) hardware to be directly accessible from within a virtual machine (VM).” Its requirement names ACS directly: “the PCIe Root Complex has to support Access Control Services (ACS), which enables Hyper-V to force all PCIe traffic through the Input-Output Memory Management Unit.” Only two classes are supported — “For Discrete Device Assignment, Microsoft supports two device classes: Graphics Adapters and NVMe Storage devices.” — and legacy interrupts disqualify a device: “Older devices that use legacy (INTx) PCI Interrupts aren’t supported.” The host needs no driver for the device; the driver is installed in the guest, and because the whole device is passed, all of its capabilities are reachable from inside the guest.

What you give up

The device is gone from the host the moment it is assigned: “the device is not available to the host anymore.” The features that move or capture a machine’s state stop working: while a device is attached, VM Save/Restore, “Live migration of a VM”, “The use of dynamic memory” and “Adding the VM to a high availability (HA) cluster” are unavailable, and Proxmox says it in one sentence: “VMs with passed-through devices cannot be migrated.” The reason is structural: the device is tied to one machine, so a copy of the VM elsewhere has no such device.

There is a security cost too, because DMA is now the guest’s to control: “You should only use Discrete Device Assignment where the tenants of the VMs are trusted.” Firmware updates and any backdoors between a multi-function device’s functions become reachable from the guest; the well-behaved case is the virtual function, the subject of the SR-IOV sheet.

When passthrough is the answer, and when it is not

The gain is the point — “lower latency, higher performance, or more features (e.g., offloading)” — and the common case is “to pass through a whole PCI(e) card, for example a GPU or a network card.” Passthrough is right when one guest genuinely needs one physical device for its whole life: a GPU for a compute workload, or a card whose firmware the guest must own end to end. It is wrong when the device must be shared, moved, snapshotted or failed over. When many guests need a network function rather than one whole card, splitting the device into virtual functions is a different trade — the SR-IOV sheet. And when a guest does not need the real hardware, a paravirtualised device is the better answer: it can be replicated, migrated and snapshotted freely.

Level and prerequisites

L4 — infrastructure design, not a single mechanism: the sheet decides when a real device is the right unit to give a guest and what that decision removes, rather than teaching one layer in isolation.

It presupposes the earlier picture of how a guest sees devices — emulated, paravirtualised and passed-through — and the sheets on hosts and virtual devices. It leads into live migration and its prerequisites, and into clusters and high availability.

Where to go next

Splitting one device among many guests — a network card as a set of virtual functions, its requirements and trade-offs — is the sibling sheet on SR-IOV. The functions passthrough blocks are developed next door: live migration, clustering and quorum, and high availability.

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.