Articles

SR-IOV: purpose, requirements and trade-offs

Splitting one NIC into many virtual functions: what it buys and what it gives up.

Reading: 5 minServer & Virtualization

Article cover: SR-IOV: purpose, requirements and trade-offs

A guest reaches the network through a synthetic adapter and a software switch, and every frame on that path costs host CPU. SR-IOV removes the path for the traffic it carries: it is a PCI Express capability that “makes one physical device appear as multiple virtual devices”, each a real hardware function handed to a guest, so one NIC serves many VMs “with full hardware features and also better performance and lower latency than software-virtualized devices”. That is what SR-IOV buys — throughput without the host’s per-packet work. Whole-device passthrough belongs to the sibling sheet on PCIe passthrough and IOMMU; this one is about the function that is split.

One physical function, many virtual functions

The split is defined at the PCI Express level. “The physical device is referred to as Physical Function (PF) while the virtual devices are referred to as Virtual Functions (VF).” The PF owns the split: allocation “can be dynamically controlled by the PF via registers encapsulated in the capability”. Until a driver enables it, “the PF behaves as traditional PCIe device”.

Each VF is a first-class PCI device, not a synthetic one. It has its own configuration space — “each VF’s PCI configuration space can be accessed by its own Bus, Device and Function Number (Routing ID)” — and “each VF also has PCI Memory Space, which is used to map its register set”. The kernel treats them as ordinary hardware: “The VF is treated as hot-plugged PCI devices in the kernel”, and the guest runs the vendor’s normal driver, because “The VF requires device driver that is same as a normal PCI device”.

   host (hypervisor)                                        guests (VMs)
   ┌───────────────────────────────┐   normal vNIC traffic crosses here:
   │  virtual switch  /  bridge    │   one port per vNIC, forwarded in software
   └───────────────┬───────────────┘
                   │ uplink
   ┌───────────────┴──────────────────────────────────────────────────┐
   │  physical NIC — one card, one PCIe port                           │
   │    ┌──────────┐  PF: Physical Function, owns the card, allocates  │
   │    │   PF     │                                                  │
   │    └────┬─────┘                                                  │
   │   ┌─────┴────┬──────────┬──────────┬──────────┐                  │
   │   │  VF 0    │  VF 1    │  VF 2    │  VF 3    │  VFs: real PCI   │
   │   └────┬─────┴────┬─────┴────┬─────┴────┬─────┘  devices, each   │
   └────────┼──────────┼──────────┼──────────┼────────  to one guest  │
            │          │          │          │   VF traffic does NOT
            ▼          ▼          ▼          ▼   cross the virtual switch
          VM A       VM B       VM C       VM D

The common case is the NIC, which “can provide multiple VFs per physical port”, letting offloads such as checksumming run “inside a VM, reducing the (host) CPU overhead”. Accelerators split too: GPU partitioning over the same interface gives each VM “a dedicated fraction of the GPU instead of the entire GPU”.

Requirements: card, firmware, driver and the VF budget

The card must expose the SR-IOV capability, and the host driver must drive the PF: “If the hardware has SR-IOV capability, loading its PF driver would enable it and all VFs associated with the PF.”

Firmware comes first. “To use SR-IOV, platform support is especially important.” and “It may be necessary to enable this feature in the BIOS/UEFI first, or to use a specific PCI(e) port for it to work.”

The VF count is finite and the kernel enforces the ceiling, keeping “numvfs <= totalvfs”: the card’s limit is the whole budget, to plan rather than exceed.

Isolation is the platform’s other requirement: a VF does DMA into host memory like any device, so the platform must contain it with an IOMMU and interrupt remapping — the sibling sheet’s mechanics. A host without working IOMMU isolation offers no boundary, and on Hyper-V the path is gated by the guest too: “SR-IOV is only available for virtual machines running a supported Windows Server guest”.

Configuring it: host, VFs and guests

The host sets the shape of the split; the guest receives a device.

On Linux there are “two methods for enabling virtual functions on a device.” The modern one writes the count to sysfs per PF, which will “enable and disable the VFs associated with a PCIe PF”. Proxmox gives the one-line form, echo 4 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs, noting “you can change the number of VFs on the fly”.

Per-VF policy is set under the PF. ip link set names the PF first, because “configuring SR-IOV Virtual Function (VF) devices, this keyword should specify the associated Physical Function”, and targets the VF with “vf NUM specify a Virtual Function device to be configured.” Two switches live there: “spoofchk on|off - turn packet spoof checking on or off for the specified VF.” and “trust on|off - trust the specified VF user”, which “enables that VF user can set a specific feature which may impact security and/or performance”. A VF is a device with its own identity, not always trusted.

Assignment is ordinary device binding: Proxmox says to “see them as separate PCI(e) devices when outputting them with lspci” and “pass them through like a normal PCI(e) device”; libvirt attaches them as “SRIOV Virtual Functions (VF) used via type=‘hostdev’ (direct device assignment)”. On Hyper-V it is a switch option: “Allow virtual machine traffic to bypass the virtual machine switch and go directly to the physical NIC.”

The trade-offs: speed bought with visibility

Everything SR-IOV gives, it gives by leaving the software path; the costs come from the same departure.

Visibility. Traffic no longer crosses the virtual switch, so its own controls — port ACLs, QoS, counters and mirroring — do not see what bypasses it. Once frames go “directly to the physical NIC”, plan monitoring and policy outside the software switch.

No live migration. A VF is a function of a card in one host. Since Proxmox passes VFs “like a normal PCI(e) device”, the caveat for such devices applies: “So it is no longer possible to migrate such machines - use with special care.” A workload that must move while running cannot hold a VF throughout; moving is the live migration sheet’s subject.

A finite VF budget. The PF exposes a fixed number of VFs and the kernel refuses to exceed it, so the split cannot be changed without reconfiguring. Proxmox mappings where “it is not important which exact virtual function is passed through” hand out interchangeable VFs, but the pool is still the ceiling.

Isolation depends on the device. The promise is real on hardware built for it: GPU partitioning “provides a hardware-backed security boundary with predictable performance for each VM”. But the boundary is the card’s, not the technology’s: where a VF shares hardware with the PF, the platform exposes trust and spoofing controls precisely because the line is drawn by the silicon. Whether SR-IOV buys a hardware security boundary must be checked per device.

Level and prerequisites

L4 — advanced infrastructure and reliability: this sheet composes an advanced host capability — what it buys, requires and costs — not one mechanism step by step. It presupposes the sibling sheet on PCIe passthrough and IOMMU (whole-device assignment and IOMMU groups), the L3 sheet on the vNIC, bridge and vSwitch whose software data path SR-IOV bypasses, and the lower layers that own the physical NIC, the PCIe topology and the LAN.

Where to go next

Giving a guest an entire device, with the IOMMU groups that govern it, is the passthrough and IOMMU sheet. The software path SR-IOV bypasses is developed in the L3 sheet on the vNIC, bridge and vSwitch; what happens to a VM that must move is the live migration sheet, whose prerequisites SR-IOV vetoes. Two siblings plan the resource a VF consumes: resource allocation and oversubscription, and affinity, anti-affinity and scheduling, for where guests holding a VF are placed. VLAN-aware virtual networking is the other way to carry a per-VM tag.

References

Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.