NICs, links and throughput: where a network connection''s real speed is decided

A 10 Gb Ethernet link carrying 4 Gb/s of application traffic is not necessarily broken. The line rate is one number; the throughput a workload actually observes is the result of the medium, the size of the frames, the adapter’s queues, and the host’s ability to feed and drain them. A NIC is the boundary where those two worlds meet, so it is also the place where “the network is slow” has to be diagnosed.

The model: host memory on one side, a link on the other

application → socket → kernel network stack
                              ↓
                     adapter (NIC)
        ┌─────────────────────┴──────────────────────┐
   host side                                    link side
   queues, interrupts, offloads, RSS        negotiated speed and flow control,
   (what the CPU must do per packet)        medium: copper / fiber / DAC,
   ────────────────────────────             frame size: MTU, jumbo frames
                              ↓
                        switch / peer

Two independent questions live in that picture: what the link can carry (medium and negotiated parameters) and what the host can move (queues, offloads, CPU). Throughput is the smaller of the two, minus overhead.

Terms used here

NIC — network interface controller: the adapter that connects the host to the link.
Link speed — the rate both ends agree on, negotiated with the peer rather than configured blind.
Autonegotiation — the process by which the two ends agree; a mismatch produces the famous “we are connected but everything is slow” state.
MTU — the largest frame payload allowed on the path; jumbo frames raise it.
Offload — work the adapter does instead of the CPU (checksum offload, large send offload).
RSS — Receive Side Scaling: spreading received traffic across several queues so several CPUs can process it.
Link aggregation (bonding/teaming) — several physical links presented as one logical link.
Counters — the adapter’s statistics: dropped, FIFO overflows, CRC errors.

The link: speed is negotiated, and the medium is modular

Link settings are not purely local decisions. Kernel driver documentation for Intel’s 10 Gb adapters, for example, exposes the speed setting through ethtool and warns that mixing speed capabilities or misconfiguring the setting produces wrong results — the reason autonegotiation is the default — and the same documentation exposes Ethernet flow control (IEEE 802.3x) the same way, with one explicit requirement: “you must have a flow control capable link partner”.

The medium is modular on most server adapters: the driver documentation for the same family describes copper modules, SFP+ fiber modules and direct-attach cables, and the driver only loads when the installed module is one it knows. Different links therefore appear under the same interface name — which is why the negotiated speed and the counters must be read before assuming anything about the cable.

The frames: why the payload size is part of the performance

Every frame on the wire carries per-frame overhead, and the smaller the payload, the larger the share of the link that overhead consumes. Raising the MTU to jumbo sizes reduces the number of frames needed for large transfers; the driver documentation for the adapter family used here puts the maximum jumbo MTU at 9710 bytes, coinciding with a maximum jumbo frame size of 9728 bytes.

The cost of that decision is coordination: every device on the path — adapter, switch, peer — must accept the larger frame, or fragments and silent drops appear. Jumbo frames are therefore a design decision for a controlled network, not a checkbox on one host.

The host side: queues, interrupts and offloads

The adapter can only deliver what the host can accept. Microsoft’s performance-tuning documentation for Windows Server describes the two sides of this:

Offload features transfer processing tasks from the CPU to the network adapter hardware, reducing system overhead; the common ones are TCP checksum offload, Large Send Offload (LSO) and Receive Side Scaling (RSS). But the same page warns that an adapter with limited hardware resources may see its maximum sustainable throughput reduced when too much is offloaded — the offload is not free.
Interrupt moderation reduces the number of interrupts per unit of traffic (saving CPU) at the cost of latency; the documentation frames it as “the trade-off between the host CPU savings and latency” and suggests it for CPU-bound workloads.
RSS queues decide how many CPUs share the receive path; the documentation describes changing the number of RSS queues from a default of two up to the maximum the adapter supports, and assigning queues to processors.

The practical reading: high throughput on a single connection can be limited by a single queue, a single CPU and a single interrupt stream, even when the link is idle.

Aggregation: more links, or a link that survives

Bonding several interfaces is a host-side decision. Linux’s bonding documentation distinguishes the modes: in active-backup “only one slave in the bond is active”, so the bond buys availability rather than capacity and failover moves traffic to the remaining link; in 802.3ad mode the bond performs IEEE 802.3ad link aggregation negotiated with the switch using LACP, which requires a switch that supports it.

Neither mode makes a single flow faster in general: aggregation distributes flows across links, and one large transfer is still one flow. Choosing between capacity and failover is a design decision that belongs with the switch configuration, not with the host alone.

When throughput is below the line rate

Kernel network statistics documentation names the counters that answer the first question — is the host dropping what the link delivered? — including rx_dropped (“received but not processed”), receiver FIFO overflow events and CRC errors. A short list to check:

the negotiated speed and the capabilities each link partner advertises;
error counters (CRC, FIFO overflow) — a physical or negotiation problem;
drop counters — the host is not keeping up;
queue and interrupt distribution — one CPU may be doing all the work;
frame size and the workload’s flow count.

A common misconception: “10 GbE means 10 Gb of application traffic”

The line rate is the rate of the link, before frame overhead, before protocol overhead, before the host has done any work. A single TCP connection, a single receive queue, one interrupt-bound CPU, disabled offloads or a congested switch will each produce a throughput far below the number on the adapter. The useful question is never “is the link at line rate?” but “which of the two sides is the constraint right now, and which counter says so?”.

What to remember

A NIC is the boundary between host memory and the link; throughput is set by the weaker side.
Speed and flow control are negotiated with the other end, not decided alone.
The medium is modular (copper, fiber, direct attach) while the interface name is not.
Frame size, offloads, RSS queues and interrupt moderation move work between adapter and CPU.
Counters separate “the link is broken” from “the host is not keeping up”.

Level and prerequisites

L1 — fundamentals: the parts of the path and what each one decides. Prerequisites: the idea of a link between two devices; the Ethernet framing itself is treated in the Networking area. Reading ethtool/ ethtool -S, configuring bonding modes, and rolling out jumbo frames are L2 operational material; VLAN- aware virtual networking and SR-IOV belong to L3–L4.

Where to go next

Server & Virtualization — the area this sheet belongs to.
Networking — frames, MAC addresses, VLANs and switching behaviour, treated as their own subject in the Networking area.

References

Linux kernel documentation — ixgbe driver (Intel 10 Gb adapter driver): speed setting and autonegotiation, flow control (IEEE 802.3x) and link-partner requirement, jumbo frames and maximum MTU, supported module types.
Microsoft Learn — Network Adapter Performance Tuning in Windows Server: offloads (TCP checksum offload, LSO, RSS), interrupt moderation and its CPU/latency trade-off, RSS queues and processor assignment.
Linux kernel documentation — Linux Ethernet Bonding Driver HOWTO: active-backup behaviour, 802.3ad aggregation and LACP, failover.
Linux kernel documentation — Network statistics: rx_dropped, receiver FIFO overflow and CRC error counters.