VXLAN and EVPN: an overlay the underlay does not have to know
How VXLAN wraps an Ethernet frame in UDP and how EVPN replaces flooding with routes.

An overlay separates the identity of a network segment from the physical topology that carries it. VXLAN (Virtual eXtensible Local Area Network) wraps an Ethernet frame inside a UDP/IP packet, so a Layer 2 segment spans a routed network, and the VNI (VXLAN Network Identifier) in that header — not a VLAN tag on the wire — names the segment. Only the tunnel endpoints see this; they are VTEPs (VXLAN Tunnel End Points), and the hosts behind them never see the encapsulation. VXLAN, though, is only the data plane. The control plane decides how the fabric behaves, and the answer the industry settled on is EVPN (Ethernet VPN), which distributes MAC and IP reachability in BGP instead of discovering it by flooding. The overlay asks the underlay for exactly one thing: IP reachability between VTEPs.
Why an overlay exists at all
The pressures are in the VXLAN framework itself: server virtualisation puts many MAC addresses behind one link, tenants’ MAC addresses and VLAN IDs may overlap, and a Layer 2 domain stretched across a site collides with the spanning tree’s disabled links. The remedy is to stop carrying segment identity in the Ethernet header at every hop. “In short, VXLAN is a Layer 2 overlay scheme on a Layer 3 network” (rfc7348.txt lines 332–333). Each segment is named by a 24-bit identifier, “the "VXLAN Network Identifier (VNI)"” (lines 343–344), which “allows up to 16 M VXLAN segments to coexist within the same administrative domain” (lines 344–346). Overlapping addresses cease to matter because the VNI, not the tagged wire, separates them.
A frame inside a UDP packet
The VNI travels in an eight-byte header between the inner Ethernet frame and an outer UDP/IP header: a flag, the 24-bit VNI, and reserved fields. The destination UDP port is the IANA-assigned 4789; the source port is normally a hash of the inner frame, which is what gives the underlay’s load balancers something to spread. “The tunnels are stateless, so each frame is encapsulated according to a set of rules” (lines 357–359). The encapsulating device is the VTEP, “An entity that originates and/or terminates VXLAN tunnels” (lines 203–204); the encapsulation is “known only to the VTEP – the VM never sees it” (lines 361–363).
overlay: what the guest actually sends
+---------------------------------------------+
| inner Ethernet frame (dst MAC, src MAC, …) |
+---------------------------------------------+
| the VTEP adds, outside it:
+-----------------v---------------------------+
| VXLAN: I flag + 24-bit VNI + reserved |
+---------------------------------------------+
| UDP: src port = hash(inner), dst port 4789 |
+---------------------------------------------+
| outer IP: src = this VTEP, dst = remote |
+---------------------------------------------+
| outer Ethernet: next hop on the underlay |
+---------------------------------------------+
|
underlay: an IP path between VTEP addresses.
The routers between them read the outer header only —
the VNI and the inner frame are opaque to them.
The control plane decides the rest
The original scheme learns in the data plane: “the association of VM’s MAC to VTEP’s IP address is discovered via source-address learning”, while broadcast and unknown frames go to an IP multicast group (lines 371–373, 433–434). That scales no better than the flooding it replaced. EVPN is the routed answer: MAC and IP reachability travel in BGP, using an Address Family Identifier (AFI) of 25 (L2VPN) and a Subsequent Address Family Identifier (SAFI) of 70 (EVPN), and “MAC learning between PEs occurs not in the data plane (as happens with traditional bridging in VPLS [RFC4761] [RFC4762]) but in the control plane” (rfc7432.txt lines 306–308). Four route types do the work: Ethernet Auto-Discovery, MAC/IP Advertisement, Inclusive Multicast Ethernet Tag and Ethernet Segment (lines 700–705).
What a routed control plane changes
Carrying the host’s IP in the MAC/IP Advertisement route lets a PE answer for a remote host and cut the ARP/ND flood: the route “may optionally carry one of the IP addresses associated with the MAC address” (lines 1844–1845), and “This option also minimizes ARP (or ND) message processing on end-stations/hosts connected to the EVPN network” (lines 1856–1857). Control-plane learning also makes all-active multihoming usable, sharing load across “CEs that are multihomed to multiple PEs” (lines 318–319) instead of idling the second link. The requirements document ratifies the goal — a solution “SHOULD minimize the flooding of broadcast frames outside the confines of a given site” (rfc7209.txt lines 647–651) — and a duplicate MAC “behind two different Ethernet segments” (lines 698–700) is the case EVPN’s MAC Mobility procedures exist to detect. Scale is why a routing protocol is acceptable here at all: the “design goal for NVO is millions of instances per common physical infrastructure” (rfc8365.txt lines 415–418).
What the underlay still has to do
The overlay assumes one service from below: “The underlay network for NVO solutions is assumed to provide IP connectivity between NVO endpoints” (rfc8365.txt lines 212–213). In a modern data centre that underlay is a routed Clos fabric, often EBGP-only with ECMP across the middle stage — “ECMP is the fundamental load-sharing mechanism used by a Clos topology” (rfc7938.txt lines 1078–1079). Two obligations fall on the overlay’s operator. First, MTU: an encapsulated frame is larger than the original, “VTEPs MUST NOT fragment VXLAN packets” (rfc7348.txt line 490), and the framework recommends raising the physical MTU so the extra header fits. Second, entropy: if every tunnel packet carried the same outer address, the underlay’s ECMP would collapse onto one path — exactly what the source-port hash avoids.
What the overlay does not fix
An overlay is often sold as making the underlay irrelevant, and it does not. MTU is still end to end. Failure domains and reconvergence are not improved by tunnelling them, and belong to the resilience sheet. Diagnosis now spans two layers whose addressing does not line up. Security changes shape rather than disappearing: a “MAC-over-IP mechanism for delivering Layer 2 traffic significantly extends this attack surface” (rfc7348.txt lines 998–1000), and because the control plane is BGP, “Compromise of the control plane could result in a PE sending customer data belonging to some EVPN to another EVPN, or black-holing EVPN customer data, or even sending it to an eavesdropper” (rfc7432.txt lines 2793–2795). Carrying the overlay between sites is a subject of its own.
Level and prerequisites
L5 — specialist. It assumes the L1 vocabulary of the broadcast domain, the MAC table and the packet journey; the L2 sheet on VLANs and 802.1Q tagging, which owns the tag itself; and the L4 sheet on BGP sessions, attributes and policy for the transport EVPN rides on. Nothing here is configuration for a named platform; the mechanics are the ones the RFCs define.
Where to go next
References
- RFC 7348 — Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks (Mahalingam, Dutt, Duda, Agarwal, Kreeger, Sridhar, Bursell, Wright, August 2014, Informational): the problem statement, the VNI and segment, the stateless tunnel and the VTEP, the UDP port and source-port hash, the MTU rule, and the extended attack surface.
- RFC 7209 — Requirements for Ethernet VPN (EVPN) (Sajassi, Aggarwal, Uttaro, Bitar, Henderickx, Isaac, May 2014, Informational): the all-active and single-active definitions, the flood-suppression requirement, and the duplicate-MAC detection requirement.
- RFC 7432 — BGP MPLS-Based Ethernet VPN (Sajassi, Aggarwal, Bitar, Isaac, Uttaro, Drake, Henderickx, February 2015, Standards Track): control-plane MAC learning, the four EVPN route types and the AFI 25 / SAFI 70 address family, the MAC/IP Advertisement route and ARP/ND suppression, all-active multihoming, and the control-plane security discussion.
- RFC 7938 — Use of BGP for Routing in Large-Scale Data Centers (Lapukhov, Premji, Mitchell, August 2016, Informational): the routed Clos underlay, EBGP as the single routing protocol, ECMP, and the single-operator-domain security assumption.
- RFC 8365 — A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN) (Sajassi, Drake, Bitar, Shekhar, Uttaro, Henderickx, March 2018, Standards Track): EVPN as a network-virtualisation overlay, the VXLAN encapsulation, the underlay assumption, control-plane learning against flooding, and the route constructions.