Designing for scale: what breaks first when the network grows
Scalability is a set of axes; each breaks for its own reason, at its own size.

Scale is not one problem, and it does not fail in one place. A network grows along several axes at once — devices, prefixes, adjacencies, sessions, routes, policies, and the people who change them — and each has its own breaking point. RFC 7938 describes designs built for data centres that “support over one hundred thousand servers”, naming the goal as the ability to run them at all: environments of that scale emphasise “operational simplicity and network stability” (rfc7938.txt lines 19–20 and 23–24).
Scale is a set of axes, not one number
Count the axes separately, because they are not interchangeable. More devices mean more links and adjacencies; more prefixes, more entries the forwarding table must hold; more sessions, more neighbours the control plane must keep current; more policies, more rules each change is judged against. The rate of change — how many events per hour the network must absorb — is a quantity of its own, and a design can scale one axis and still fail on another.
axis what it multiplies what gives first
────────────── ────────────────────────────── ─────────────────────────
devices links and adjacencies control-plane state
prefixes forwarding-table entries table memory and lookup
sessions neighbours to keep current session scaling
policies rules a change is judged by configuration volume
rate of change events per unit time propagation and stability
people decisions to make and record approval capacity
What breaks first, in order
State goes first. Advertising every point-to-point link and its prefix everywhere “may create Forwarding Information Base (FIB) overload” and adds “additional path computation stress on the BGP control plane for little benefit” (RFC 7938, lines 921–924). Propagation goes next: the disruption a routing change causes “is determined by a number of factors”, and the more devices must be told, the longer it lasts (RFC 5714, lines 268–269) — “Larger failure impact scope typically means slower convergence” (RFC 7938, line 1332). Configuration goes with them: every device is another place a change can go wrong. Many problems called “scalability” are therefore operational — a process a larger network makes more expensive, not a protocol fault.
Hierarchy is the lever that scales
The structural move that survives growth shortens how far an event must travel and shrinks what each router must hold. OSPF’s area boundary is the model: “The topology of an area is invisible from the outside of the area”, and that isolation buys “a marked reduction in routing traffic as compared to treating the entire Autonomous System as a single link-state domain” (RFC 2328, lines 1278 and 1281–1283). Summarisation applies the same idea to the address plan — one route standing for many, which the L3 sheet on summarisation owns — but only where the topology supports it: with a single path between tiers, “route summarization is not possible” without risking black-holes (RFC 7938, line 1465). Lower the cost per device rather than spread a fixed cost over more devices.
Keep an event where it happened
The second lever is containment — how far a change is allowed to be heard. A protocol that propagates only its best path keeps an event local: “every BGP router calculates and propagates only the best-path selected”, so a failure “is masked as soon as the BGP speaker finds an alternate path”, where “the event propagation scope of a link-state IGP is an entire area, regardless of the failure type” (RFC 7938, lines 747–752). Route reflection applies this to the session mesh: a full mesh needs “n*(n-1)/2 unique Internal BGP (IBGP) sessions” and “clearly does not scale”, while a reflector “summarizes routing information and only reflects its best path” (RFC 4456, lines 88–90 and 389–390). What does not contain instability is a feature that hides it: route flap dampening, to mute a flapping prefix, is “it is not recommended to enable this feature” here, because it suppresses real events (RFC 7938, line 1322).
The convergence budget, and which lever scales
Convergence has a budget — detection, propagation, computation, installation — whose phases and mechanisms belong to the L4 sheet. A design’s choice is between two ways to spend less: tune the terms, or reduce the domain. The first has a hard floor: “the distributed nature of the network imposes an intrinsic limit on the minimum convergence time that can be achieved” (RFC 5714, lines 101–103). The cost also tracks how much changed — “the IGP convergence duration is proportional to the number of routes to update” (RFC 6571, lines 864–866) — so tuning a timer sets a constant a growing event count eventually overruns, while reducing the domain lowers the term itself. The same line separates repairs that scale from those that do not: one whose time “does not scale with the number of destinations to be fast-rerouted” (RFC 6571, lines 1509–1510).
The limit is the people
At some size the binding constraint stops being a table and becomes a team. RFC 7938 states the aim of its whole design as letting “a small group of people can effectively support a significantly sized network” (lines 145–152): every device added, prefix advertised and timer tuned is more for someone to read, approve and repair. Who signs an approved change belongs to the governance area — the L3 sheet on change management owns that act — so scale is partly a staffing question. A design that needs a person per site does not scale, however well the protocol behaves.
Level and prerequisites
L5 — design at scale. It assumes the L4 sheets on OSPF areas, on BGP sessions and policy, and on resilience and convergence (which owns the phase model), plus the L3 sheet on summarisation and address-space design.
Where to go next
References
- RFC 7938 — Use of BGP for Routing in Large-Scale Data Centers (Lapukhov, Premji, Mitchell; Informational; August 2016) — the scale of the design and its stated goal (lines 19–20, 23–24, 145–152); the failure-domain framing (lines 246–251); best-path propagation and flooding scope (lines 747–752); prefix advertisement and control-plane load (lines 921–924); convergence and the failure-impact scope (line 1332); the summarisation caution (line 1465).
- RFC 2328 — OSPF Version 2 (Moy; STD 54; April 1998) — the area as a flooding domain and the reduction in routing traffic it buys (lines 1278, 1281–1283).
- RFC 4456 — BGP Route Reflection: An Alternative to Full Mesh Internal BGP (IBGP) (Bates, Chen, Chandra; Standards Track; April 2006) — the full-mesh session count and its scaling (lines 88–90); a reflector reflecting only its best path (lines 389–390).
- RFC 5714 — IP Fast Reroute Framework (Shand, Bryant; Informational; January 2010) — the intrinsic limit on convergence time (lines 101–103); the factors that set the disruption duration (lines 268–269).
- RFC 6571 — Loop-Free Alternate (LFA) Applicability in SP Networks (Filsfils et al.; Informational; June 2012) — convergence proportional to the routes to update (lines 864–866); a repair whose cost does not scale with destinations (lines 1509–1510).