Articles

Designing for scale: what breaks first when the network grows

Scalability is a set of axes; each breaks for its own reason, at its own size.

Reading: 5 minNetworking

Article cover: Designing for scale: what breaks first when the network grows

Scale is not one problem, and it does not fail in one place. A network grows along several axes at once — devices, prefixes, adjacencies, sessions, routes, policies, and the people who change them — and each has its own breaking point. RFC 7938 describes designs built for data centres that “support over one hundred thousand servers”, naming the goal as the ability to run them at all: environments of that scale emphasise “operational simplicity and network stability” (rfc7938.txt lines 19–20 and 23–24).

Scale is a set of axes, not one number

Count the axes separately, because they are not interchangeable. More devices mean more links and adjacencies; more prefixes, more entries the forwarding table must hold; more sessions, more neighbours the control plane must keep current; more policies, more rules each change is judged against. The rate of change — how many events per hour the network must absorb — is a quantity of its own, and a design can scale one axis and still fail on another.

axis            what it multiplies               what gives first
──────────────  ──────────────────────────────   ─────────────────────────
devices         links and adjacencies            control-plane state
prefixes        forwarding-table entries         table memory and lookup
sessions        neighbours to keep current       session scaling
policies        rules a change is judged by      configuration volume
rate of change  events per unit time             propagation and stability
people          decisions to make and record     approval capacity

What breaks first, in order

State goes first. Advertising every point-to-point link and its prefix everywhere “may create Forwarding Information Base (FIB) overload” and adds “additional path computation stress on the BGP control plane for little benefit” (RFC 7938, lines 921–924). Propagation goes next: the disruption a routing change causes “is determined by a number of factors”, and the more devices must be told, the longer it lasts (RFC 5714, lines 268–269) — “Larger failure impact scope typically means slower convergence” (RFC 7938, line 1332). Configuration goes with them: every device is another place a change can go wrong. Many problems called “scalability” are therefore operational — a process a larger network makes more expensive, not a protocol fault.

Hierarchy is the lever that scales

The structural move that survives growth shortens how far an event must travel and shrinks what each router must hold. OSPF’s area boundary is the model: “The topology of an area is invisible from the outside of the area”, and that isolation buys “a marked reduction in routing traffic as compared to treating the entire Autonomous System as a single link-state domain” (RFC 2328, lines 1278 and 1281–1283). Summarisation applies the same idea to the address plan — one route standing for many, which the L3 sheet on summarisation owns — but only where the topology supports it: with a single path between tiers, “route summarization is not possible” without risking black-holes (RFC 7938, line 1465). Lower the cost per device rather than spread a fixed cost over more devices.

Keep an event where it happened

The second lever is containment — how far a change is allowed to be heard. A protocol that propagates only its best path keeps an event local: “every BGP router calculates and propagates only the best-path selected”, so a failure “is masked as soon as the BGP speaker finds an alternate path”, where “the event propagation scope of a link-state IGP is an entire area, regardless of the failure type” (RFC 7938, lines 747–752). Route reflection applies this to the session mesh: a full mesh needs “n*(n-1)/2 unique Internal BGP (IBGP) sessions” and “clearly does not scale”, while a reflector “summarizes routing information and only reflects its best path” (RFC 4456, lines 88–90 and 389–390). What does not contain instability is a feature that hides it: route flap dampening, to mute a flapping prefix, is “it is not recommended to enable this feature” here, because it suppresses real events (RFC 7938, line 1322).

The convergence budget, and which lever scales

Convergence has a budget — detection, propagation, computation, installation — whose phases and mechanisms belong to the L4 sheet. A design’s choice is between two ways to spend less: tune the terms, or reduce the domain. The first has a hard floor: “the distributed nature of the network imposes an intrinsic limit on the minimum convergence time that can be achieved” (RFC 5714, lines 101–103). The cost also tracks how much changed — “the IGP convergence duration is proportional to the number of routes to update” (RFC 6571, lines 864–866) — so tuning a timer sets a constant a growing event count eventually overruns, while reducing the domain lowers the term itself. The same line separates repairs that scale from those that do not: one whose time “does not scale with the number of destinations to be fast-rerouted” (RFC 6571, lines 1509–1510).

The limit is the people

At some size the binding constraint stops being a table and becomes a team. RFC 7938 states the aim of its whole design as letting “a small group of people can effectively support a significantly sized network” (lines 145–152): every device added, prefix advertised and timer tuned is more for someone to read, approve and repair. Who signs an approved change belongs to the governance area — the L3 sheet on change management owns that act — so scale is partly a staffing question. A design that needs a person per site does not scale, however well the protocol behaves.

Level and prerequisites

L5 — design at scale. It assumes the L4 sheets on OSPF areas, on BGP sessions and policy, and on resilience and convergence (which owns the phase model), plus the L3 sheet on summarisation and address-space design.

Where to go next

References

  • RFC 7938 — Use of BGP for Routing in Large-Scale Data Centers (Lapukhov, Premji, Mitchell; Informational; August 2016) — the scale of the design and its stated goal (lines 19–20, 23–24, 145–152); the failure-domain framing (lines 246–251); best-path propagation and flooding scope (lines 747–752); prefix advertisement and control-plane load (lines 921–924); convergence and the failure-impact scope (line 1332); the summarisation caution (line 1465).
  • RFC 2328 — OSPF Version 2 (Moy; STD 54; April 1998) — the area as a flooding domain and the reduction in routing traffic it buys (lines 1278, 1281–1283).
  • RFC 4456 — BGP Route Reflection: An Alternative to Full Mesh Internal BGP (IBGP) (Bates, Chen, Chandra; Standards Track; April 2006) — the full-mesh session count and its scaling (lines 88–90); a reflector reflecting only its best path (lines 389–390).
  • RFC 5714 — IP Fast Reroute Framework (Shand, Bryant; Informational; January 2010) — the intrinsic limit on convergence time (lines 101–103); the factors that set the disruption duration (lines 268–269).
  • RFC 6571 — Loop-Free Alternate (LFA) Applicability in SP Networks (Filsfils et al.; Informational; June 2012) — convergence proportional to the routes to update (lines 864–866); a repair whose cost does not scale with destinations (lines 1509–1510).
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.