Multi-site architecture: choosing the failure domain before the topology
An estate is multi-site when its parts fail alone: name the unit of failure first.

An estate is multi-site not because it holds several sites but because its parts can fail independently. The first design decision is therefore the failure domain — the set of things that stop together — not the topology. The word site is topological: a set of systems is a site when they “have mutual IP interconnectivity that doesn’t require use of the backbone”, and RFC 4364 gives the test — if the private line between two locations fails and traffic falls back to the wide area, “one site has become two” (rfc4364.txt lines 418–431).
Sites exist to fail apart
A second site is bought for two reasons, and only one is architectural. “The geographical spread of enterprise-based (on-premises) IT resources (e.g., multiple data centers, headquarters, and branch offices)” drives the landscape; resources are spread “also for reliability and disaster recovery (DR) considerations” (nist-sp800-215.txt lines 254–255, 384–386). The second is the design one: a second site buys a second failure domain, so an estate is judged by how far a failure travels. RFC 7938 makes that a requirement — “Minimize the failure domain of equipment or protocol issues as much as possible” — and reports that a large “failure impact scope” “typically means slower convergence since more devices have to be notified” (rfc7938.txt lines 303–304, 1329–1334). Convergence is the L4 sheet’s; the domain’s size is this one’s.
choose the failure domain, then the topology
what fails together the topology is drawn
──────────────────── to fit that choice
link ─▶ the path site A ──────── site B
device ─▶ the node │ ╲ ╱ │
site ─▶ power, links, │ ╲ ╱ │
the whole building │ ╲ ╱ │
region ─▶ provider, geography site C ──────── site D
a second path is a second failure domain only if it
shares no conduit, no power feed, no device, no operator
The decisions, in order
Three decisions follow from the failure domain. First, what to centralise: a shared service is one failure domain used by every site, so a service every site depends on undoes independence. NIST’s SD-WAN account shows both pulls — the technology “removes the tight coupling between the control plane and data plane functions of the network” (nist-sp800-215.txt lines 1327–1329) — and where the control plane lives is part of the choice; what moves when control is centralised belongs to the sibling sheet on SDN. Second, what must survive losing a site: the service, its capacity, its state, and the name resolution and identity it used — a record the L2 Cybersecurity Governance sheet on asset and service inventory keeps. Third, what the addressing plan presumes: that a site’s prefixes are unique and summarisable (the L3 sheet on summarisation and address-space design).
Two paths, one failure
Redundancy is counted in failure domains, not in links. Two paths are redundant only if nothing beneath them is shared; where they share a conduit, a power feed, a device or the automation that pushes both, they are one failure carrying two labels — the L4 resilience sheet’s rule, applied here at the site boundary. Before calling a second site a standby, ask what it shares with the primary: the same provider, the same power grid, the same directory. RFC 7209 puts it as a placement requirement: provider edges should “be spread geographically (e.g., in different Central Offices (COs) or Points of Presence (POPs))” to give a “geo-redundant solution that ensures business continuity for critical applications in the case of power outages, natural disasters, etc.” (rfc7209.txt lines 347–351).
Losing a link is not losing a site
A link failure costs a path; losing a whole site costs a capacity, a state, and every dependency that lived only there. Detection shows the difference: Bidirectional Forwarding Detection (BFD) detects “failures in the path between adjacent forwarding engines” (rfc5880.txt lines 150–153) — a broken path, not a lost site — and detection is the L4 resilience sheet’s subject. What a whole site takes with it is an inventory question: the services hosted only there, their state, and the name resolution and identity that site alone answered for. Capacity is easiest to forget: a standby sized for a fraction of the load cannot carry the primary’s. Distance matters: a hairpinned path “results in extra distance, increases network latency, and has the potential to cause traffic bottlenecks” (nist-sp800-215.txt lines 537–539).
The interconnection is a policy, not links
Once the failure domains are settled, the interconnection is a policy about who may reach whom, not a list of tunnels — the L4 sheet on VPN architectures and site interconnection owns the tunnels and the CE/PE models. RFC 4364 expresses it as membership: “two sites may have IP interconnectivity over that backbone only if at least one of these subsets contains them both” (rfc4364.txt lines 179–181). The same mechanism shapes traffic, not merely permits it: one can “allow every site to have a direct route to every other site” or “force traffic between certain pairs of sites to be routed via a third site”, as when traffic must cross a firewall at a site (rfc4364.txt lines 217–221). RFC 7209 asks for flexible topologies, citing the E-Tree, where leaves reach only the roots (rfc7209.txt lines 664–667).
The test rule
A design has proved nothing until it has failed over on purpose: failover is verified by test, never by configuration, as the L4 resilience sheet states. The order is the argument — name what fails together, decide what must survive losing a site, draw the interconnection, then exercise the loss. A topology drawn first has a failure domain nobody chose.
Level and prerequisites
L5 — specialist, design-level. It assumes the L3 address-space sheet, the L4 sheets on VPN architectures and on resilience, convergence and failure domains, and the L2 Cybersecurity Governance record of services and dependencies.
Where to go next
References
- RFC 4364 — BGP/MPLS IP Virtual Private Networks (VPNs) (Rosen, Rekhter, February 2006, Standards Track): the topological definition of a site, interconnectivity as membership in a subset of sites, the full-mesh and third-site policies.
- RFC 7209 — Requirements for Ethernet VPN (EVPN) (Sajassi, Aggarwal, Uttaro, Bitar, Henderickx, Isaac, May 2014, Informational): the geo-redundant placement of provider edges and the E-Tree topology as an example of a flexible VPN topology.
- RFC 7938 — Use of BGP for Routing in Large-Scale Data Centers (Lapukhov, Premji, Mitchell, August 2016, Informational): the requirement to minimise the failure domain and the failure impact scope as a determinant of convergence.
- RFC 5880 — Bidirectional Forwarding Detection (BFD) (Katz, Ward, June 2010, Standards Track): the goal of detecting failures in the path between adjacent forwarding engines.
- NIST SP 800-215 — Guide to a Secure Enterprise Network Landscape (Chandramouli, November 2022): the geographic spread of IT resources and the reliability and disaster-recovery driver, the centralised specification of policy, and the latency of a hair-pinned path.