Articles

Multi-site architecture: choosing the failure domain before the topology

An estate is multi-site when its parts fail alone: name the unit of failure first.

Reading: 5 minNetworking

Article cover: Multi-site architecture: choosing the failure domain before the topology

An estate is multi-site not because it holds several sites but because its parts can fail independently. The first design decision is therefore the failure domain — the set of things that stop together — not the topology. The word site is topological: a set of systems is a site when they “have mutual IP interconnectivity that doesn’t require use of the backbone”, and RFC 4364 gives the test — if the private line between two locations fails and traffic falls back to the wide area, “one site has become two” (rfc4364.txt lines 418–431).

Sites exist to fail apart

A second site is bought for two reasons, and only one is architectural. “The geographical spread of enterprise-based (on-premises) IT resources (e.g., multiple data centers, headquarters, and branch offices)” drives the landscape; resources are spread “also for reliability and disaster recovery (DR) considerations” (nist-sp800-215.txt lines 254–255, 384–386). The second is the design one: a second site buys a second failure domain, so an estate is judged by how far a failure travels. RFC 7938 makes that a requirement — “Minimize the failure domain of equipment or protocol issues as much as possible” — and reports that a large “failure impact scope” “typically means slower convergence since more devices have to be notified” (rfc7938.txt lines 303–304, 1329–1334). Convergence is the L4 sheet’s; the domain’s size is this one’s.

choose the failure domain, then the topology

  what fails together            the topology is drawn
  ────────────────────            to fit that choice
  link      ─▶ the path            site A ──────── site B
  device    ─▶ the node             │  ╲        ╱  │
  site      ─▶ power, links,         │   ╲      ╱   │
              the whole building     │    ╲    ╱    │
  region    ─▶ provider, geography   site C ──────── site D

  a second path is a second failure domain only if it
  shares no conduit, no power feed, no device, no operator

The decisions, in order

Three decisions follow from the failure domain. First, what to centralise: a shared service is one failure domain used by every site, so a service every site depends on undoes independence. NIST’s SD-WAN account shows both pulls — the technology “removes the tight coupling between the control plane and data plane functions of the network” (nist-sp800-215.txt lines 1327–1329) — and where the control plane lives is part of the choice; what moves when control is centralised belongs to the sibling sheet on SDN. Second, what must survive losing a site: the service, its capacity, its state, and the name resolution and identity it used — a record the L2 Cybersecurity Governance sheet on asset and service inventory keeps. Third, what the addressing plan presumes: that a site’s prefixes are unique and summarisable (the L3 sheet on summarisation and address-space design).

Two paths, one failure

Redundancy is counted in failure domains, not in links. Two paths are redundant only if nothing beneath them is shared; where they share a conduit, a power feed, a device or the automation that pushes both, they are one failure carrying two labels — the L4 resilience sheet’s rule, applied here at the site boundary. Before calling a second site a standby, ask what it shares with the primary: the same provider, the same power grid, the same directory. RFC 7209 puts it as a placement requirement: provider edges should “be spread geographically (e.g., in different Central Offices (COs) or Points of Presence (POPs))” to give a “geo-redundant solution that ensures business continuity for critical applications in the case of power outages, natural disasters, etc.” (rfc7209.txt lines 347–351).

A link failure costs a path; losing a whole site costs a capacity, a state, and every dependency that lived only there. Detection shows the difference: Bidirectional Forwarding Detection (BFD) detects “failures in the path between adjacent forwarding engines” (rfc5880.txt lines 150–153) — a broken path, not a lost site — and detection is the L4 resilience sheet’s subject. What a whole site takes with it is an inventory question: the services hosted only there, their state, and the name resolution and identity that site alone answered for. Capacity is easiest to forget: a standby sized for a fraction of the load cannot carry the primary’s. Distance matters: a hairpinned path “results in extra distance, increases network latency, and has the potential to cause traffic bottlenecks” (nist-sp800-215.txt lines 537–539).

Once the failure domains are settled, the interconnection is a policy about who may reach whom, not a list of tunnels — the L4 sheet on VPN architectures and site interconnection owns the tunnels and the CE/PE models. RFC 4364 expresses it as membership: “two sites may have IP interconnectivity over that backbone only if at least one of these subsets contains them both” (rfc4364.txt lines 179–181). The same mechanism shapes traffic, not merely permits it: one can “allow every site to have a direct route to every other site” or “force traffic between certain pairs of sites to be routed via a third site”, as when traffic must cross a firewall at a site (rfc4364.txt lines 217–221). RFC 7209 asks for flexible topologies, citing the E-Tree, where leaves reach only the roots (rfc7209.txt lines 664–667).

The test rule

A design has proved nothing until it has failed over on purpose: failover is verified by test, never by configuration, as the L4 resilience sheet states. The order is the argument — name what fails together, decide what must survive losing a site, draw the interconnection, then exercise the loss. A topology drawn first has a failure domain nobody chose.

Level and prerequisites

L5 — specialist, design-level. It assumes the L3 address-space sheet, the L4 sheets on VPN architectures and on resilience, convergence and failure domains, and the L2 Cybersecurity Governance record of services and dependencies.

Where to go next

References

  • RFC 4364 — BGP/MPLS IP Virtual Private Networks (VPNs) (Rosen, Rekhter, February 2006, Standards Track): the topological definition of a site, interconnectivity as membership in a subset of sites, the full-mesh and third-site policies.
  • RFC 7209 — Requirements for Ethernet VPN (EVPN) (Sajassi, Aggarwal, Uttaro, Bitar, Henderickx, Isaac, May 2014, Informational): the geo-redundant placement of provider edges and the E-Tree topology as an example of a flexible VPN topology.
  • RFC 7938 — Use of BGP for Routing in Large-Scale Data Centers (Lapukhov, Premji, Mitchell, August 2016, Informational): the requirement to minimise the failure domain and the failure impact scope as a determinant of convergence.
  • RFC 5880 — Bidirectional Forwarding Detection (BFD) (Katz, Ward, June 2010, Standards Track): the goal of detecting failures in the path between adjacent forwarding engines.
  • NIST SP 800-215 — Guide to a Secure Enterprise Network Landscape (Chandramouli, November 2022): the geographic spread of IT resources and the reliability and disaster-recovery driver, the centralised specification of policy, and the latency of a hair-pinned path.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.