Articles

Multi-site disaster recovery

Two sites, one decision: replication modes, failover, and the difference from backup.

Reading: 5 minServer & Virtualization

Article cover: Multi-site disaster recovery

When an entire site is lost, recovery was decided before the loss: choose a strategy, choose a replication mode, and choose the proof it works. Each choice commits you to a recovery point objective (RPO), how much data you may lose, and a recovery time objective (RTO), how long you may be down. This sheet follows that decision across two sites, closing on two rules that outlive the technology: an untested disaster recovery plan does not exist, and failback is a project, not a command.

Four strategies, one decision

The strategies are “four approaches, ranging from the low cost and low complexity of making backups to more complex strategies using multiple active Regions”; in the passive designs “The passive site does not actively serve traffic until a failover event is triggered”. What differs is the second site: backup and restore leaves only data there, a pilot light keeps the core on, a warm standby “decreases the time to recovery because your workload is always-on in another Region” and “can handle traffic (at reduced capacity levels) immediately”, and multi-site active/active “serves traffic from all regions to which it is deployed” with “no such thing as failover in this scenario”.

   one site is lost
        |
        v
   choose a strategy by the recovery it commits you to
        |
        +-- backup and restore ....... RPO = last backup      second site holds data only
        |
        +-- pilot light .............. RPO low (continuous)   core on, app servers off
        |
        +-- warm standby ............. RPO low                full copy on, scaled down
        |
        +-- multi-site active/active . RPO low, RTO near zero both sites serve; no failover
   cost and complexity rise from the top of the list to the bottom

Continuity is not recovery

NIST SP 800-34 Revision 1 separates two ideas that vendors blend. Contingency planning “refers to interim measures to recover information system services after a disruption”. “Continuity planning normally applies to the mission/business itself; it concerns the ability to continue critical functions and processes during and after an emergency event”, while contingency planning “applies to information systems”. Continuity keeps the business answering; recovery puts the system back.

The site-specific plan is the DRP, which “applies to major, usually physical disruptions to service that deny access to the primary facility infrastructure for an extended period” and restores the target system at an alternate site. A restart in place is an earlier sheet.

Three numbers size it: “The MTD represents the total amount of time the system” owner will accept for an outage; “RTO defines the maximum amount of time that a system” resource may be unavailable; RPO “is a factor of how much data loss the mission/business process can tolerate”. The ordering is a rule — “the RTO must normally be shorter than the MTD”.

Replication mode is an RPO choice

“Data replication is the process of writing the same data to at least two separate locations”, and there are “two types of replication: synchronous and asynchronous”. Synchronous “involves the real-time copying of data from site A”; asynchronous “involves time delay and may be performed continuously or using a designated” frequency. The decisive sentence is next: “The time delay and frequency are dictated by the enterprise’s disaster recovery policy and are described in terms of specific recovery time objective (RTO) and recovery point objective (RPO) goals.” The mode is a business choice, not an operational preference.

Storage Replica marks the two ends. Synchronous “guarantees that the application writes data to two locations at once before it completes the I/O operation” — RPO zero. Asynchronous lacks “a guarantee that both sites have identical copies of the data at the time of a potential failure”, so its “higher-than-zero RPO” makes it “less suitable for high-availability solutions like failover clusters”. Order is preserved too — “Write ordering guarantees that applications like SQL Server can write to multiple replicated volumes” — but the same fidelity copies a deletion “instantly to the other volume, irrevocably removing the data from both servers”. A replica is not a backup, and the link is its own trust boundary: data in transit during replication “should be protected using encryption”.

The third site: stretch clusters and quorum

A stretch cluster answers the two-site decision at the storage layer: “A stretch cluster spreads hosts across geographically separated data centers”. Each zone holds a copy of the data, and “a tiebreaker (arbiter, witness)” sits in a third, which “picks a winner when a network connection between zones fails and both data centers remain alive”.

The requirement is exact: “Two Monitors must be run in each data center, plus a tiebreaker in a third (possibly in the cloud) for a total of five Monitors.” Stretch mode “is designed to handle netsplit scenarios between two data centers as well as the loss of one data center”, resolving a site loss by “reducing the min_size of all pools to 1” — but each zone then holds “two replicas”, “OSDs must be located at exactly two zones”, and “Erasure-coded pools cannot be used with stretch mode”. Pool mechanics are the distributed-storage sheet’s subject.

Replication at the service level

Spanning one cluster across sites is often wrong: Ceph warns that “Running a single geographically distributed Ceph storage cluster is NOT recommended unless you have low-latency WAN connections”. The service-level alternative needs “at least two Ceph storage clusters”, each with a gateway. Zones run “in an active-active configuration by default, which means that a gateway client may write data to any zone and the zone will replicate the data to all other zones within the zonegroup”; a --read-only zone is active-passive instead.

Ownership is explicit: “Each bucket is owned by the zonegroup where it was created”, and a request sent elsewhere “will redirect to the zonegroup where the bucket resides”. Metadata is stricter: an operation such as user creation “must be run on a host within the master zone”.

Failover is a decision; failback is a project

Failover is the rehearsed part, and Ceph’s promotion is three steps, each of them written down: “If the master zone fails, you can fail over to the secondary zone for disaster recovery”.

radosgw-admin zone modify --rgw-zone=<zone-name> --master --default
radosgw-admin period update --commit
systemctl restart ceph-radosgw@rgw.`hostname -s`

The return is the project: reverting pulls the realm from the new master and re-promotes the recovered zone, because the surviving site has been accepting writes the recovered site never saw. That reconciliation is why Storage Replica’s cross-cluster setups “requires manual intervention for failover”: writes made during the outage must be reconciled first.

Two rules close the subject. An untested plan does not exist: “It is critical to regularly assess and test your disaster recovery strategy so that you have confidence in invoking it”, and “Your backup strategy must include testing your backups” — the only evidence a recovery works is a recovery that has run. And failback is a project, with its own reconciliation and its own proof.

Level and prerequisites

L5 — platform-level design across whole sites: choosing a recovery strategy and a replication mode, and pricing a failover and a failback. The mechanism of replication and the restart of a workload inside one site are earlier sheets; pool consistency is a neighbouring sheet’s subject.

Where to go next

  • Server & Virtualization — the area this sheet belongs to.
  • Sister sheets, in prose: Distributed storage: consistency and availability trade-offs; Backup, replication and continuity across complex architectures; Distributed systems and failure domains; Replication and the mistakes it replicates; Retention, immutability and separate copies; High availability and failure handling.

References

  • Amazon Web Services, Disaster Recovery of Workloads on AWS: Recovery in the Cloud (AWS Well-Architected Framework whitepaper) — URL not recorded in this batch. The four strategies, the active/passive and active/active shapes, and the testing guidance.
  • NIST, Special Publication 800-34 Revision 1, Contingency Planning Guide for Federal Information Systems — URL not recorded in this batch. Continuity versus contingency, the DRP, the alternate-site types, and the MTD/RTO/RPO definitions.
  • NIST, Special Publication 800-209, Security Guidelines for Storage Infrastructure — URL not recorded in this batch. The definition of data replication and its synchronous/asynchronous forms.
  • Microsoft, Storage Replica overview (Microsoft Learn) — URL not recorded in this batch. Synchronous versus asynchronous behaviour, consistency groups, and the manual-intervention note for cross-cluster failover.
  • Ceph Documentation, Stretch Clusters — https://docs.ceph.com/en/latest/rados/operations/stretch-mode/ — the two-zone-plus-tiebreaker topology, stretch mode, and its limits.
  • Ceph Documentation, Multi-Site — https://docs.ceph.com/en/latest/radosgw/multisite/ — realms, zonegroups and zones, bucket ownership, and failover to the secondary zone.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.