Multi-site disaster recovery
Two sites, one decision: replication modes, failover, and the difference from backup.

When an entire site is lost, recovery was decided before the loss: choose a strategy, choose a replication mode, and choose the proof it works. Each choice commits you to a recovery point objective (RPO), how much data you may lose, and a recovery time objective (RTO), how long you may be down. This sheet follows that decision across two sites, closing on two rules that outlive the technology: an untested disaster recovery plan does not exist, and failback is a project, not a command.
Four strategies, one decision
The strategies are “four approaches, ranging from the low cost and low complexity of making backups to more complex strategies using multiple active Regions”; in the passive designs “The passive site does not actively serve traffic until a failover event is triggered”. What differs is the second site: backup and restore leaves only data there, a pilot light keeps the core on, a warm standby “decreases the time to recovery because your workload is always-on in another Region” and “can handle traffic (at reduced capacity levels) immediately”, and multi-site active/active “serves traffic from all regions to which it is deployed” with “no such thing as failover in this scenario”.
one site is lost
|
v
choose a strategy by the recovery it commits you to
|
+-- backup and restore ....... RPO = last backup second site holds data only
|
+-- pilot light .............. RPO low (continuous) core on, app servers off
|
+-- warm standby ............. RPO low full copy on, scaled down
|
+-- multi-site active/active . RPO low, RTO near zero both sites serve; no failover
cost and complexity rise from the top of the list to the bottom
Continuity is not recovery
NIST SP 800-34 Revision 1 separates two ideas that vendors blend. Contingency planning “refers to interim measures to recover information system services after a disruption”. “Continuity planning normally applies to the mission/business itself; it concerns the ability to continue critical functions and processes during and after an emergency event”, while contingency planning “applies to information systems”. Continuity keeps the business answering; recovery puts the system back.
The site-specific plan is the DRP, which “applies to major, usually physical disruptions to service that deny access to the primary facility infrastructure for an extended period” and restores the target system at an alternate site. A restart in place is an earlier sheet.
Three numbers size it: “The MTD represents the total amount of time the system” owner will accept for an outage; “RTO defines the maximum amount of time that a system” resource may be unavailable; RPO “is a factor of how much data loss the mission/business process can tolerate”. The ordering is a rule — “the RTO must normally be shorter than the MTD”.
Replication mode is an RPO choice
“Data replication is the process of writing the same data to at least two separate locations”, and there are “two types of replication: synchronous and asynchronous”. Synchronous “involves the real-time copying of data from site A”; asynchronous “involves time delay and may be performed continuously or using a designated” frequency. The decisive sentence is next: “The time delay and frequency are dictated by the enterprise’s disaster recovery policy and are described in terms of specific recovery time objective (RTO) and recovery point objective (RPO) goals.” The mode is a business choice, not an operational preference.
Storage Replica marks the two ends. Synchronous “guarantees that the application writes data to two locations at once before it completes the I/O operation” — RPO zero. Asynchronous lacks “a guarantee that both sites have identical copies of the data at the time of a potential failure”, so its “higher-than-zero RPO” makes it “less suitable for high-availability solutions like failover clusters”. Order is preserved too — “Write ordering guarantees that applications like SQL Server can write to multiple replicated volumes” — but the same fidelity copies a deletion “instantly to the other volume, irrevocably removing the data from both servers”. A replica is not a backup, and the link is its own trust boundary: data in transit during replication “should be protected using encryption”.
The third site: stretch clusters and quorum
A stretch cluster answers the two-site decision at the storage layer: “A stretch cluster spreads hosts across geographically separated data centers”. Each zone holds a copy of the data, and “a tiebreaker (arbiter, witness)” sits in a third, which “picks a winner when a network connection between zones fails and both data centers remain alive”.
The requirement is exact: “Two Monitors must be run in each data center, plus a tiebreaker in a third (possibly in the cloud) for a total of five Monitors.” Stretch mode “is designed to handle netsplit scenarios between two data centers as well as the loss of one data center”, resolving a site loss by “reducing the min_size of all pools to 1” — but each zone then holds “two replicas”, “OSDs must be located at exactly two zones”, and “Erasure-coded pools cannot be used with stretch mode”. Pool mechanics are the distributed-storage sheet’s subject.
Replication at the service level
Spanning one cluster across sites is often wrong: Ceph warns that “Running a single geographically distributed Ceph storage cluster is NOT recommended unless you have low-latency WAN connections”. The service-level alternative needs “at least two Ceph storage clusters”, each with a gateway. Zones run “in an active-active configuration by default, which means that a gateway client may write data to any zone and the zone will replicate the data to all other zones within the zonegroup”; a --read-only zone is active-passive instead.
Ownership is explicit: “Each bucket is owned by the zonegroup where it was created”, and a request sent elsewhere “will redirect to the zonegroup where the bucket resides”. Metadata is stricter: an operation such as user creation “must be run on a host within the master zone”.
Failover is a decision; failback is a project
Failover is the rehearsed part, and Ceph’s promotion is three steps, each of them written down: “If the master zone fails, you can fail over to the secondary zone for disaster recovery”.
radosgw-admin zone modify --rgw-zone=<zone-name> --master --default
radosgw-admin period update --commit
systemctl restart ceph-radosgw@rgw.`hostname -s`
The return is the project: reverting pulls the realm from the new master and re-promotes the recovered zone, because the surviving site has been accepting writes the recovered site never saw. That reconciliation is why Storage Replica’s cross-cluster setups “requires manual intervention for failover”: writes made during the outage must be reconciled first.
Two rules close the subject. An untested plan does not exist: “It is critical to regularly assess and test your disaster recovery strategy so that you have confidence in invoking it”, and “Your backup strategy must include testing your backups” — the only evidence a recovery works is a recovery that has run. And failback is a project, with its own reconciliation and its own proof.
Level and prerequisites
L5 — platform-level design across whole sites: choosing a recovery strategy and a replication mode, and pricing a failover and a failback. The mechanism of replication and the restart of a workload inside one site are earlier sheets; pool consistency is a neighbouring sheet’s subject.
Where to go next
- Server & Virtualization — the area this sheet belongs to.
- Sister sheets, in prose: Distributed storage: consistency and availability trade-offs; Backup, replication and continuity across complex architectures; Distributed systems and failure domains; Replication and the mistakes it replicates; Retention, immutability and separate copies; High availability and failure handling.
References
- Amazon Web Services, Disaster Recovery of Workloads on AWS: Recovery in the Cloud (AWS Well-Architected Framework whitepaper) — URL not recorded in this batch. The four strategies, the active/passive and active/active shapes, and the testing guidance.
- NIST, Special Publication 800-34 Revision 1, Contingency Planning Guide for Federal Information Systems — URL not recorded in this batch. Continuity versus contingency, the DRP, the alternate-site types, and the MTD/RTO/RPO definitions.
- NIST, Special Publication 800-209, Security Guidelines for Storage Infrastructure — URL not recorded in this batch. The definition of data replication and its synchronous/asynchronous forms.
- Microsoft, Storage Replica overview (Microsoft Learn) — URL not recorded in this batch. Synchronous versus asynchronous behaviour, consistency groups, and the manual-intervention note for cross-cluster failover.
- Ceph Documentation, Stretch Clusters — https://docs.ceph.com/en/latest/rados/operations/stretch-mode/ — the two-zone-plus-tiebreaker topology, stretch mode, and its limits.
- Ceph Documentation, Multi-Site — https://docs.ceph.com/en/latest/radosgw/multisite/ — realms, zonegroups and zones, bucket ownership, and failover to the secondary zone.