Chapter 5 / Reliability

High-Reliability
Design

Failures happen even in the cloud. High-reliability design makes systems resilient, limits impact, and enables quick recovery.

Diagram of high-reliability design
Assuming failures happen, design redundancy, retries, failover, and health checks.
Resiliency

Not zero failures, but smaller impact

Eliminate single points of failure, detect failures, and enable automatic or procedural switchover. Choose a right-sized configuration aligned with SLA, RTO, and RPO.

Reduce single points of failure with redundancy.Design retries and timeouts.Actually test failover.

Make it resilient

  • Identified single points of failure.
  • Considered using availability zones.
  • Considered regional redundancy as needed.

Make recovery easy

  • Chose a configuration matching RTO and RPO.
  • Prepared switchover procedures.
  • Actually tested recovery procedures.

Make it detectable

  • Set up health checks.
  • Decided anomaly detection conditions.
  • Decided failure notification recipients.