Resilience engineering
DR drills & chaos engineering
Scheduled failovers, region evictions, and game days. Prove recovery time objectives in production — not in slideware.
RTO (region-loss)
< 4 min
RPO (data-loss)
0 sec
Drills passed (Q2)
11 of 12
Active experiments
3
Drill calendar
- Region failover us-east → us-west · Jun 4, 02:00 UTCscheduled
- Kafka broker brownout (50% loss) · Jun 8, 14:00 UTCscheduled
- Postgres primary stop-the-world · Jun 12, 22:00 UTCscheduled
- Edge node disconnect — Westside clinic · Jun 18, 09:00 UTCscheduled
Last drill: store-and-forward
- Disconnect: 47 minutes
- Local writes queued: 1,284 charts
- Reconciliation conflicts: 0
- RPO observed: 0 sec · RTO 3m 12s
Game day runbooks
- Ransomware tabletop
- Cloud account compromise
- DNS provider outage
- Identity provider (SSO) hard-down