Aurora/RDS Failover Connection Storms: When the Database Is Up but the App Is Not

The problem: failover finished, app still down

RDS Multi-AZ or Aurora storage fault triggers a failover. The console shows a new writer in minutes. Your ALB targets are healthy. Yet error rates stay elevated for 10–20 minutes: too many connections, authentication spikes, or clients stuck on the old writer IP. The database recovered; the connection layer did not.

DNS, writers, and the client cache

Classic RDS Multi-AZ keeps a single cluster endpoint CNAME that flips to the standby’s host. Aurora separates cluster (writer) and reader endpoints. Failover updates DNS, but:

• Clients with aggressive DNS TTLs or JVM/golang DNS caches keep the old IP.
• Connection pools hand out sockets still TCP-connected to a host that is now a reader (Aurora) or gone (MySQL crash).
• ORMs retry the same pooled connection instead of recycling the pool on failover errors.

You need a deliberate strategy: short DNS TTL where possible, failover-aware pool eviction (AWS JDBC wrapper, rds.force_ssl + smart drivers, or PgBouncer/Proxy), and application retries that distinguish serializable/failover errors from query bugs.

Aurora vs RDS Multi-AZ under failure

• RDS Multi-AZ: synchronous standby in another AZ; failover promotes standby; storage is engine-managed. Good for lift-and-shift; scaling reads means more replicas you manage.
• Aurora: shared storage volume across AZs; writer failover to an existing reader is typically faster; you can have up to 15 replicas. Storage faults are handled below the engine.

Aurora still fails apps that pin to instance endpoints. Always prefer cluster/reader endpoints unless you are doing advanced custom routing.

Connection storms

When the writer returns, 200 app tasks reopen pools of 50 connections each at once → thousands of auths. Symptoms:

• CPU spike on auth threads
• remaining connection slots / max_connections errors
• Cascading retries that make recovery longer

Mitigations:

1. RDS Proxy (or PgBouncer) to multiplex and absorb storms.
2. Jittered reconnect backoff at the pool layer.
3. Lower per-task pool size; raise only with measured need.
4. Separate analytical sessions from OLTP pools.

Read replica lag is a correctness bug

Sending read-after-write to a reader endpoint without checking lag causes “missing row” ghosts. Patterns that work:

• Session stickiness to writer for N milliseconds after writes.
• Aurora replica_lag metrics / AuroraGlobalDBRPOLag for global DBs.
• Explicit “read your writes” via writer endpoint for that request path.

Parameter groups and the silent killers

Production outages often come from:

• max_connections set above what memory can handle → OOM under load, not during failover.
• autovacuum / MySQL purge lag after bulk deletes → disk full on replicas.
• Idle transactions holding locks across deploys.

Alarm on FreeableMemory, DatabaseConnections, ReplicaLag, DiskQueueDepth, and login failures — not just CPU.

Checklist before the next GameDay

• Force a failover in staging; measure app recovery time, not only DB “available” time.
• Verify pools recycle on 08xxx / admin shutdown SQLSTATEs.
• Confirm no one hardcoded instance endpoints in Secrets Manager.
• Load test reconnect storms through RDS Proxy.

If you only rehearse happy-path schema migrations, you have not tested the failure mode customers actually feel.