VPC NAT Gateway Asymmetric Routing: Why Return Packets Disappear

The problem: packets leave fine, replies never come back

You stand up a private subnet, add a NAT Gateway in a public subnet, update the route table (0.0.0.0/0 → nat-xxx), and outbound HTTPS works. Then you attach a Network Load Balancer, a second AZ, or a VPC peering connection — and suddenly half your flows time out. tcpdump on the instance shows SYN leaving, nothing returning. Security groups look open. NACLs look open. Welcome to asymmetric routing in AWS networking.

How NAT Gateway actually forwards traffic

A NAT Gateway is AZ-scoped and stateful for SNAT. When an ENI in a private subnet sends a packet whose destination matches the default route to the NAT:

1. The packet is rewritten: source IP becomes the NAT’s Elastic IP (or private IP for NAT Gateway with private connectivity in some designs).
2. Return traffic must hit the same NAT that owns that SNAT mapping.
3. The NAT rewrites the destination back to the original private IP and delivers into the VPC.

If the return path lands on a different NAT, peering attachment, or Transit Gateway route that does not know the SNAT mapping, the flow dies. There is no shared connection table across NAT Gateways.

Failure pattern 1: multi-AZ NATs with a shared route

A common “HA” diagram puts a NAT Gateway in every AZ and points all private subnets at one NAT “for cost.” That is fine until that AZ fails — or until you “optimize” by letting each subnet use its local NAT while a stateful appliance or firewall in another AZ inspects return traffic. Keep this rule:

• Outbound path and return path must share the same NAT for that flow.
• Prefer local NAT per AZ (private subnet in AZ-a → NAT in AZ-a) to avoid cross-AZ data processing charges and hairpin latency.

Failure pattern 2: Transit Gateway + NAT “hairpin”

Workloads in Spoke A need internet. Someone attaches a shared egress VPC with NAT and advertises 0.0.0.0/0 from the egress attachment. Traffic goes Spoke → TGW → egress → NAT → internet. Return traffic must reverse: internet → NAT EIP → egress VPC → TGW → Spoke.

Breaks when:

• TGW route tables are segmented and the egress attachment cannot route back to the spoke CIDR.
• Appliance mode is off and TGW hashes flows across ENIs inconsistently with a stateful firewall.
• The spoke still has a local 0.0.0.0/0 → local NAT competing with the TGW default route (longest prefix / more specific routes win — defaults collide in surprising ways across associations).

Enable appliance mode on the TGW attachment that fronts stateful inspection so both directions of a flow pin to the same AZ/ENI.

Failure pattern 3: NLB / PrivateLink source IPs

Network Load Balancers preserve client IP (for TCP) unless you terminate TLS on an ALB. If your target’s security group allows only the NLB subnet CIDRs but you expected the client’s VPC CIDR, health checks pass (NLB nodes) while real clients fail — or the reverse. For PrivateLink, the consumer sees an ENI in their VPC; provider security groups must allow the endpoint subnet, not the consumer’s instance CIDR.

How to debug systematically

1. VPC Flow Logs (ACCEPT/REJECT) on the source ENI, NAT subnet, and TGW attachment subnets. Look for ACCEPT out + missing ACCEPT in.
2. Reachability Analyzer for path intent — great for SG/NACL/route misses, weaker for SNAT state.
3. Confirm route tables: private subnet default route target, egress VPC return routes to spoke CIDRs, no conflicting more-specific routes.
4. Check AZ alignment: instance AZ, NAT AZ, TGW appliance mode.
5. For hybrid: verify on-prem does not advertise overlapping CIDRs that blackhole return traffic.

Design checklist that survives production

• One NAT Gateway per AZ that has private subnets needing egress; route locally.
• Central egress via TGW only with explicit return routes + appliance mode for firewalls.
• Never assume security groups “see” the original client IP through every AWS load balancer mode — verify proxy protocol / client IP preservation docs per service.
• Prefer PrivateLink for east-west private exposure of services instead of opening wide CIDR paths across peering.

Lab idea

Deploy two private instances in different AZs, two NATs, break the return path on purpose (point AZ-b subnet at AZ-a NAT while a NACL denies cross-AZ ephemeral ports), capture Flow Logs, then fix with local NAT routing. That single exercise teaches more than a week of diagram-only study.