Running into a situation where the VPC networks you manage have overlapping CIDR blocks? Read on!

Problem

Anyone can run into this type of problem due to the following reasons:

• Inheriting new infrastructure via mergers
• Not accounting correctly for multiple networks in a multi-account AWS setup
• A service provider with an IP range must provide access to two different customers that have the same IP range

In our case, it was due to acquiring a company and inheriting their infrastructure so this article focuses on the solutions we explored to solve this problem.

Note: While the rest of the article focuses on AWS-specific services and terms, the solutions described here apply to any cloud provider or self-hosted setups.

Available solutions

Now, depending on your use case, multiple solutions can be available to solve it. However, what matters is which is the most scalable and low effort in terms of implementation.

According to documentation from AWS, there are a few options that I’ll summarize below:

Renumber IP networks

Simply put, it means moving your resources/hosts from the problematic VPC (network) to a new one. This is the recommended solution from AWS.

AWS PrivateLink

This leverages the internal network (hyperplane-based service) to establish connectivity between VPCs via network-level load balancers. It is the most straightforward option as it doesn’t require any changes to the network address scheme.

Use multiple IP address ranges in VPC

If your application* is broken into front-end and back-end tiers (middleware, database, cache, etc.), then you can have a set of front-end subnets with non-overlapping IP addresses while the backend subnets overlap. This option is similar to option #1, but your work is reduced here as just the front-end subnets need to be moved over.

The trick is to use AWS transit gateway and control the advertisement of the front-end subnets accordingly so that end users can reach them.

Hide subnets using a private NAT gateway

Private NAT gateways allow you to hide the VPC overlapping with another network by deploying it in another non-overlapping subnet so that NAT can translate the IP addresses of the workloads residing in the overlapping VPC.

While AWS recommends option #1 — renumbering the network, it is by far the most intensive in terms of effort as we need to migrate all workloads in the VPC to a new one and if it’s production there’s added risk and cost associated.

Whereas option #3 is only applicable to a specific set of applications that have separated front-end/back-end tiers.

We liked option #2 the most, i.e., PrivateLink, as it’s low effort, simple, highly scalable, and has built-in redundancy with one caveat, but it didn’t apply to our stack.

What solution we ended up with

Before beginning the implementation of option #2, we did a deep dive into the overlapping CIDR blocks for the networks in question and did some calculations — yes, think Excel sheets and all.

We realized that while CIDR blocks overlap at the VPC level, the subnet CIDRs do not & are mutually exclusive

This meant we could easily create peering between VPC A and the required subnets (total of 3, one per AZ) in VPC B and update the route tables accordingly for each subnet.

Peering with specific routes ended up being an ideal solution for us as it allowed for network connectivity between the VPCs residing in different AWS accounts without requiring us to make any changes to the existing networking setup and eliminated the risk of this change affecting our remaining application networks.

Note: We also verified the aforementioned approach with AWS support ✅

How did we test network connectivity?

• We first created a separate test VPC in the target account B, which mimicked the problem we encountered where the CIDR block overlapped with one of our VPCs in account A
• Next, we set up peering between the two VPCs both ways
• Created two EC2 instances, one in each VPC
• Tested for connectivity by pinging/telnet the private IP from EC2 A → EC2 B and vice versa

Once the test was successful, we rolled out the change to production & it worked as expected 🎉

Takeaways

Before evaluating all the above options to solve the overlapping CIDR block problem, dive deeper and calculate if the CIDR blocks overlap at the subnet level. Since it’s possible that some VPCs are over-generously allocated, thereby blocking a huge IP address space, and therefore the subnets within the VPC barely use the available IPs.

If there is indeed a direct overlap at the subnet level, then we suggest going with the PrivateLink option. It’s the ideal solution in terms of effort, scale, and redundancy.

Bonus FAQ

• Q: Given two VPCs, A and B, which can talk to each other and CIDR blocks do not overlap and given VPC C, where CIDR block overlaps with VPC A. Can we establish connectivity between VPC C to B?
• A: Yes, as long as they are two separate VPC peering/connections, it is possible for VPC C to talk to VPC B as VPC's peering is not transitive. However, if it’s one large open network (transit gateway), you must choose from the options above.

Dev works as a Lead SRE on the platform infrastructure team at VTS. He is passionate about building developer tooling, reliable systems & driving continuous improvements.