Skip to Content

IP Failover with Overlay IP Addresses

 AWS networking allows creating routing entries for routing tables, which direct all traffic for an IP address to an EC2 instance. This concept allows directing the traffic to any instance in a Virtual Private Network (VPC) no matter which subnet it is in and no matter which availability zone (AZ) it is in. Changing this routing entry for the subnets in a given VPC allows redirecting traffic when needed. This concept is known as “IP Overlay” routing in AWS. It is normally being used in a static way for routers and Network Address Translation (NAT) instances. Overlay IP routing can however be used in a dynamic fashion.

diagrams with overlay IP address

The diagram in Figure X shows a network topology in which this concept can get used. Two instances named node1 (EC2 instance i-a) and node2 (EC2 instance i-b) are connected to two different subnets. The two subnets are assigned to the same VPC in two different Availability Zones (AZ). It is not mandatory that both nodes are located in different availalibility zones and subnets, it’s however desirable in many cases. Failover nodes in high availability architectures should be independent of common failure root causes.

Both nodes are part of the same Virtual Private Network (VPC). Both subnets share the same routing table named rtb_A.

The idea is to route traffic from on premises consumers or consumers from within the VPC to the IP address 10.2.0.1 in this case. It’s important that the IP address is outside of the Classless Inter-Domain Routing (CIDR) block of the VPC.

It takes 4 steps to route traffic through an Overlay IP address to EC2 node1 or node2

  1. Create a routing entry in the routing table which sends the traffic to the EC2 instance in question
  2.  Disable the source/destination check for the network traffic to the two instances in the EC2 network management. The AWS network doesn’t by default send network packets to instances which don’t match the normal routing entries
  3. Enable the operating system of the EC2 instances to accept these packets
  4. The two EC2 instances are likely to monitor each other. They are likely to initiate the routing change when needed. The EC2 instances require policies in the IAM roles which authorize them make these changes in the routing table

Creating and managing the routing Entries

The AWS command line interface (AWS-CLI) allows creating such a route with the command:

aws ec2 create-route --route-table-id ROUTE_TABLE --destination-cidr-block CIDR --instance-id INSTANCE

Where as ROUTE_TABLE is the identifier of the routing table which needs to me modified. CIDR is an IP address with the filter. INSTANCE is the node to which the traffic gets directed.

Once the route exists it can be changed whenever traffic is supposed to be routed to a different node with the command:

aws ec2 replace-route --route-table-id ROUTE_TABLE --destination-cidr-block CIDR --instance-id INSTANCE

There are chances if there is a need to delete such a route entry. This happens with the command:

aws ec2 delete-route --route-table-id ROUTE_TABLE --destination-cidr-block CIDR 

It may be as well important to check for the current status of the routing table. A routing table can be checked with this command:

aws ec2 describe-route-tables --route-table-ids ROUTE_TABLE

The output will list all routing entries. The user will have to filter out the line with the CIDR in question.

Disable the Source/Destination Check for the Failover Instances

AWS console, change source destination check

The source/destination check can be disabled through the EC2 console. It takes the execution of the following pull down menu in the console for both EC2 instances (see left).

The same operation can be performed through scripts using the AWS command line interface (AWS-CLI). The following command needs to be executed one time for both instances, which are supposed to receive traffic from the Overlay IP address:

 ec2-modify-instance-attribute EC2-INSTANCE --source-dest-check false

The system on which this command gets executed needs temporarily a role with the following policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1424870324000",
"Effect": "Allow",
"Action": [ "ec2:ModifyInstanceAttribute"],
"Resource": [
"arn:aws:ec2:REGION:ACCOUNT-ID:instance/INSTANCE-A",
"arn:aws:ec2:REGION:ACCOUNT-ID:instance/INSTANCE-B"
]
}
]
}

Replace the individual parameters (bold letters) for the region, the account identifier and the two identifiers for the EC2 instances with the placeholders in bold letters.

Configure the Network Interfaces to receive the Network Traffic of the Overlay IP Address

Linux systems need the overlay IP addresses to be configured as secondary IP address on their standard interface eth0. This can be achieved by the command:

ip address add OVERLAY-IPD/CIDR dev eth0:1

The tools to make the secondary IP address permanent vary across the Linux distributions. Please use the individual documentation to lookup the commands.

Enable the Instances to change the Routes

Switching routes from node to node typically happens in failover cluster. Failover clusters with two nodes monitor each other and take action when the other node doesn’t seem to be alive anymore. The following policy has to be applied to the EC2 instances, which are supposed to monitor each other and be allowed to switch the route when needed:

{
  "Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1424860166260",
"Action": [
"ec2:CreateRoute",
"ec2:DeleteRoute",
"ec2:DescribeRouteTables",
"ec2:ReplaceRoute"
],
"Effect": "Allow",
"Resource": "*"
}
]
}

 

Comments

Access the Overlay IP over VPN

Hi,

First I'd like to say, you blog is fantastic. We've implemented SAPHANASR configuration out lined, and the Overlay IP seems to work fine only when working within AWS. (subnet to subnet)

However, we're having issues directing traffic from our main site over a site to site VPN which leverages the AWS VPN Gateway. Was this part successfully implemented? Anything special had to be done on the main site other that implementing a route to the Overlay IP?

I'm asking to know if this is an issue on our side?

Thanks,

Philippe

Limitations of Overlay IP addresses

Yes,
overlay IP addresses can be reached from inside the VPN only.
You will want to run your users like application servers inside the VPN.
You will need a proxy like an ELB/ALB or a SAP GUI router if you want to reach the Overlay IP address from outside of the VPC.

Regarding ELB as a proxy

Hi Stefan,

Thanks for a wonderful explanation.However i am not still able to understand how ELN/ALB will help n forwarding the traffic to the Overlay IP address.Since ELB has no understanding of the OverlayIP address which is plumbed inside the instance and no where mapped in the (just in the route tables).
So how will ELB/ALB will forward traffic to the Overlay IP address.

ELBs/ALBs

Hi,

this page is based on my daily work.

ELBs are great to load balance http and https protocols.
ELBs are great for web facing applications.

I'm working with legacy ERP applications which use lasting TCP connections and my users want to use everything in a VPC and intranet setup.
My GUI users may be idle for more than an hour (see: http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-id...).



book | by Dr. Radut