This concept is based on the fact that multiple EC2 instances are reachable through their standard network interface (eth0) and an additional IP address that is used to provide the high available service. The high available service IP address has to belong to a different subnet. This IP address gets modeled with an AWS Elastic Network Interface (ENI). The ENI gets detached from an EC2 instance when the instance fails, it gets then attached to the EC2 instance which is supposed to take a service over.

The Elastic Network Interface with the name eni-x is currently attached to instance i-a and it can be attached on demand to instance i-b. The highly available service is provided through subnet A. The two service providing instances i-a and i-b can be reached through subnet B with their standard network interfaces (eth0). This architecture requires at least two subnets: A service provider subnet (here A) A subnet which allows access to the normal (eth0) interfaces of the two instances (here B) It takes the following steps to make such a failover scenario work Create and configure an ENI Enable the Linux instances to accept traffic and to send traffic back through the dynamically attached ENI Create policies for the two instances to allow them pull over the dynamic IP address Execute the appropriate AWS commands to detach and reattach the ENI

Create and configure an Elastic Network Interface (ENI)

Prerequisites to create an Elastic Network Interface are to have the following information:

• Name of the AWS subnet in which the IP address fits
• IP addresses which matches the CIDR of the AWS subnet
• A security group for the ENI, which allows the required service, protocols to pass. The allowed protocols to pass are typically a subset of the protocols one would use for the primary interface in a non-high-available configuration.

The creation can be done manually using the AWS console. You have to choose the EC2 console. Select the entry “Network Interfaces” in the left column. Click on “Create Network Interface” and you’ll see a dialog that looks like the one to the left.

Enter the required information. The ENI will be created. It’ll have a unique AWS-identifier in the form eni-XYZ,

The alternative is to use the command line with the AWS-CLI tools . The equivalent AWS-CLI command is:

create-network-interface --subnet-id <value>[--description <value>] [--private-ip-address <value>]

Network Configuration for the Linux Instances

Linux instances have to learn that they have to return the network traffic through the new network interface (eth1) once it is attached. It takes a number of instance specific routing changes once the interface gets attached. It’s important to undo these routing changes after the secondary network interface (eth1) gets detached.

The scripts below work for SLES 11 SP3 instances other Linux distributions will need the routing entries to be performed in different network configuration scripts.

/etc/sysconfig/network/scripts/ifup.local.eth1

The script below needs to be adopted by replacing the two following variables, which are printed in bold letters

• DEFAULT-SUBNET-CIDR: This would be 10.1.0.0/8 according to the network diagram above
• DEFAULT-ROUTER: The default router for the default subnet. This would be 10.1.0.1 for the network diagram from above:

#!/bin/bash
if [ "$1" = 'eth1' ]
then
ip route flush table MYHA
ip rule add from DEFAULT-SUBNET-CIDR table MYHA priority 100
ip route add default via DEFAULT-ROUTER dev eth1 table MYHA
fi

This script needs to be executable. A root user will have to perform this command to achieve this:

chmod +x /etc/sysconfig/network/scripts/ifup.local.eth1

/etc/sysconfig/network/scripts/ifdown.local.eth1

The script below needs to be adopted a replacing the variable shown in bold letters

• DEFAULT-SUBNET-CIDR: This would be 10.1.0.0/8 according to the network diagram above

#!/bin/bash
if [ "$1" = 'eth1' ]
        then
        ip route flush table MYHA
        ip rule del from DEFAULT-SUBNET-CIDR table MYHA priority 100 fi

This script needs to be executable. A root user will have to perform this command to achieve this:

chmod +x /etc/sysconfig/network/scripts/ifdown.local.eth1

Linking the Scripts to the right Directories

The scripts above need to be found by the help of soft links which have to be created by a root user the following way:

cd /etc/sysconfig/network/if-down.d
ln –s ../scripts/ifdown.local.eth1 ifdown.local.eth1
cd /etc/sysconfig/network/if-up.d
ln –s ../scripts/ifup.local.eth1 ifup.local.eth1

Adding an additional Routing Table

The scripts from this section will need an additional routing table. This table can be declared with the following command getting executed by a root user:

echo "100 MYHA" >> /etc/iproute2/rt_tables

Policies needed to Detach and Attach ENIs to EC2 Instances

It’s common that two highly available nodes monitor the other one and take action when the monitored node fails. It takes the following policy to enable a node to perform the required AWS configuration change. Attach this policy to all nodes which are supposed to change the network configuration:

{ 
  "Statement": [
      {
       "Sid": "Stmt1346888659253", 
       "Action": [
           "ec2:AttachNetworkInterface",
           "ec2:DescribeInstances",
           "ec2:DescribeInstanceStatus",
           "ec2:DescribeNetworkInterfaces",
           "ec2:DetachNetworkInterface",
           "ec2:DescribeNetworkInterfaceAttribute",
         ], 
         "Effect": "Allow",
         "Resource": [ 
             "*" 
         ] 
      } 
    ] 
}

Scripts to detach and reattach an ENI

The script getInterface.sh below is an example of how the AWS-CLI can be used to dynamically attach an Elastic Network Interface. It requires the dynamic IP to be entered as first command line parameter. The second command line parameter is the primary IP address of the system, which will then host the dynamic address.

The script

1. Identifies the name of the ENI by using it’s IP address
2. it determines the EC2 instance from which the ENI needs to be detached
3. it detaches the ENI
4. it waits until the operation has completed
5. it attaches the ENI to the second system once it’s available

#!/bin/bash
# This scripts detaches as secondary interface from an instance.
# It then attaches the interface to the instance where it has been executed
# 
# Command line parameter
#=======================
# First parameter: IP address which needs to be detached and moved to a
# different instance 
echo "Move IP adress: $1 to system with primary IP address $2" 
INTERFACE=`ec2-describe-network-interfaces | grep $1 | \
awk /NETWORKINTERFACE/'{print $2 }'` 
TONODE=`curl -silent http://169.254.169.254/latest/meta-data/instance-id` 
echo "move eni: $INTERFACE to instance id: $TONODE " 
DETACH=`aws ec2 describe-network-interfaces --network-interface-ids $INTERFACE | \
awk /ATTACHMENT/'{print $3 }'` INTERFACESTATUS=`aws ec2 describe-network-interfaces --network-interface-ids $INTERFACE | \
awk -F"\t" /NETWORKINTERFACE/'{print $10 }'`
echo "$DETACH to be detached. Current interface status is $INTERFACESTATUS"
aws ec2 detach-network-interface --attachment-id $DETACH --force
echo "Command to detach Interface $INTERFACE submitted"
while [ "$INTERFACESTATUS" = 'in-use' ]
do
 echo "Will sleep 1 second"
 sleep 1
 INTERFACESTATUS=`aws ec2 describe-network-interfaces --network-interface-ids $INTERFACE | \
 awk -F"\t" /NETWORKINTERFACE/'{print $10 }'`
Done
echo "Will attach interface $INTERFACE to $TONODE "
aws ec2 attach-network-interface --instance-id $TONODE --network-interface-id $INTERFACE

 More Resources

• AWS article: Leveraging Multiple IP Addresses for Virtual IP Address Fail-over in 6 Simple Steps