Customer wants all traffic to go to a single IP and come from the same IP. Here are the steps to configure a custom NAT instance to translate outgoing traffic and forward incoming traffic to internal ALB

10-nat-settings.conf

# Enable packet forwarding
net.ipv4.ip_forward = 1
# Disable ICMP redirects
net.ipv4.conf.eth0.send_redirects = 0

README.md

SCENARIO

1. The customer wants to communicate with a single IP in our cloud environment, for any indisputable reason;
2. They want to send requests to our single IP, but their systems also require that we connect to their environment for some reason, and they only allow that SAME IP as the connection source;
3. To complicate things, we need to route different incoming requests to different resources inside our cloud environment;

SOLUTION

Use a custom NAT for incoming and outgoing traffic. The incoming traffic (from internet) is forwarded to an LB. The LB uses a CNAME which may have multiples IPs, but our NAT uses only 1 IP.

DISCLAIMER

This solution is not suitable to High Available environments as our NAT instance is a single point of failure. If the LB is under heavy use, it might scale out and this solution doesn't account for that in two aspects:

1. It doesn't distribute requests to the different IPs of the LB;
2. It doesn't scale up when needed - requires manual intervention;

If you have a better solution, feel free to contribute!

STEPS

1. Create a private subnet;
2. Create a route table for the private subnet;
3. Remove the IGW route from the new route table, but keep the VPC route;
4. Launch a EC2 using the Amazon Linux AMI in which we'll configure as our NAT gateway. Put it in a public subnet as it needs internet access, and associate a role that has the IAM permissions specified in the iam-policy-nat-gw.json policy;
5. Disable the Source/Dest Check for this instance;
6. Copy 10-nat-settings.conf to /etc/sysctl.d/10-nat-settings.conf;
7. Copy configure-pat.sh to /usr/local/sbin/configure-pat.sh;
8. Copy configure-dnat.sh to /usr/local/sbin/configure-dnat.sh;
9. Update the instance and install jq:

 yum update && yum install -y jq

11. Add the following lines to /etc/rc.local:

/usr/local/sbin/configure-pat.sh
/usr/local/sbin/configure-dnat.sh

12. Adjust the permissions:

chmod +x /etc/rc.d/rc.local
chmod +x /usr/local/sbin/configure-pat.sh
chmod +x /usr/local/sbin/configure-dnat.sh

13. Add a route to 0.0.0.0/0 pointing to the nat-gw instance's ENI;
14. Create your internal Load Balancer;
15. Change the LB_NAME value in /usr/local/sbin/configure-dnat.sh (line 4) to the name you gave to your Load Balancer;
16. When you boot the NAT gateway instance, it will automatically retrieve the Load Balancer configuration and configure itself to forward requests to the listener ports.
17. Check if everything was configured properly:

grep vpc /var/logs/message
iptables -t nat -L

configure-dnat.sh

#!/bin/bash
# Configure the instance to forward requests to the specified Load Balancer.
# Please, change the `LB_NAME` value accordingly.

LB_NAME="<YOUR_LB_NAME>"

function log { logger -t "vpc" -- $1; }

function die {
  [ -n "$1" ] && log "$1"
  log "Configuration of DNAT failed!"
  exit 1
}

# Sanitize PATH
PATH="/usr/sbin:/sbin:/usr/bin:/bin"

log "Determining the MAC address on eth0..."
ETH0_MAC=$(cat /sys/class/net/eth0/address) ||
    die "Unable to determine MAC address on eth0."
log "Found MAC ${ETH0_MAC} for eth0."

VPC_CIDR_URI="http://169.254.169.254/latest/meta-data/network/interfaces/macs/${ETH0_MAC}/vpc-ipv4-cidr-block"
log "Metadata location for vpc ipv4 range: ${VPC_CIDR_URI}"

VPC_CIDR_RANGE=$(curl --retry 3 --silent --fail ${VPC_CIDR_URI})
if [ $? -ne 0 ]; then
   die "Unable to retrieve VPC CIDR range from meta-data."
fi
log "Retrieved VPC CIDR range ${VPC_CIDR_RANGE} from meta-data."

log "Determining the region in which the instance is running..."
export AWS_DEFAULT_REGION=$(curl --retry 3 --silent --fail http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region) ||
  die "Unable to determine in which region the instance is running."
log "Detected region ${AWS_DEFAULT_REGION}."

log "Determining the LB configuration..."
LB_CONFIG=$(aws elbv2 describe-load-balancers --names "${LB_NAME}")
LB_ARN=$(echo "$LB_CONFIG" | jq -r '.LoadBalancers[] | .LoadBalancerArn')
LB_ENDPOINT=$(echo "$LB_CONFIG" | jq -r '.LoadBalancers[] | .DNSName')
log "LB arn: ${LB_ARN}"
log "LB endpoint: ${LB_ENDPOINT}"

log "Determining the ALB ipv4 addresses..."
LB_IPV4_ADDRESSES=( $(dig +short ${LB_ENDPOINT}) )
if [ $? -ne 0 ]; then
  log "Unable to resolve ALB ipv4 addresses"
  exit 1
else
  LB_FIRST_IPV4_ADDRESS=${LB_IPV4_ADDRESSES[1]}
  echo -n "ALB ipv4 addresses: ${LB_IPV4_ADDRESSES}" | tr '\n' ';' | log
fi

log "Determining the LB listener ports..."
LB_PORTS=( $(aws elbv2 describe-listeners --load-balancer-arn "${LB_ARN}" | jq -r '.Listeners[] |  .Port') )
log "LB listener ports: ${LB_PORTS[@]}"

for port in "${LB_PORTS[@]}"; do
  log "Configuring DNAT to ${LB_FIRST_IPV4_ADDRESS}:$port..."
  iptables -t nat -A PREROUTING -i eth0 \
    ! -s ${VPC_CIDR_RANGE} -p tcp --dport $port -j DNAT \
    --to ${LB_FIRST_IPV4_ADDRESS}:$port 2> /dev/null ||
      die;
done

log "Configuration of DNAT complete."
exit 0

configure-pat.sh

#!/bin/bash
# Configure the instance to run as a Port Address Translator (PAT) to provide
# Internet connectivity to private instances.

function log { logger -t "vpc" -- $1; }

function die {
    [ -n "$1" ] && log "$1"
    log "Configuration of PAT failed!"
    exit 1
}

# Sanitize PATH
PATH="/usr/sbin:/sbin:/usr/bin:/bin"

log "Determining the MAC address on eth0..."
ETH0_MAC=$(cat /sys/class/net/eth0/address) ||
    die "Unable to determine MAC address on eth0."
log "Found MAC ${ETH0_MAC} for eth0."

VPC_CIDR_URI="http://169.254.169.254/latest/meta-data/network/interfaces/macs/${ETH0_MAC}/vpc-ipv4-cidr-block"
log "Metadata location for vpc ipv4 range: ${VPC_CIDR_URI}"

VPC_CIDR_RANGE=$(curl --retry 3 --silent --fail ${VPC_CIDR_URI})
if [ $? -ne 0 ]; then
   log "Unable to retrieve VPC CIDR range from meta-data, using 0.0.0.0/0 instead. PAT may masquerade traffic for Internet hosts!"
   VPC_CIDR_RANGE="0.0.0.0/0"
else
   log "Retrieved VPC CIDR range ${VPC_CIDR_RANGE} from meta-data."
fi

log "Enabling PAT..."
sysctl -q -w net.ipv4.ip_forward=1 net.ipv4.conf.eth0.send_redirects=0 && (
   iptables -t nat -C POSTROUTING -o eth0 -s ${VPC_CIDR_RANGE} -j MASQUERADE 2> /dev/null ||
   iptables -t nat -A POSTROUTING -o eth0 -s ${VPC_CIDR_RANGE} -j MASQUERADE ) ||
       die

sysctl net.ipv4.ip_forward net.ipv4.conf.eth0.send_redirects | log
iptables -n -t nat -L POSTROUTING | log

log "Disable TCP offloading to improve network performance"
ethtool -K eth0 sg off | log

log "Configuration of PAT complete."
exit 0

iam-policy-nat-gw.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeLoadBalancers",
                "ec2:DescribeInstances",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTargetGroups"
            ],
            "Resource": "*"
        }
    ]
}