Notes about using point-to-point links for routing IP addresses in virtual machines

transfer-network-config

Some experiments, which worked but were eventually not used.
See http://www.greenhills.co.uk/2011/06/10/lxc.html

Now, the next big milestone is to get networking working.
In the config, we told lxc-create to use br0 as the link,
and inside the container we have eth0:

    root@thunder:/# brctl show br0
    bridge name	bridge id		STP enabled	interfaces
    br0		8000.962e6bb72480	no		vethP70LNx
    root@thunder:/# ip link list br0
    7: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
        link/ether 96:2e:6b:b7:24:80 brd ff:ff:ff:ff:ff:ff

    root@natty1:/# ip link list eth0
    68: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
        link/ether ee:4d:fd:34:c2:59 brd ff:ff:ff:ff:ff:ff

By way of initial test I did a simple internal routing setup akin to [this description](http://www.mail-archive.com/[email protected]/msg01969.html),
to a RFC1918 address on the container.

    root@thunder:/# ifconfig br0 up
    root@thunder:/# ip route add 192.168.0.0/24 dev br0
    root@thunder:/# ip addr add dev br0 172.16.0.1/24 brd +
    root@thunder:/# ip route add to 192.168.0.99 via 172.16.0.2
    root@thunder:/# echo 1 > /proc/sys/net/ipv4/ip_forward

    root@natty1:/# ip addr add 192.168.0.99/32 dev eth0
    root@natty1:/# ip route add default via 172.16.0.1
    root@natty1:/# ping 46.43.35.202
    PING 46.43.35.202 (46.43.35.202): 48 data bytes
    56 bytes from 46.43.35.202: icmp_seq=0 ttl=64 time=0.047 ms

    root@thunder:/# ping 192.168.0.99
    PING 192.168.0.99 (192.168.0.99) 56(84) bytes of data.
    64 bytes from 192.168.0.99: icmp_req=1 ttl=64 time=9.26 ms

and after copying /etc/resolv.conf from the controlling host to the container and
starting sshd in the container (<code>service sh start</code>) I could <code>ssh 192.168.0.99</code> in.

Now with the actual IP addresses I've been allocated by my ISP:

    root@natty1:/# ip addr add 46.43.55.73 dev eth0

    root@thunder:/# ip route add to 46.43.55.73 via 172.16.0.2

This allows me to ssh in over the internet.
Next problem: making outbound connections (e.g. ping www.google.com) fails.
tcpdump on the controlling host shows:

    14:15:13.182730 IP 172.16.0.2 > ez-in-f105.1e100.net: ICMP echo request, id 49409, seq 512, length 56

Clearly that's not going to work; it needs to have 46.43.55.73.
You can force that with (from the console, not an ssh session):

    root@natty1:/# ip route del default dev eth0
    root@natty1:/# ip route add default via 172.16.0.1 dev eth0 src 46.43.55.73

or by the looks of it by making the internal address non-global:

    root@natty1:/# ip addr del 172.16.0.2/4 dev eth0
    root@natty1:/# ip addr add 172.16.0.2/4 brd + scope link dev eth0

    root@natty1:/# ip route del default 
    root@natty1:/# ip route add default via 172.16.0.1 dev eth0    
            
    root@natty1:/# ping -q -c 1 www.google.com 
    PING www.l.google.com (209.85.146.103): 48 data bytes
    --- www.l.google.com ping statistics ---
    1 packets transmitted, 1 packets received, 0% packet loss
    round-trip min/avg/max/stddev = 13.763/13.763/13.763/0.000 ms

This is all good, but do I need Proxy-ARP here?
It appears not: if I ping 46.43.55.74 from the internet the hosting provider duly
forwards it to the main IP address, as shown by tcpdump on the controlling host.

So, let's start with a new container, natty2 on 46.43.55.73,
and copy some more in, and optimise some.

NAME=natty2
IP=46.43.55.73
TRANSFER_SOURCE=172.16.73.2
TRANSFER_ROUTER=172.16.73.1

ip addr add dev br0  $TRANSFER_ROUTER/24 brd +
ip route add to $IP via $TRANSFER_SOURCE
echo 1 > /proc/sys/net/ipv4/ip_forward

TRANSFER_MASK=255.255.255.0
LXCDIR=/var/lib/lxc
ROOTFS=$LXCDIR/${NAME}/rootfs
CONFIG=/root/lxc-${NAME}-config.tmp
cat > $CONFIG <<EOM
lxc.network.type = veth
lxc.network.link = br0
lxc.network.name = eth0
EOM


cat > $ROOTFS/etc/network/interfaces <<EOM
auto lo
iface lo inet loopback

# public routable address
auto eth0
iface eth0 inet static
  address $IP
  netmask 255.255.255.255

# transfer network
auto eth0:0
iface eth0:0 inet static
  name transfer network
  address $TRANSFER_SOURCE
  netmask $TRANSFER_MASK
  pointopoint $TRANSFER_ROUTER
  post-up ip route add default via $TRANSFER_ROUTER dev eth0 src $IP
  pre-down ip route del default dev eth0
EOM