How to set up static nspawn IP address networking without a bridge

README.md

How to set up static nspawn IP address networking without a bridge

Note - this article assumes you have reasonable familiarity with systemd, networkd and nspawn/machinectl

What is this guide for

There seem to be a lot of references on the internet for nspawn networking with a bridge - but not on simply using the stock veth networking with static addresses. The default behaviour of nspawn (as of 24/03/20 and systemd:245 on Arch) is to:

• Make a virtual ethernet port on the host (defined by /lib/systemd/network/80-container-ve.network)
• Make a virtual ethernet port in the container (defined by /lib/systemd/network/80-container-host0.network)
• Based on those configurations, serve the host port a random IP from the local network ranges with a 28 bit subnet, and a DHCP server for the host port serves a random IP in the subnet to the container

And with that - internet "just works". Ain't that nice

What I was trying to do

Essentially exactly what the default behaviour was but with two key changes

• The host port should have a statically set IP - so that applications on the host expecting output from the container can bind to a specific IP range via their own configs
• Set the container port statically as well - so a forwarding application (like NGINX) knows where to send traffic

No bridges - just virtual ethernet ports. If this sounds like you - read on.

How to do it

First - systemd-networkd will evaluate matches for assinging configs to ports based on alphabetical order. So - we need to inject config before we hit 80-.

To keep things simple - make sure your container is poweroff'd with machinectl.

On the host - copy the /lib/systemd/network/80-container-ve.network into /etc/systemd/network/50-mycontainer.network. Crack it open - it should look like

[Match]
Name=ve-*
Driver=veth

[Network]
# Default to using a /28 prefix, giving up to 13 addresses per container.
Address=0.0.0.0/28
LinkLocalAddressing=yes
DHCPServer=yes
IPMasquerade=yes
LLDP=yes
EmitLLDP=customer-bridge

We need to change the Name in [Match] to be the interface we expect. This is either defined by container name in machinectl to fit in the 16 character limit - or defined by the container config itself with VirtualEthernetExtra to give it a special name.

We also change the Address to be a static IP - I picked something in the 10.0.0.0/8 space, then restricted it to a 28 bit subnet (ie: 10.34.60.49/28 -> 10.34.60.48 - 10.34.60.51). This will sort the host side out.

For the container - we will need to either do

• Very restricted DHCP
• Another static .network config

Neither are amazingly hard

For DHCP - add to the 50-mycontainer.network config

[DHCPServer]
PoolOffset=2
PoolSize=1

This gives the DHCP server 1 address to serve, at 10.34.60.48 + 2 => 10.34.60.50

Doing it this way means that you have all networking controlled by the host - the container is fully stock systemd-networkd.

However - we can also do what we did for the host in the container. Start your container, log into it as root and copy /lib/systemd/network/80-container-host0.network to somewhere sane. It should look like

[Match]
Virtualization=container
Name=host0

[Network]
DHCP=yes
LinkLocalAddressing=yes

[DHCP]
UseTimezone=yes

We need to remove DHCP and instead define an Address=10.34.60.50/28, plus a Gateway=10.34.60.49 if you want this to talk to the net/wider network. I still haven't figured out how to get systemd-resolved happy with this configuration however - workaround could be just to point it directly at an internet DNS.

In summary

This should give you a predictable pair of IPs for things to talk over! systemd as always has great documentation - so if the internet is still turning up nothing its best to sit with a good cup of tea and just read, reread and reread it again.

So I've been fighting with this quite a while, but finally figured it out. So when you're running a nspawn container in private networking mode (the service is started with --network-veth in /etc/systemd/system/[email protected], which is configured by default), then two things must be configured:

NOTE: This assigns a static IP to an individual container.

On the host

Configure the host interface which will be exposed to the container.

$ sudo cp /lib/systemd/network/80-container-ve.network /etc/systemd/network/80-my-container.network

Modify the new file as desired:

$ cat /etc/systemd/network/80-my-container.network
[Match]
Name=ve-my-container
Driver=veth

[Network]
Address=192.100.100.1/24
LinkLocalAddressing=yes
DHCPServer=yes
IPMasquerade=yes
LLDP=yes
EmitLLDP=customer-bridge

Note that Name in [Match] must match the container interface displayed to you by ip link (remove the @if2 suffix):

$ ip link
1: ...
2: ...
3: ...
4: ve-my-container@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether f2:93:48:9b:99:8c brd ff:ff:ff:ff:ff:ff link-netnsid 0

Then specify your desired, static IP address in [Network], which represents the GATEWAY exposed to the container, NOT the container itself. I chose 192.100.100.1/24. Some other fields could probably be removed, but I'm leaving it as is.

Finally, restart the networking service:

$ sudo systemctl restart systemd-networkd

In the container

Set the static IP address of the container, and specify the Host/Gateway IP.

$ sudo cp /lib/systemd/network/80-container-host0.network /etc/systemd/network/80-container-host0.network

Modify the new file as desired:

$ cat /etc/systemd/network/80-container-host0.network
[Match]
Virtualization=container
Name=host0

[Network]
DNS=1.1.1.1
Address=192.100.100.10
Gateway=192.100.100.1

Address specifies the static IP address of the container, where I chose 192.100.100.10 (*.10 is the container IP, *.1 is the host/gateway IP). Set a custom DNS, such as 8.8.8.8 (Google) or 1.1.1.1 (Cloudflare). Note that Gateway MUST match the IP address as specified in the host, as described above. Now, exit the container and restart it (from the host):

$ sudo machinectl reboot my-container

The container should now have a static IP address. For convenience, add it to /etc/hosts (which simplifies SSH access, for example):

192.100.100.10 my-container

Note about firewalls:

If your host has a (passive) firewall enabled, an additional rule is required (compatible with ufw). Make sure you persist that rule:

$ sudo iptables -A FORWARD -i ve-+ -o internet0 -j ACCEPT

Note that internet0 is the interface to the internet, from the host. In my case, it's enp82s0.

More info: https://wiki.archlinux.org/index.php/Systemd-nspawn#Use_a_virtual_Ethernet_link

@lamafab : 192.100.100.x belongs to public IP-space. You might want to switch it to 192.168.100.x