Debian network configuration for Proxmox VE server running on a Hetzner host

msys_hetzner-etc_network_interfaces

# /etc/network/interfaces
#
auto lo
iface lo inet loopback

# device: eth0
iface eth0 inet manual

# IPv4 bridge
# (connect ONLY your firewall/router KVM instance here, this is the WAN device!)
auto vmbr0
iface vmbr0 inet static
        # Hetzner primary WAN IP
        address   176.9.xxx.xxx

        # Hetzner primary WAN IP broadcast address
        broadcast 176.9.xxx.xyz

        # This netmask needs to have all bits set
        netmask   255.255.255.255

        # Our gateway is reachable via Point-to-Point tunneling
        # put the Hetzner gateway IP address here twice
        pointopoint 176.9.xxx.yyy
        gateway   176.9.xxx.yyy

        # Virtual bridge settings
        # this one is bridging physical eth0 interface
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

        # add a static route through the Hetzner gateway IP
        # for the subnet our primary IP belongs to
        up route add -net 176.9.xxx.zyx netmask 255.255.255.224 gw 176.9.xxx.yyy vmbr0

        # Add routing for up to 4 dedicated IP's we get from Hetzner
        # You need to 
        up ip route add 176.9.xxx.xx1/32 dev vmbr0
        up ip route add 176.9.xxx.xx2/32 dev vmbr0
        up ip route add 176.9.xxx.xx3/32 dev vmbr0
        up ip route add 176.9.xxx.xx4/32 dev vmbr0

        # Assure local routing of private IPv4 IP's from our
        # Proxmox host via our firewall's WAN port
        up ip route add 192.168.0.0/16 via 176.9.xxx.xx1 dev vmbr0
        up ip route add 172.16.0.0/12 via 176.9.xxx.xx1 dev vmbr0
        up ip route add 10.0.0.0/8 via 176.9.xxx.xx1 dev vmbr0

# IPv6 bridge
# (connect ONLY your firewall/router KVM instance here, this is the WAN device!)
iface vmbr0 inet6 static
        address 2a01:4f8:151:XXX::3
        netmask 64
        up ip -6 route add 2a01:4f8:151:XXX::1 dev vmbr0
        up ip -6 route add default via 2a01:4f8:151:XXX::1

# Virtual switch for DMZ
# (connect your firewall/router KVM instance and private DMZ hosts here)
auto vmbr1
iface vmbr1 inet manual
        bridge_ports none
        bridge_stp off
        bridge_fd 0

# Virtual switch for Private LAN
# (connect your firewall/router KVM instance and private hosts here)
auto vmbr2
iface vmbr2 inet manual
        bridge_ports none
        bridge_stp off
        bridge_fd 0

# Virtual switch for Test Data Center
# (connect your firewall/router KVM instance and private hosts here)
auto vmbr3
iface vmbr3 inet manual
        bridge_ports none
        bridge_stp off
        bridge_fd 0

msys_hetzner-etc_pve_qemu-server_100.conf

# /etc/pve/qemu-server/100.conf
#
# This is an example KVM host configuration I use for my
# Vyatta router instance.
# It was created with the Proxmox web interface and should
# just introduce you to the used network configuration
# (see the 5 network interfaces and their bridge connections).
# That means you won't need to hack the configuration file yourself,
# just keep settings here in mind when creating your virtual host
# in the Proxmox interface.
#
# net0 - WAN interface used for dedicated IPv4 connection handling
# net1 - WAN interface used for dedicated IPv6 connection handling
# net2 - DMZ interface for the majority of my virtual OpenVZ and KVM hosts
# net3 - Private hosts only accessible via VPN
# net4 - Private hosts only accessible via VPN
#
# Of course you could use less interfaces (net0+net1 could be combined
# in a normal dual-stack configuration and net2/3/4 could be one
# interface only if you don't want/need such a complex configuration).
#
# IMPORTANT: net0 uses the MAC address that was assigned by Hetzner to one
# of my additional IPv4 addresses. This is essential to have a working connection.
# I also requested MAC addresses for the other IP's but it's sufficient to have only one of the MAC's
# in your configuration.
#
# Now go and have your firewall instance listening to all your additional
# IPv4 addresses and setup destination NAT to your private IPv4 addressed
# hosts connected to net2.


boot: cd
bootdisk: virtio0
cores: 2
cpu: host
ide2: none,media=cdrom
memory: 2048
name: msys-firewall-instance
# IMPORTANT: net0 has set the MAC address assigned by Hetzner
net0: virtio=00:50:56:00:XX:YY,bridge=vmbr0
net1: virtio=56:4A:05:76:D6:F8,bridge=vmbr0
net2: virtio=76:EE:88:BA:17:DF,bridge=vmbr1
net3: virtio=EE:10:82:D7:3C:14,bridge=vmbr2
net4: virtio=DA:18:23:03:D0:C7,bridge=vmbr3
onboot: 1
ostype: l26
sockets: 2
startup: order=1
virtio0: local:100/vm-100-disk-1.qcow2,cache=writeback

I used you config on a Hetzner server successfully (thanks!) however only the KVMs with public IPs are working. I cannot access the internet from inside KVMs with private IPs.
Any ideas why?

I am guessing I did not provide the correct settings inside the gues KVM. Can you please post an example?

Thank you!!!!

You need a running firewall KVM (like pfSense) which is dual-homed (1 public IP and 1 private IP for each of the other virtual switches) to vmbr0 and vmbr1/2/3 etc. This instance needs to be configured to rewrite and translate your private IP traffic.

Just connect all your private KVM's to vmbr1/2/3 then and use the private IP of your firewall instance as gateway.

Thank you for your gist. Helped much! But I have a problem anyway. Maybe because I have no experience in networking :)

First of all I set up a firewall machine with 2 interfaces: one for vmbr0 and one for vmbr1. Then I installed pfSense on it, and pfSense initiated first interface as WAN interface with my additional hetcner IP and LAN with static IP (192.168.0.1) on the second. Also it started DHCP server for LAN.

When I'm tryng to create new private VM I'm connecting it to vmbr1, but OS installer (ex. Debian) can't see my firewall and it's DHCP. So my new VM become cut from the network. Even if I set network manually and set it's address like 192.168.0.10 and gateway 192.168.0.1 (LAN IP of my firewall) it's still can't connect to the internet or LAN. What I'm doing wrong?

Thanks!

I have problems to undersstand this. Well, my subnet works fine, but I have one single IP that I can't make work.

The most important part is working (subnet IPs), and VMs are working fine. The wiki is clear on this part (http://wiki.hetzner.de/index.php/Proxmox_VE/en).

But need to know how to make extra IP work, I can't handle this and support doesn't seem to explain anything, just repeat something that is inconsistent with the wiki. Probably it's me, that I'm stupid .... but I need to understand.

The wiki says that I must configure the host like you say here in the host: up ip route add 176.9.xxx.xx1/32 dev vmbr0. Okay, but in the KVM connected to vmbr0 is it correct to configure it like this?

iface eth0 inet static
address
netmask 255.255.255.255
pointopoint

gateway

This doesn't work :(

Two years later this is still the only source for the setup with proxmox and a firewall/router as vm.
But now with a firewall build in proxmox 3.4 and new ways to configure the networt in proxmox:
Is this still the best solution? Is this the solution you still use?

Thanks for share this!

This configuration still working, I have use a bit different configuration, but I'm started from this!

What I do with the hetzner root server is use the single ipv4 for management in the physical server and restrict it to be accessed from certain ip, and the ipv6 /64 complete subnet to the pfsense; below the configuration I have use to be done with my porpouse:

auto lo
iface lo inet loopback
iface lo inet6 loopback


auto eth0
iface eth0 inet static
        address 195.xxx.yyy.zzz
        netmask 255.255.255.255 # netmask have to be .255 (/32) unless you want the server talk to neighbors 
        gateway 195.xxx.yyy.zzz # this is the gateway you find just after image restored
        pointopoint 195.xxx.yyy.zzz # insert gateway here! (yes, twice) 

iface eth0 inet6 static
        address aaaa:bbbb:cccc:dddd::2
        netmask 128 # change this to /128 (same reason of before and like this we can assign entire /64 ipv6 to the bridge)
        gateway fe80::1

auto vmbr0
iface vmbr0 inet manual
        # WAN Interface
#      address 195.xxx.yyy.zzz # Same address of physical interface
#      netmask 255.255.255.255 # same reason of before
        bridge_ports none
        bridge_stp off
        bridge_fd 0
        # Route an eventually ipv4 subnet (!NOTE: in hetzner an ipv4 /29 subnet when routed is completely usable)
#        up route add -host 195.xxx.yyy.zz0 dev vmbr0
#        up route add -host 195.xxx.yyy.zz1 dev vmbr0
#        up route add -host 195.xxx.yyy.zz2 dev vmbr0
#        up route add -host 195.xxx.yyy.zz3 dev vmbr0
#        up route add -host 195.xxx.yyy.zz4 dev vmbr0
#        up route add -host 195.xxx.yyy.zz5 dev vmbr0
#        up route add -host 195.xxx.yyy.zz6 dev vmbr0
#        up route add -host 195.xxx.yyy.zz7 dev vmbr0

        # Says the host where the answer for subnet behind pfsense should be routed
        #up ip route add 172.xxx.yyy.zzz/24 via 195.xxx.yyy.zz0 dev vmbr0 #(the ip of via have to be the pfsense ipv4 WAN ip)

iface vmbr0 inet6 static
        address aaa:bbb:ccc:ddd::2
        netmask 126
        # Metric 1 because kernel set up a ipv6 route send the /64 subnet over (::) no next hop
        up ip -6 route add aaa:bbb:ccc:ddd::/64 via aaa:bbb:ccc:ddd::3 dev vmbr0 metric 1 #(the ip of via have to be the pfsense ipv6 WAN ip)

auto vmbr1
iface vmbr1 inet manual
        # LAN INTERFACE 
        bridge_ports none
        bridge_stp off
        bridge_fd 0

In the routed configuration do not need any additional subnet and/or second mac address, if you want to have "just!" the /64 subenet 💃

Thank you so much ... it helped me understand a lot with those two files! 🎁