wireguard, wireguard layer 2, wireguard over TCP

wireguard_layer2.md

Intro

This note describes how to connect two networks/devices/VMs over public network using Wireguard with Layer 2 support (ARP, IPv6 link-local, etc).

This can also be achieved using SSH and its "tap" tunnel, however, it does not provide the same level of latency and bandwidth as full-blown VPN such as Wireguard.

In addition, this note describes how to tunnel Wireguard over TCP connection. This may be of use if you encounter firewall in-between so, for instance, you can use TCP port 443 only.

Objective

Let's draw a network diagram of what we would like to have in as a result.

 Host on private LAN                                         Host on the Internet
 +---------------------------------+                       +-------------------------------+
 |   brtun bridge                  |                       |                  brtun bridge |
 | +-------------+                 |                       |                 +--------+    |
 | | ethX gretun |<->wg0<->udp2raw | <-Internet, TCP 443-> | udp2raw<->wg0<->| gretun |    |
 | +-------------+                 |                       |                 +--------+    |
 +---------------------------------+                       +-------------------------------+
 
 brtun: 192.168.0.200/24                                    brtun: 192.168.0.50/24
 wg0: 12.12.12.2/24                                         wg0: 12.12.12.1/24
 gretun: 12.12.12.2->12.12.12.1                             gretun: 12.12.12.1->12.12.12.2
 

Setting things up

Distinguishing between server and client here is based on the answer to the question: who can connect to Wireguard listener socket.

Server

Generating a key:

wg genkey | tee wgkeyprivs | wg pubkey > wgkeypubs

Creating a Wireguard interface, setting the private key and a unique private IPv4 address:

ip l a wg0 type wireguard
wg set wg0 private-key ./wgkeyprivs
ip a a 12.12.12.1/24 dev wg0

In case we want Wireguard over TCP, we have to decrease MTU:

ip l set dev wg0 mtu 1200

Configuring peer's public key and setting interface up (you can restrict allowed-ips to the range you want):

ip l set dev wg0 up
wg set wg0 listen-port 51820 peer $(cat wgkeypubc) allowed-ips 0.0.0.0/0

Creating GRETAP interface which will provide Layer 2 over our Wireguard tunnel:

ip l a gretun type gretap local 12.12.12.1 remote 12.12.12.2
ip l s gretun up

Configuring a bridge which includes GRETAP interface. This step is in fact optional, but it allows you some flexibility: you can assign an IP address to this bridge interface and/or you can add additional interfaces to this bridge.

ip l add name brtun type bridge
ip l set dev brtun up
ip l set gretun master brtun

Let's assign an IP address from the target network to this bridge:

sudo ip a a 192.168.0.51/24 dev brtun

If we want Wireguard over TCP, launch UDP2RAW listener which will do its magic. You can get UDP2RAW from this Git repo: https://github.com/wangyu-/udp2raw-tunnel/releases/

./udp2raw_amd64 -s -l YOUR_PUBLIC_IP:443 -r 127.0.0.1:51820 -a --fix-gro

Client

Client is very similar to the server described above.

Generating a key:

wg genkey | tee wgkeyprivc | wg pubkey > wgkeypubc

Creating a Wireguard interface, setting the private key and a unique private IPv4 address:

ip l a wg0 type wireguard
wg set wg0 private-key ./wgkeyprivc
ip a a 12.12.12.2/24 dev wg0

In case we want Wireguard over TCP, we have to decrease MTU:

ip l set dev wg0 mtu 1200

Configuring peer's public key, endpoint address and setting interface up (you can restrict allowed-ips to the range you want):

ip l set dev wg0 up
wg set wg0 listen-port 51820 peer $(cat wgkeypubs) allowed-ips 0.0.0.0/0 endpoint YOUR_PUBLIC_IP:51820 persistent-keepalive 15

In case you want Wireguard over TCP, set endpoint address to a localhost interface:

wg set wg0 listen-port 51820 peer $(cat wgkeypubs) allowed-ips 0.0.0.0/0 endpoint 127.0.0.1:7777 persistent-keepalive 15

Creating GRETAP interface which will provide Layer 2 over our Wireguard tunnel:

ip l a gretun type gretap local 12.12.12.2 remote 12.12.12.1
ip l s gretun up

Configuring a bridge which includes GRETAP interface.

ip l add name brtun type bridge
ip l set dev brtun up
ip l set gretun master brtun

Let's add a physical interface belonging to a LAN network to the bridge and start DHCP client:

ip l set ethX master brtun
dhcpcd ethX

If we want Wireguard over TCP, launch UDP2RAW listener on the localhost interface which connects to the corresponding public listener.

./udp2raw_amd64 -c -l 127.0.0.1:7777 -r YOUR_PUBLIC_IP:443 -a --fix-gro

Results

That's it. Now you will have Layer 2 connectivity between two separated machines over Wireguard VPN (optionally, over TCP). Enjoy your ARP and IPv6 link-local experience.

References

• https://notes.superlogical.ch/pages/note_wg/nolayer2/
• https://lafibre.info/vpn/tunnel-gre-sur-wireguard/&prev=search&pto=aue

@AndrewL733 , Great explanation. But I did notice drastic difference when I bumped up the MTU of wg interface to 1550. I'm testing in a lab environment were all the devices have 10GBPS NIC. When wg0 interface is left to default of 1420 I got around 2GBPS of throughput. But after changing it to 1550 I'm getting throughput around 4GBPS.https://www.reddit.com/r/networking/comments/19ecn6v/comment/kjeetzs/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button redditor put dose similar with VXLAN and wg and has got 870+ on a 1GBPS link. I'm expecting atlease a throughput of 6GBPS with this setup. I hope I'm not missing anything.

Thanks

@Gaitonde007
Im not an expert but in general lines a big MTU can cause packet fragmentation and a little one to much overload.
So... probably at that fast speeds put a more tiny MTU like 1420 make load in the network and you are only able to get 2 GBPS, and a bigger one, allow to get 4 GBPS.
Take in mind the kind of traffic it's traveling too, it's not the same UDP packets than for example TCP.

Reading about what Andrew test, the default 1420 don't work very well like he said for navigation, so probably after testing get that 1500 it's the lowest working MTU quote:

The default MTU that gets created for the GRETAP interface that we named "gretun" is too low and even with the "ignore-df" and "nopmtudisc" options. Setting the MTU to 1500 ensures that all normal network packets will get through.

So... if you need more throughput can test changing the MTU, at the end of the day need to be set based on the network equipment and topology.
But it's a general line opinion probably a person with more experience can tell.

@AndrewL733 thank you very much for the tutorial, I have finally been able to connect my two homes!

I have the following problem, I'm wondering if someone can help me:

• In both homes A and B I have a router that broadcasts IPTV multicast content.
• I have multicast receivers in both homes.
• I want the multicast receiver in apartment B to receive content from apartment A.
• Receivers and routers use IGMPv2.
• On receiver B I have set a static IP with gateway to router A

I can't get it to work, in the firewalls of the two wireguard servers I have allowed all incoming/outgoing traffic to 224.0.0.0/4

for fake tcp,try phantun or mimc(debian 12 or kernel 6.1+),faster if you want.And I prefer use vxlan via wg.

@AndrewL733 - Your guide is really good, and it works. :) However, I do have an issue, and I'm a bit lost.

I have tried it between 2 data centers. When I install 2 Windows servers, one on each site (Site A and Site B), they do timeout when pinging each other. I did manage to get a speed test with over a Gigabit connection through the tunnel, but after the test, the connection dies. Then I need to reboot the WireGuard "switches" and wait 3-10 minutes to have a somewhat ok connection again. I have tried to change the MTU and also did an MTU test through the tunnel, but with no luck.

I can see some have commented on the high throughput and suggested changing the MTU. Has anyone figured it out? :)

The point of this for me is because I need to migrate some servers from one data center to another, and if I could make a big "switch", it would be nice to just take parts of the environment at a time.

I will paste my configuration down here, for Site A and Site B, and I hope you, or maybe someone else, have experienced the same issue and maybe have a solution.

Thank you for your time :)

Site A

#Ubuntu 22.04 LTS
#Remember to have Promiscuous mode, MAC address changes, and Forged transmits enabled on the interfaces.
#Change to root.
sudo su -
apt purge snapd -y;apt update -y ; apt full-upgrade -y;apt install wireguard bridge-utils openvswitch-switch-dpdk traceroute net-tools -y
reboot now
#Login again
sudo su -
wg genkey | tee wgkeyprivs | wg pubkey > wgkeypubs
cat wgkeyprivs
cat wgkeypubs
Private: SiteAPrivateKey
Public: SiteAPublicKey

nano /etc/netplan/00-installer-config.yaml
#Change the netplan configuration. Modify it for the respective interfaces, IPs, and WireGuard keys.
# This is the network config written by 'subiquity'
network:
  ethernets:
    #Uplink/switching interface 10.76.0.0/24
    ens192:
      dhcp4: false
      dhcp6: false

  bridges:
    br1:
      dhcp4: false
      dhcp6: false
      #IP to Wireguard VM
      addresses: [10.76.1.10/24]
      #DNS - This can be changed to the Domain Controller if needed
      nameservers:
        addresses:
          - 1.1.1.1
          - 1.0.0.1
      routes:
        - to: default
          via: 10.76.1.1
      interfaces:
        - ens192
      parameters:
        stp: false

  tunnels:
    wg0:
      mode: wireguard
      port: 51820
      #Site A Privatekey
      key: SiteAPrivateKey
      addresses:
        - 172.16.0.1/24
      peers:
        - allowed-ips: [172.16.0.0/24]
          endpoint: SiteB_WAN:51820
          #Peers public key
          keys:
            public: SiteBPublicKey
          keepalive: 90
  version: 2

#Activate Layer2
#Create bash script, with execute permission.
nano ~/gretun_cmd
  #!/bin/bash
  ip link add gretun1 type gretap local 172.16.0.1 remote 172.16.0.2 ignore-df nopmtudisc
  ip link set gretun1 up
  ip link set gretun1 master br1
  ip link set gretun1 mtu 1500
chmod u+rwx,g+r-wx,o-rwx ~/gretun_cmd
bash ~/gretun_cmd

#Create bash script, to Block DHCP Req over WireGuard VPN. 
nano ~/block_dhcp_cmd
  #!/bin/bash
  ebtables -A INPUT --in-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
  ebtables -A INPUT --in-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
  ebtables -A FORWARD --out-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
  ebtables -A FORWARD --out-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
chmod u+rwx,g+r-wx,o-rwx ~/block_dhcp_cmd
bash ~/block_dhcp_cmd

#Now Layer2 is active

Site B

#Ubuntu 22.04 LTS
#Remember to have Promiscuous mode, MAC address changes, and Forged transmits enabled on the interfaces.
#Change to root.
sudo su -
apt purge snapd -y;apt update -y ; apt full-upgrade -y;apt install wireguard bridge-utils openvswitch-switch-dpdk traceroute net-tools -y
reboot now
#Login again
sudo su -
wg genkey | tee wgkeyprivs | wg pubkey > wgkeypubs
cat wgkeyprivs
cat wgkeypubs
Private: SiteBPrivateKey
Public: SiteBPublicKey

nano /etc/netplan/00-installer-config.yaml
#Change the netplan configuration. Modify it for the respective interfaces, IPs, and WireGuard keys.
# This is the network config written by 'subiquity'
network:
  ethernets:
    #Uplink interface 10.76.2.0/24. This allows connection through WireGuard, with traffic passing through this interface.
    ens7:
      dhcp4: true
      dhcp6: false

    #Switching interface for bridge br1 10.76.1.0/24 
    #This is a dummy interface, BUT STILL REQUIRING Promiscuous mode, MAC address changes, and Forged transmits TO BE ACTIVE ON THE INTERFACE.
    #No gateway should be configured, as it goes through the WireGuard tunnel.
    ens3:
      dhcp4: false
      dhcp6: false

  bridges:
    br1:
      dhcp4: false
      dhcp6: false
      addresses: [10.76.1.11/24]
      #DNS - This can be changed to the Domain Controller if needed
      nameservers:
        addresses:
          - 1.1.1.1
          - 1.0.0.1
      #No routing needs to be configured; everything is routed through the WireGuard tunnel.
#      routes:
#        - to: default
#          via: 10.76.1.1
      interfaces:
        - ens3
      parameters:
        stp: false

  tunnels:
    wg0:
      mode: wireguard
      port: 51820
      #Site B Private key
      key: SiteBPrivateKey
      addresses:
        - 172.16.0.2/24
      peers:
        - allowed-ips: [172.16.0.0/24]
          endpoint: SiteA_WAN:51820
          #Peers public key
          keys:
            public: SiteAPublicKey
          keepalive: 90
  version: 2

#Activate Layer2
#Create bash script, with execute permission.
nano ~/gretun_cmd
  #!/bin/bash
  ip link add gretun1 type gretap local 172.16.0.2 remote 172.16.0.1 ignore-df nopmtudisc
  ip link set gretun1 up
  ip link set gretun1 master br1
  ip link set gretun1 mtu 1500
chmod u+rwx,g+r-wx,o-rwx ~/gretun_cmd
bash ~/gretun_cmd

#Create bash script, to Block DHCP Req over WireGuard VPN.
nano ~/block_dhcp_cmd
  #!/bin/bash
  ebtables -A INPUT --in-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
  ebtables -A INPUT --in-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
  ebtables -A FORWARD --out-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-destination-port 67:68 -j DROP
  ebtables -A FORWARD --out-interface gretun1 --protocol ipv4 --ip-protocol udp --ip-source-port 67:68 -j DROP
chmod u+rwx,g+r-wx,o-rwx ~/block_dhcp_cmd
bash ~/block_dhcp_cmd

#Now Layer2 is active

Can someone help me out here. This is what I am trying to accomplish.

Server in data center

• Linux/Ubuntu single interface public ip. I can add more interfaces/vlans if needed but would like to keep this as simple as possible.

Server at home/lab (proxmox/vmware)

• Home network is like others a public ip -> router/gateway (ubuquiti unifi) -> nat/lan
• VM with two interfaces. eth0 is 192.168.40.X/24 and has outbound network access to connect to the server at DC
• Eth1 is a blank interface that I want to bridge to a vswitch. From that vswitch I want to spin up other vms and use some of the public ips from my server network.

So once l2/wg is setup. Ideally all I would need to do is spin up a fresh server. Select that bridged vswitch and could use the public ips/gw's as if the vm was running in the data center. Obviously the traffic is going to be backhauled through wg/l2/vpn.

Is this possible? Can any assist me with a modified guide? Is wireguard the best tool for this? Gre tunnel instead? I am open to suggestions. Please see below picture example.

Image backup url https://imgur.com/a/KJc6kdM

Can someone help me out here. This is what I am trying to accomplish.

Server in data center

• Linux/Ubuntu single interface public ip. I can add more interfaces/vlans if needed but would like to keep this as simple as possible.

Server at home/lab (proxmox/vmware)

• Home network is like others a public ip -> router/gateway (ubuquiti unifi) -> nat/lan
• VM with two interfaces. eth0 is 192.168.40.X/24 and has outbound network access to connect to the server at DC
• Eth1 is a blank interface that I want to bridge to a vswitch. From that vswitch I want to spin up other vms and use some of the public ips from my server network.

So once l2/wg is setup. Ideally all I would need to do is spin up a fresh server. Select that bridged vswitch and could use the public ips/gw's as if the vm was running in the data center. Obviously the traffic is going to be backhauled through wg/l2/vpn.

Is this possible? Can any assist me with a modified guide? Is wireguard the best tool for this? Gre tunnel instead? I am open to suggestions. Please see below picture example.

Image backup url https://imgur.com/a/KJc6kdM

@JonEhh1337
It sounds like my config, try give that a look, it might help you :)
(Check the comment above)

For the datacenter part, I would look at my Site A, and at Home give Site B a look ;)

Hello @zOrg1331, @SandeepAthiyarath

@zOrg1331 I asked you some questions a couple of weeks ago and I'm now reporting back to you. @SandeepAthiyarath I hope my findings can be useful for you.

I currently have a Wireguard + GRETAP system working successfully and I'm going to document here what I did. .......

Hi, thanks for your work, I followed your guidelines and it is working properly. A question: why are only the two servers that make the tunnel not reachable from network trunks on the opposite side?
Thanks again.
Marco

Is such a tunnel a full-fledged tunnel that openvpn provides in tap mode? What is the speed like? How much slower will it be than a standard wireguard connection? Has anyone tried to make a tunnel on an openwrt client? If you have server A and client B, can the client receive a DHCP address from server A (as in the case of openvpn), or does it have to set the address itself?