Using ZeroTier to route between two UniFi USG LANs

README.md

Routing between two UniFi USG based LANs with ZeroTier

Intro

This is a quick explanation of how to configure both ZeroTier and two Ubiquiti UniFi USGs to allow routing between two IPv4 networks. Both networks are in private (RFC1918) address space and each one has its own DHCP service. There is no need for NAT between them, only IP routing.

The two networks are 192.168.1.0/24 (call this the "left" network) and 192.168.10.0/24 (the "right" network) but they can be anything. Also, you can have multiple CIDR blocks on one side or the other of the ZeroTier route; if you do, you just have to create more routing table entries.

Configure ZeroTier

1. Create a ZeroTier network to serve as the connection between the two LANs. Give it a private address space that does not overlap either of the two LANs' RFC1918 spaces. In this example, the ZeroTier network uses 172.25.0.0/16. Note again that this has to be different from the space used by either of the LANs.
2. Add at least one device from each LAN to the ZeroTier network (install client software, add it as a member, etc.) to act as a gateway between the LAN and ZeroTier. These devices should have static addresses on their respective LANs. Note the ZeroTier IPv4 address that each gets assigned, or manually assign them ones within the address range that you've decided to use for ZeroTier (172.25.x.x here). Use ifconfig or a similar tool to check that each device has both its regular LAN network interface and also the ZT interface (zt0 usually), and that they each have the IPv4 addresses you expect.
3. Set up the routes within ZeroTier. In the upper-right of the ZeroTier Central web interface, under "Managed Routes", add a route for each of the LANs. The routes you want to add are from each LAN's subnet to the ZeroTier address of the gateway that's on that LAN. So, if the 192.168.1.0/24 network has a gateway device with the ZeroTier address 172.25.1.1, you would create a route with 192.168.1.0/24 in the "Network/Bits" field and 172.25.1.1 in the LAN field. You need to set up at least two routes, one for each LAN at either end.

Configure UniFi USGs

At this point you have ZeroTier set up, but you still need to put the routes into the LAN devices' routing tables and create appropriate firewall rules. This can be done through the UniFi web interface, and you need to do it on either end. These instructions assume you have a USG at both sites.

1. In the UniFi web interface, go to "Settings", then "Routing and Firewall", then "Static Routes".
2. On each LAN, create a static route for the other LAN's IP address range, with the "Next Hop" set to the LAN IPv4 address of the ZeroTier gateway. (This is the address that the gateway machine has on the interface that connects to the LAN, not its ZeroTier-defined IPv4 address on the zt0 interface.) E.g.: on the USG for the 192.168.1.0/24 (left) network, you would need to create a route to 192.168.10.0/24 (right) address range, with the "Next Hop" set to 192.168.1.X where .X is whatever the gateway machine's LAN IP address is. Set the "Hop Distance" to 1.
3. This is the important part: In addition to these routes, which are pretty standard, you will probably also want to create a route on either end routing to/from the ZeroTier address range. If you don't do this, devices that are directly connected to the ZeroTier network won't be able to talk to LAN devices, which at the very least makes troubleshooting difficult. This route would look like 172.25.0.0/16 as the network, with the "Next Hop" set to the same LAN IPv4 gateway machine address as in Step 2 above.

It can take a few minutes for the routes in the UniFi controller to propagate down to the USG, so give it a chance before testing.

This is based in large part on the Zerotier Support article designed for use on Amazon EC2 instances, but there are some differences when used within conventional LANs.

https://support.zerotier.com/knowledgebase.php?entry=show&search-for=&article=OWZhOGU5NTYzZmFjMDlmZmIxNTQyNDVlZjRlYzI1MjI_

Don't forget to turn on IP forwarding on your zerotier machines.

I had to set up some iptables rules on the "bridge" too (as per https://askubuntu.com/a/1052796)