This is a modified instruction excerpt from the guide Beyond Cloudflare Tunnels: A High-Bandwidth Solution for Home Lab Remote Access seen on the StrangeMatter Blog.
This new instruction set specifies how to do a single application mapped to the gateway rather than a full through-traffic version. It focuses on vLLM as the expected usecase (but any application can be used).
On both the Gateway and the vLLM Server, I installed the latest headless Ubuntu. This is mostly because I am most familiar with the OS.
The Gateway is the VM we just purchased that has a public IP address. Typically, VMs like this use root as the default user to login as. This machine will run our LiteLLM Server & Gateway.
The vLLM Server will host our actual AI models. In my case, I set up a Proxmox Container for this and gave it a static IP on my subnet. Adding the static IP address means I always know where the container is in the future. I had the container configured with 1 CPU, 512MB of RAM, and 8GB of SSD storage. The default user is also root here.
Once both VMs are up and running, SSH into both machines. I did this in two different terminal windows (Gateway and vLLM Server).
ssh root@<vm-ip-address>
As usual, once I set up the accounts and log in, I update everything.
sudo apt update && sudo apt upgrade -y
This will ensure that all packages are up to date and that we have a clean baseline on both machines. It is typically good to set up your SSH keys here, too. This will make it easier to log in and work on the machine in the future.
# Example on adding a key to allow quicker logins
echo "<public ssh key>" >> ./.ssh/authorized_keys
On both machines, we need to install WireGuard. We want to keep the installation light since we have limited drive space.
sudo apt install wireguard-tools --no-install-recommends -y
The --no-install-recommends will ensure we only install WireGuard and nothing extra. Once this finishes, we will need to generate keys for both the Gateway and the vLLM Server, allowing the two to identify each other and preventing a third party from sniffing our data.
On both VMs, in the home directory, run this:
umask 077
wg genkey | tee privatekey | wg pubkey > publickey
This will generate our keys and dump our private and public keys in the home directory as privatekey and publickey, respectively.
To set up WireGuard on the LiteLLM Gateway server, we need to define a WireGuard configuration file. To create and edit the file, we'll use nano:
sudo nano /etc/wireguard/wg0.conf
[Interface]
PrivateKey = <Gateway PrivateKey>
Address = 10.28.0.1/24
ListenPort = 51820
[Peer]
PublicKey = <vLLM Server PublicKey>
AllowedIPs = 10.28.0.2/32
To note, our virtual network is defined by the 10.28.0.0/24 subnet. In our configuration above, we set the vLLM Server to 10.28.0.2 and lock it by setting the configurable bits to /32. This means the IP address must match exactly. Because we aren't forwarding external traffic to the vLLM machine's local home network, the Gateway completely ignores the vLLM server's physical local address, keeping the tunnel strictly isolated to the two virtual IPs.
On the vLLM Server, we also need to define a WireGuard configuration file:
sudo nano /etc/wireguard/wg0.conf
[Interface]
PrivateKey = <vLLM Server PrivateKey>
Address = 10.28.0.2/24
# Strict Firewall: No Masquerading. Drop everything except the specific vLLM port.
PostUp = iptables -A INPUT -i wg0 -m state --state RELATED,ESTABLISHED -j ACCEPT; iptables -A INPUT -i wg0 -p tcp --dport 8000 -j ACCEPT; iptables -A INPUT -i wg0 -j DROP
PostDown = iptables -D INPUT -i wg0 -m state --state RELATED,ESTABLISHED -j ACCEPT; iptables -D INPUT -i wg0 -p tcp --dport 8000 -j ACCEPT; iptables -D INPUT -i wg0 -j DROP
[Peer]
PublicKey = <Gateway PublicKey>
Endpoint = <Gateway Public IPv4 Address>:51820
AllowedIPs = 10.28.0.0/24
PersistentKeepalive = 25
We can see on the vLLM server that an address is defined and that we have the additional values of PostUp and PostDown which add custom firewall policies to our WireGuard connection.
Instead of opening up the whole machine or using network masquerading (NAT), we use a strict "lockdown" approach. The PostUp commands will run when the WireGuard instance is initiated:
iptables -A INPUT -i wg0 -m state --state RELATED,ESTABLISHED -j ACCEPT: This ensures that any network responses or ongoing conversations initiated by the vLLM server are allowed to flow back through the tunnel without being blocked.iptables -A INPUT -i wg0 -p tcp --dport 8000 -j ACCEPT: This opens up the exact endpoint port that the vLLM service runs on (port8000). This is the only designated doorway allowed through the tunnel. Note: If you change the expected vLLM port, you will need to change it here as well.iptables -A INPUT -i wg0 -j DROP: This is our security catch-all. It explicitly drops all other traffic coming across the WireGuard tunnel. If someone on the gateway attempts to SSH (port 22) or access other local resources on the vLLM machine, they are completely blocked.
Similarly, the PostDown commands will run immediately after the WireGuard instance is disabled. This will cleanly delete all of the custom firewall rules we just injected so the system's baseline table stays clean.
We can also see that we now define the Endpoint in the Peer, allowing the vLLM Server to safely initiate the connection out to the public Gateway. We also set PersistentKeepalive to 25 seconds to ensure that firewall NAT tables along the way don't drop our silent tunnel connection.
Once these are setup, we can run the following command to ensure that WireGuard is running on both VMs.
sudo wg-quick up wg0
One of the cleanest benefits of this architecture is how easily it scales. If you need to bring a second vLLM server online to handle more model traffic, you don't need to change anything about your network topology. You simply assign the new machine the next virtual IP in our block (10.28.0.3) and register it as a new peer on the Gateway.
To allow the Gateway to recognize the second vLLM machine, append a new [Peer] block to the bottom of your existing /etc/wireguard/wg0.conf file on the Gateway:
# Existing configuration stays above...
[Peer]
PublicKey = <vLLM Server #1 PublicKey>
AllowedIPs = 10.28.0.2/32
# ADD THIS NEW BLOCK:
[Peer]
PublicKey = <vLLM Server #2 PublicKey>
AllowedIPs = 10.28.0.3/32
Restart WireGuard on the Gateway to apply the changes:
sudo systemctl restart wg-quick@wg0
On your second vLLM machine, follow the exact same key generation steps as the first one. When creating its /etc/wireguard/wg0.conf file, change the interface address to 10.28.0.3/24. The security rules and gateway endpoint remain exactly identical:
[Interface]
PrivateKey = <vLLM Server #2 PrivateKey>
Address = 10.28.0.3/24
# Strict Firewall: Drop everything except the specific vLLM port (8000)
PostUp = iptables -A INPUT -i wg0 -m state --state RELATED,ESTABLISHED -j ACCEPT; iptables -A INPUT -i wg0 -p tcp --dport 8000 -j ACCEPT; iptables -A INPUT -i wg0 -j DROP
PostDown = iptables -D INPUT -i wg0 -m state --state RELATED,ESTABLISHED -j ACCEPT; iptables -D INPUT -i wg0 -p tcp --dport 8000 -j ACCEPT; iptables -D INPUT -i wg0 -j DROP
[Peer]
PublicKey = <Gateway PublicKey>
Endpoint = <Gateway Public IPv4 Address>:51820
AllowedIPs = 10.28.0.0/24
PersistentKeepalive = 25
Bring the interface up on the new server:
sudo wg-quick up wg0
Your LiteLLM Gateway can now seamlessly route requests to either 10.28.0.2:8000 or 10.28.0.3:8000 via its load-balancing configuration, completely isolated from both local networks!