In Russia, VPN services are being actively blocked by Roskomnadzor. Traditional protocols, like OpenVPN, L2TP, SSTP and others stopped working a long time ago. Shadowsocks still works, but it's getting flimsier every day. The only solution, it seems, is to use Xray with split tunneling, and do so with your own private Xray server, because public servers will inevitably be blocked.
This presents a problem, however. You can easily set up an Xray server, that's for sure, but it'll just send packets out with its real IP. First of all, that may potentially lead to the server being blocked, if you start visiting Russian IP addresses using it. Second of all, you don't neccessarily want your cloud hosting provider to see everything you do.
It's easy to come up with a solution to this problem: install a VPN client on your Xray server and connect to a regular VPN service. However, doing this 'the dumb way' presents a set of certain inconveniences:
- What if the VPN client goes down? You'll suddenly start using the real IP of your Xray server, which you won't be alerted to.
- While regular VPN providers usually have apps that allow you to conveniently switch between servers, you will need to login into your Xray server's shell to do that.
The two problems, fortunately, have the same solution. Firewall marks.
The concept is fairly simple. We will tell xray-core to put a firewall mark on outbound packets. Then, the linux networking stack will choose a routing table to use based on that firewall mark. The chosen routing table will be responsible for saying 'hey, forward this packet to this OpenVPN client' if the chosen VPN client is up and saying 'hey, drop that into a black hole' if the client is down.
Unfortunately, OpenVPN doesn't have a --routing-table option, so we will have to use --route-up and --route-pre-down scripts to populate the routing tables ourselves using the environment variables OpenVPN client gives us. We can't just use static tables because VPN providers don't necessarily use static IP addresses, and if they do now, it's not a guarantee that this will not change in the future. We need to populate the tables dynamically.
I will be using Ubuntu Server 24.04 LTS. I will be putting my scripts and config files in /opt and my systemd service files in /etc/systemd/system. If you prefer to do things differently, take care to adjust commands and files as you go for your usecase.
WARNING! By continuing with this guide, you agree that if you follow my instructions, and they brick your server, whether it was running Ubuntu Server 24.04 LTS or not, or result in you (or anyone else) having to pay fines, or harm you (or anyone or anything else) in any other way, it will be your fault and I will not be held responsible, because you shouldn't mindlessly follow my instructions. You've been warned.
Assuming a brand new server with root login over SSH, set up an administrator account:
adduser admin
usermod -a -G sudo admin
Then, disable root login from ssh. In /etc/ssh/sshd_config, find
PermitRootLogin yes
and replace with
PermitRootLogin no
In the same file, the port of the ssh daemon needs to be changed, preferrably to something pretty high up (>30000, <=65535):
Port <ssh port>
Reboot to apply, ssh using the port you've just entered.
To make sure you aren't leaking DNS requests to your cloud hosting provider, install stubby:
apt install stubby
Edit the /etc/stubby/stubby.yml config file to choose your DoH servers (I usually configure it to use Cloudflare).
Then, edit netplan config in /etc/netplan to make sure you're using stubby:
...
eth0:
match: macaddress: "XX:XX:XX:XX:XX:XX"
nameservers:
addresses: [127.0.0.1]
dhcp4: true
dhcp4-overrides: use-dns: false set-name: "eth0"
If you configured stubby to listen on an IP different from 127.0.0.1, make sure to replace the address above accordingly.
At this point, it's also a good idea to set up the usual things an Xray server needs:
- A cronjob to restart every so often
- Firewall to deny anything but tcp/80, udp/443, tcp/443 and tcp/
<ssh port> - Fail2ban to combat ssh abuse
- RealiTLScanner to scan for websites to mirror
- A script to mirror some other IP address on ports tcp/80 and udp/443 using iptables
The routing tables that we will use will be generated in two simple steps:
- A script that runs on startup will populate the necessary tables with blackhole routes and create routing rules that make packets with the right firewall marks use the right tables.
- A script that runs on OpenVPN client will add routes to routing tables upon connection and delete routes from routing tables upon disconnection.
I will use the same number for tables and firewall marks, because it makes things easier.
Place this script in /opt/setup-routing.sh:
#!/bin/bash
set -euo pipefail
IFS=$'\n\t'
marks=(100 110 120)
echo "Running for marks: $marks"
for mark in "${marks[@]}"
do
echo "Mark $mark, setting up blackhole route"
/usr/sbin/ip route add blackhole default table $mark metric 1000
echo "Mark $mark, setting up routing rule"
/usr/sbin/ip rule add fwmark $mark table $mark
echo "Mark $mark, finished"
done
For each firewall mark, it will set up a blackhole route in the corresponding table, then the routing rule. To make things easier later on, I also recommend setting up a second script, /opt/undo-routing.sh:
#!/bin/bash
set -euo pipefail
IFS=$'\n\t'
marks=(100 110 120)
echo "Running for marks: $marks"
for mark in "${marks[@]}"
do
echo "Mark $mark, deleting blackhole route"
/usr/sbin/ip route del blackhole default table $mark metric 1000
echo "Mark $mark, deleting routing rule"
/usr/sbin/ip rule del fwmark $mark table $mark
echo "Mark $mark, finished"
done
This file simply deletes the routes and rules that /opt/setup-routing.sh created, which is great for debugging.
By editing the marks array (in both files), you can add or remove firewall marks and routing tables that get initialized. For example, you could do this if you wanted to use five VPN clients:
marks=(100 110 120 130 140)
The first one would use the mark 100, the second — 110, and so on.
Don't forget to modify permissions to make the scripts executable:
chmod ug+x /opt/setup-routing.sh
chmod ug+x /opt/undo-routing.sh
We can now create a systemd service that runs these scripts for us on startup. Place this in /etc/systemd/system/setup-routing.service:
[Unit]
Description=Setup routing for xray
After=network-online.target network.target
Requires=network-online.target
Before=x-ui.service
[Service]
Type=oneshot
ExecStart=/opt/setup-routing.sh
ExecStop=/opt/undo-routing.sh
RemainAfterExit=yes
Restart=on-failure
RestartSec=1
StartLimitInterval=30
StartLimitBurst=10
StartLimitAction=reboot
[Install]
WantedBy=multi-user.target
Note the line Before=x-ui.service.
I like the modifed X-UI panel from alireza0. If you want to use a different panel or none at all, you will want to replace this with whatever service starts your xray-core. This is so that the blackhole routes are initialized before users have the chance to connect to the Xray server.
Make sure to enable the service after creating the file:
systemctl daemon-reload
systemctl enable --now setup-routing
We will be using a few options of the OpenVPN client.
First of all, we will have to pass it the --script-security 2 argument to allow the OpenVPN client to run scripts at all.
Secondly, we will use --route-noexec to prevent the client from creating its own routing rules in the main routing table. We won't use them, and it'd be a catastrophe with multiple clients.
Finally, we will use --route-up and --route-pre-down to make the client execute a script where it would otherwise add routes on its own.
The script itself is fairly simple:
#!/bin/bash
if [ "$script_type" == "route-up" ]; then
/usr/sbin/ip route add default via $route_vpn_gateway dev $dev table $1 metric 100
elif [ "$script_type" == "route-pre-down" ]; then
/usr/sbin/ip route del default via $route_vpn_gateway dev $dev table $1 metric 100
fi
Put it in /opt/ovpn/route-config.sh and don't forget to set execute permissions:
chmod ug+x /opt/ovpn/route-config.sh
The OpenVPN client passes $route_vpn_gateway and $dev as environment variables, as well as $script_type to indicate what the script is supposed to do. We use these variables to add and delete the routes.
We will be reusing this script for multiple OpenVPN client instances, so the first argument passed to the script — $1, which we will control, will be the table number. Note the metric value (100) and compare it to the one we used for the blackhole routes (1000). Because the metric value for the blackhole routes is higher, the OpenVPN client routes will automatically be preferred over blackholes by the networking stack.
It is probably possible to get fancy and use one systemd service file for multiple OpenVPN clients. But I simply don't want to be bothered, so I use multiple service files, all named along the lines of /etc/systemd/system/ovpn_<name>.service, where <name> is replaced with something meaningful but short. The service files all look similarly:
[Unit]
Description=<Something Descriptive> Client
After=network-online.target
Wants=network-online.target
Documentation=man:openvpn(8)
Documentation=https://openvpn.net/community-resources/reference-manual-for-openvpn-2-6/
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO
[Service]
Type=notify
PrivateTmp=true
WorkingDirectory=/opt/ovpn
ExecStart=/usr/sbin/openvpn --suppress-timestamps --nobind --script-security 2 --route-noexec --route-up "route-config.sh <mark>" --route-pre-down "route-config.sh <mark>" --config <name>.conf
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_ADMIN CAP_NET_RAW CAP_SETGID CAP_SETUID CAP_SETPCAP CAP_SYS_CHROOT CAP_DAC_OVERRIDE
LimitNPROC=10
DeviceAllow=/dev/null rw
DeviceAllow=/dev/net/tun rw
ProtectSystem=true
ProtectHome=true
KillMode=process
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
Here, <mark> must be replaced with the firewall mark number you wish to be associated with the client. For example, 100.
<Something Descriptive> should be, well, something descriptive. Say, 'USA West Nord VPN', so that the service file description reads 'USA West Nord VPN Client'.
<name> must be replaced with the name of your openvpn config file, which must be placed in /opt/ovpn.
These files are usually obtained from your VPN provider. Sometimes they're called 'Config files for routers' or simply 'OVPN files', since they often have .ovpn extension. You want to rename them to <name>.conf, however. Take care to also modify them to use your login credentials.
Note that, since all the arguments necessary for our system to work are contained in the systemd service file, you do not need to modify the client config file but to make it use your login credentials.
When you're finished adding OpenVPN client service files, reload systemd and enable the clients:
systemctl daemon-reload
systemctl enable --now ovpn_<name1>.service
systemctl enable --now ovpn_<name2>.service
...
systemctl enable --now ovpn_<nameN>.service
If you leave the routing setup at that, however, you will encounter a peculiar problem of routes and routing tables disappearing every so often and revealing the real server IP. And if it seems like the times at which they disappear are entirely random, that's because they are.
The disappearances are caused by apt-daily-upgrade.service. It is triggered pretty much randomly twice per day and will cause systemd reexecution, which will make netplan reapply its configuration.
And the way netplan applies configuration, in its INFINITE wisdom, is by nuking EVERYTHING that has anything to do with routes and then happily applying its lil rules.
No problem though, to fix this, we can just use a netplan hook to execute a script after appl- Oh. netplan doesn't support hooks. Okay, doesn't matter. Fine, they want us to use netplan config files for network configuration, we will just use netpl- Oh. Wait. OpenVPN tunnel interfaces are created dynamically by the client, we can't use netplan! FUUUUU-
I'll be working on a solution to this, but in the meantime, the only effective way of solving this seems to be disabling apt-daily-upgrade.service and its timer which would otherwise trigger it anyway:
systemctl disable apt-daily-upgrade.timer
systemctl disable apt-daily-upgrade.service
Make sure to log into your servers every so often and do the updates.
You can now inspect what you've achieved by following this guide.
You can view the routing rules using
ip rule
and the routing tables using
ip route show table <mark>
where <mark> is replaced with the firewall mark number.
The rules should look kind of like this:
0: from all lookup local
32763: from all fwmark 0x78 lookup 120
32764: from all fwmark 0x6e lookup 110
32765: from all fwmark 0x64 lookup 100
32766: from all lookup main
32767: from all lookup default
The tables should look kind of like this:
default via <IP> dev tun<number> metric 100
blackhole default metric 1000
I like using the modifed X-UI panel. You can find instructions on installing it in its README file in the linked GitHub repo.
After you install and configure it, you will want to click 'Xray Configs', then the 'Advanced' tab, then 'Outbounds'. You will be presented with something along the lines of:
[
{
"tag": "direct",
"protocol": "freedom",
"settings": {
"domainStrategy": "UseIP"
}
},
{
"protocol": "blackhole",
"settings": {},
"tag": "blocked"
}
]
You will want to modify this JSON and add entries along the lines of
{
"tag": "direct-<mark>",
"protocol": "freedom",
"settings": {
"domainStrategy": "UseIP"
},
"streamSettings": {
"sockopt": {
"mark": <mark>
}
}
}
For each entry, <mark> must be replaced with the firewall mark number. These are the outbound configurations that we will use that tell xray-core to mark the packets with the firewall marks.
You may also want to add a mark of your choice to the outbound configuration tagged 'direct', which will make absolutely sure that no packet leaks from the real IP address, even if you make a configuration mistake:
{
"tag": "direct",
"protocol": "freedom",
"settings": {
"domainStrategy": "UseIP"
},
"streamSettings": {
"sockopt": {
"mark": <mark>
}
}
}
All in all your outbound configuration may look like this:
[
{
"tag": "direct",
"protocol": "freedom",
"settings": {
"domainStrategy": "UseIP"
},
"streamSettings": {
"sockopt": {
"mark": 100
}
}
},
{
"tag": "direct-100",
"protocol": "freedom",
"settings": {
"domainStrategy": "UseIP"
},
"streamSettings": {
"sockopt": {
"mark": 100
}
}
},
{
"tag": "direct-110",
"protocol": "freedom",
"settings": {
"domainStrategy": "UseIP"
},
"streamSettings": {
"sockopt": {
"mark": 110
}
}
},
{
"tag": "direct-120",
"protocol": "freedom",
"settings": {
"domainStrategy": "UseIP"
},
"streamSettings": {
"sockopt": {
"mark": 120
}
}
},
{
"protocol": "blackhole",
"settings": {},
"tag": "blocked"
}
]
You will want to hit 'Save', then 'Restart Xray' above the tabs. For good measure, restart the panel as well:
x-ui restart
The outbounds we've just configured allow the traffic to be passed to the OpenVPN clients, but, unless you modified the default 'direct' outbound, they currently do nothing.
We can, however, do something with them using the xray-core's internal routing rules. Assuming you're still on outbound configuration page, switch to the 'Routing Rules' tab. Click the 'Add Rule' button, and you will be presented with a lot of options.
Note the 'Outbound Tag' rule, where you can select one of the outbounds we've just created.
Also note the 'User' field, which allows you to assign an outbound to use for a specific 'user'. Or, more correctly, a specific configuration.
So what you can do, is you can create a number of configurations (that the panel refers to as users) per actual user, and create routing rules that assign to them the outbounds, that use firewall marks to feed packets into OpenVPN clients.
For example, if you create 'users' called 'client-100' and 'client-110' and add the corresponding xray-core routing rules forwarding everything to outbounds 'direct-100' and 'direct-110', respectively, then in the Xray client application, the end user will have the choice between using the 'client-100' and 'client-110' configurations. Which will, effectively, allow the user to connect to different OpenVPN clients, using only one Xray server.
From here on out, you're only limited by your own imagination. Set up configurations and rules how you want them.
If you need some inspiration, however, you can head back to the 'Advanced' tab, select 'Routing Rules' and, assuming it looks like this:
[
{
...
}
]
make it look like this:
[
{
...
},
{
"type": "field",
"user": [
"user1-100",
"user2-100",
"user3-100"
],
"outboundTag": "direct-100"
},
{
"type": "field",
"user": [
"user1-110",
"user2-110",
"user3-110"
],
"outboundTag": "direct-110"
},
{
"type": "field",
"user": [
"user1-120",
"user2-120",
"user3-120"
],
"outboundTag": "direct-120"
}
]
Don't forget to click 'Save' and 'Restart Xray' to apply.
The user<number>-<mark> format of naming users is just for illustration purposes. I think it makes much more sense to follow the <nickname>-<country code><optional additional descriptor> format.
For example, if I were to commit the heinous crime of using a VPN (of course, this is purely a hypothetical and I would never do such a thing) and if I had three OpenVPN clients connecing to Nord VPN servers in Dallas (US), London (UK) and Zurich (Switzerland), I'd name my configurations nullcaller-usds, nullcaller-ukl and nullcaller-ch.
The US one has a ds optional descriptor, because Nord VPN also has servers in Detroit and Denver, thus d alone wouldn't be descriptive enough. The UK one has an l optional descriptor that stands for London, because Nord VPN also has servers in Edinburgh, Glasgow and Manchester, and the Switzerland one doesn't have an optional descriptor, because Nord VPN's Switzerland servers are all in Zurich, which makes the descriptor unnecessary.
Note that this gist isn't sponsored by Nord VPN. I am using them as an example because everybody knows about them anyway. I don't use them, nor do I encourage anyone to do so. But then again, I would never stoop so low as to use a VPN. Now, speaking of stooping low, this gist is sponsored by Sq—