How to setup Lerd + Private Internet Access in Ubuntu.

pia-vpn-debug-handbook.md

Debugging "PIA VPN connected but no Internet" on Linux (Lerd-sandbox edition)

A self-contained playbook distilled from a real debugging session on a Linux box running Private Internet Access 3.7.2 inside a Lerd dev sandbox (podman + pasta + dnsmasq on 127.0.0.1:5300). Two distinct root causes were found and fixed. Everything below is meant to be re-runnable on a different machine, with per-machine values called out so you can adapt it.

---

Table of contents

1. Symptoms
2. Root cause 1 — UDP DNS poisoning of the PIA zone
3. Root cause 2 — PIA DNS collides with Lerd on port 5300
4. How to diagnose on a NEW machine
5. The fixes (files + commands)
6. Post-reboot health check
7. The safety rig: breaker + capture-while-connected
8. Key learnings
9. Per-machine values to adapt
10. Cleanup / uninstall

---

Symptoms

• The PIA app either can't connect at all, or connects (shows a VPN IP) but no site loads.
• While connected, Chrome shows DNS_PROBE_STARTED / DNS_PROBE_FINISHED_NO_INTERNET.
• Disconnecting restores the Internet immediately.
• Sometimes DNS works for the first curl right after connect, then dies.

The root pattern here: the VPN tunnel and routing are perfectly healthy; the system DNS chain is what breaks while connected.

---

Root cause 1 — UDP DNS poisoning of the PIA zone

The network (ISP/router/upstream filter) returns forged NXDOMAIN for every *.privateinternetaccess.com name over plain UDP port 53 — while DNS-over-HTTPS resolves the zone fine, DNSSEC-validated. Because the PIA daemon resolves its API over the system resolver, it fails with QNetworkReply::HostNotFoundError and can't refresh status/tokens → "app can't connect".

Signature check (do this on any new machine):

# 1. Plain UDP, every resolver you can reach:
dig @127.0.0.1 -p 5300 api.privateinternetaccess.com +short   # NXDOMAIN -> poisoned
dig @192.168.88.1       api.privateinternetaccess.com +short  # NXDOMAIN -> poisoned
dig @8.8.8.8            api.privateinternetaccess.com +short  # NXDOMAIN -> poisoned (even direct!)

# 2. DNS-over-HTTPS (bypasses UDP tampering):
curl -sS -m 8 -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=api.privateinternetaccess.com&type=A'
# -> Status:0 + real A records means the zone is healthy globally and UDP is being poisoned.

If @8.8.8.8/@1.1.1.1 also answer NXDOMAIN (with aa flag) while DoH succeeds → network-level UDP DNS poisoning/filtering of that zone. Fix by pinning the zone locally.

---

Root cause 2 — PIA DNS collides with Lerd on port 5300

• PIA has overrideDNS: "pia" in /opt/piavpn/etc/settings.json. On connect it:
  1. sets tun0's per-link DNS to its own resolver (10.0.0.243 over the tunnel) — this works,
  2. starts its internal resolver (hnsd) bound to 127.0.0.1:5300,
  3. installs firewall rules (blockDNS/allowDNS) so only that resolver may be used.
• The Lerd sandbox permanently owns 127.0.0.1:5300 (pasta forwards it to a podman dnsmasq). PIA's hnsd therefore never binds, and Lerd's DNS watcher (a Go daemon, not just the NetworkManager dispatcher!) re-claims the VPN interface's per-link DNS:

tun0: Bus client set DNS server list to: 10.0.0.243       <- PIA  (good)
tun0: Bus client set search domain list to: ~.
tun0: Bus client set DNS server list to: 127.0.0.1:5300   <- Lerd (kills it)
tun0: Bus client set search domain list to: ~test, ~.

• Result: every system DNS query is forced at the hijacked 127.0.0.1:5300, which is blackholed by PIA's blockDNS while the tunnel is up → DNS_PROBE_* errors even though ping 8.8.8.8, curl to IPs, and direct dig @10.0.0.243 all work through the tunnel.

journalctl -u systemd-resolved is the forensic tool that shows exactly who set what per-link DNS.

---

How to diagnose on a NEW machine

1. Inventory the stack (read-only)

id -u; command -v piactl openvpn wg dig resolvectl systemctl
ls -la /opt/piavpn/var /opt/piavpn/etc 2>/dev/null        # daemon.log, settings.json, pia.ovpn
systemctl list-units --all | grep -iE 'pia|openvpn'       # piavpn.service etc.
ip -brief addr; ip route show; ip rule show               # routes, policy rules
find ~/.config /etc /opt -maxdepth 3 \( -iname '*lerd*' -o -iname '*pia*' \) 2>/dev/null | head -30

2. Ask the daemon what it thinks

piactl get connectionstate; piactl get region; piactl get protocol
grep -E 'api/client/status|Request for "api' /opt/piavpn/var/daemon.log | tail -10
# HostNotFoundError here => resolver can't reach api.privateinternetaccess.com

3. DNS forensics (do this BEFORE connecting — safe)

cat /opt/piavpn/etc/settings.json | grep -o '"overrideDNS":"[^"]*"'   # pia|custom|system
resolvectl status | grep -E 'DNS Servers|Current DNS'
journalctl -u systemd-resolved --since '10 min ago' | grep 'tun0:'    # who sets tun0 DNS

4. The decisive "inside the connected window" test

Never connect blind: arm the breaker + capture rig first, connect, read the capture. It answers, in one run:

• tunnel data path (ping/curl) OK? → then it's DNS
• dig @10.0.0.243 answers over the tunnel? → PIA tunnel DNS works
• dig @127.0.0.1 -p 5300 and dig @<router> fail while connected? → the 5300/blocker conflict
• getent hosts google.com empty while connected? → system DNS broken

---

The fixes (files + commands)

Fix 1 — Pin the poisoned PIA zone (Lerd dnsmasq, user-level)

Verifies IPs first via DoH, then adds a separate dnsmasq conf so Lerd's regenerated lerd.conf (which only rewrites lerd.conf) won't clobber it:

# Re-verify the real IPs (they can change; Cloudflare-fronted):
IP1=$(curl -sS -m 8 -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=api.privateinternetaccess.com&type=A' \
  | grep -oE '"data":"[0-9.]+"' | head -1 | cut -d'"' -f4)
IP2=$(curl -sS -m 8 -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=api.privateinternetaccess.com&type=A' \
  | grep -oE '"data":"[0-9.]+"' | sed -n 2p | cut -d'"' -f4)

# Lerd dnsmasq reads every *.conf in this user dir:
mkdir -p ~/.local/share/lerd/dnsmasq
printf 'address=/privateinternetaccess.com/%s/%s\n' "$IP1" "$IP2" \
  > ~/.local/share/lerd/dnsmasq/99-pia.conf

# Restart the Lerd DNS container (user-level podman container via quadlet):
systemctl --user restart lerd-dns

# Verify:
dig @127.0.0.1 -p 5300 api.privateinternetaccess.com +short # -> real IPs now
curl -m 8 -sS -o /dev/null -w 'HTTP %{http_code}\n' https://api.privateinternetaccess.com/api/client/status

On a non-Lerd machine, skip the pin and instead check whether you can reach the API; the poisoning may be absent there.

Fix 2 — Durable DNS helper (the core fix that won)

On every Connected, re-assert tun0 → 10.0.0.243 after Lerd's watcher claims the interface. It relies on the passwordless resolvectl grant that Lerd installs via /etc/sudoers.d/lerd (test with sudo -n resolvectl dns enp1s0 127.0.0.1:5300 192.168.88.1).

~/.local/bin/pia-dns-helper.sh (full file — self-contained so it survives reboots):

#!/usr/bin/env bash
# pia-dns-helper: keep tun0 DNS pinned to PIA tunnel resolver (10.0.0.243)
# after each connect. PIA's DNS works, but the Lerd watcher claims tun0 right
# after connect and points it at 127.0.0.1:5300 (dead while the VPN is up).
# This re-asserts PIA's tunnel DNS a few seconds after every Connected.
#
# Self-contained: state/logs live under ~/.local/state/pia-dns-helper so it
# survives reboots. (The original version logged to /tmp/pia_safe/, which is
# wiped on reboot: absent parent dir => every ">> log" redirect fails => the
# sudo resolvectl commands silently never ran.)
STATEDIR="${XDG_STATE_HOME:-$HOME/.local/state}/pia-dns-helper"
mkdir -p "$STATEDIR" 2>/dev/null || exit 1
LOG="$STATEDIR/helper.log"
STOPFLAG="$STATEDIR/stop"
log(){ printf '[%s] %s\n' "$(date '+%F %T')" "$*" >> "$LOG" 2>/dev/null || true; }
fix(){
  sleep 5   # let PIA set tun0 and Lerd claim it first
  log "pass1: tun0 -> 10.0.0.243"
  sudo -n resolvectl dns tun0 10.0.0.243   >> "$LOG" 2>&1 || log "pass1 dns FAILED"
  sudo -n resolvectl domain tun0 ~.        >> "$LOG" 2>&1 || log "pass1 domain FAILED"
  sleep 12
  log "pass2: re-assert tun0 -> 10.0.0.243"
  sudo -n resolvectl dns tun0 10.0.0.243   >> "$LOG" 2>&1 || log "pass2 dns FAILED"
  sudo -n resolvectl domain tun0 ~.        >> "$LOG" 2>&1 || log "pass2 domain FAILED"
  log "fix done"
}
log "dnshelper started"
LAST=""
while :; do
  s=$(piactl get connectionstate 2>/dev/null || echo Unknown)
  if [ "$s" != "$LAST" ]; then
    LAST="$s"
    log "state -> $s"
    [ "$s" = Connected ] && fix &
  fi
  [ -f "$STOPFLAG" ] && { log "stopped"; exit 0; }
  sleep 2
done

Reboot gotcha (hit in the field): never log/state into /tmp from a service you expect to survive reboots — with the parent dir missing, every cmd >> /tmp/... redirect fails and the command never runs, so the "fix" silently becomes a no-op after reboot while the unit keeps showing "active". Use ~/.local/state/... and mkdir -p the dir at start.

~/.config/systemd/user/pia-dns-helper.service (full file):

[Unit]
Description=PIA DNS helper: keep tun0 DNS pinned to PIA tunnel resolver (10.0.0.243) after each connect
After=default.target

[Service]
ExecStart=/home/wilsenhc/.local/bin/pia-dns-helper.sh
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

Install + enable (run as your user):

cp /tmp/pia_safe/dns-helper.sh ~/.local/bin/pia-dns-helper.sh   # or write the file
chmod +x ~/.local/bin/pia-dns-helper.sh
mkdir -p ~/.config/systemd/user
# create ~/.config/systemd/user/pia-dns-helper.service with the content above
systemctl --user daemon-reload
systemctl --user enable --now pia-dns-helper

Gotcha from the field: a leftover stop_dnshelper flag makes the service exit instantly and bounce forever (systemd auto-restart). Remove the flag file after stopping a previous instance.

Manual fallback (one-liner while connected, no automation)

sudo resolvectl dns tun0 10.0.0.243
sudo resolvectl domain tun0 ~.

Fix 3 — overrideDNS: system (helps, but the GUI re-asserts pia)

sudo sed -i 's/"overrideDNS":"pia"/"overrideDNS":"system"/' /opt/piavpn/etc/settings.json
sudo systemctl restart piavpn
grep -o '"overrideDNS":"[^"]*"' /opt/piavpn/etc/settings.json

Caveat: if the PIA app (GUI) is running, it re-asserts its cached overrideDNS: "pia" as soon as it reconnects to the daemon — the file edit alone is not durable. The helper (Fix 2) works in both modes, so it's the durable layer.

Optional / partial fixes kept as context (did not fully solve it alone)

Skip tun0 in the Lerd NM dispatcher (root):

sudo cp /etc/NetworkManager/dispatcher.d/99-lerd-dns /etc/NetworkManager/dispatcher.d/99-lerd-dns.bak
sudo sed -i 's|if \[ "$IFACE" = "lerd0" \]; then|if [ "$IFACE" = "lerd0" ] \|\| [ "${IFACE%%[0-9]*}" = "tun" ]; then|' \
  /etc/NetworkManager/dispatcher.d/99-lerd-dns

Unmanage tun* in NM (root) — mirrors the existing wgpia* rule Lerd ships:

sudo cp /etc/NetworkManager/conf.d/wgpia.conf /etc/NetworkManager/conf.d/wgpia.conf.bak
printf '[keyfile]\nunmanaged-devices=interface-name:wgpia*,tun*\n' | sudo tee /etc/NetworkManager/conf.d/wgpia.conf
sudo systemctl restart NetworkManager

Both reduce interference but do not stop Lerd's Go watcher (it reacts to the interface at a lower level than the NM unmanaged list), which is why the after-the-fact re-assert helper is the reliable fix.

---

Post-reboot health check

After any reboot, three things degrade silently and must be verified before blaming the app:

1. The helper must not live in /tmp. Check systemctl --user is-active pia-dns-helper and that its log exists at ~/.local/state/pia-dns-helper/helper.log. If anything references /tmp, it broke: on reboot /tmp is wiped, so every cmd >> /tmp/... redirect fails and the command never runs while the unit still shows "active" — the fix silently becomes a no-op.

2. systemd-resolved may silently fall back to the poisoned router. At boot, lerd-dns isn't up yet, so resolved marks 127.0.0.1:5300 degraded and switches the per-link server to the second entry (192.168.88.1) — the very resolver that fakes NXDOMAIN for the PIA zone. Symptom: resolvectl status shows Current DNS Server: 192.168.88.1 on the main link; getent hosts google.com works but api.privateinternetaccess.com fails, while dig @127.0.0.1 -p 5300 api... answers. Fix (the router stays reachable as dnsmasq's own upstream, so it doesn't need to be a link-level server):

   sudo -n resolvectl dns enp1s0 127.0.0.1:5300          # drop the router from the link list
   resolvectl flush-caches
   getent hosts api.privateinternetaccess.com             # expect the pinned IP
   curl -m 8 -sS -o /dev/null -w 'HTTP %{http_code}\n' \
     https://api.privateinternetaccess.com/api/client/status   # expect 200

3. Daemon API health and system DNS health are independent. PIA's daemon keeps fetching api/client/status with 200 even while the system resolver returns NXDOMAIN (it resolves the API over its own path). So "daemon log says 200" ≠ "system DNS is fine" — check both separately before concluding.

---

The safety rig: breaker + capture-while-connected

Why you need this: connecting to a VPN can yank the network out from under the very session doing the debugging. Two pre-armed, detached scripts keep the session safe and capture evidence inside the connected window.

Breaker (/tmp/pia_safe/breaker.sh) — auto-disconnect on failure modes only

#!/usr/bin/env bash
# Auto-disconnects ONLY on failure modes (stuck Connecting, Interrupted/Reconnecting,
# or lifeline lost while Connected). Never interrupts a healthy connection.
# Stop with: kill "$(cat /tmp/pia_safe/breaker.pid)"
set -u
DIR=/tmp/pia_safe; LOG="$DIR/breaker.log"; PIDFILE="$DIR/breaker.pid"; FAILS=0
echo $$ > "$PIDFILE"
log(){ printf '[%s] %s\n' "$(date '+%H:%M:%S')" "$*" >> "$LOG"; }
state(){ piactl get connectionstate 2>/dev/null || echo Unknown; }
connected(){ timeout 3 bash -c 'exec 3<>/dev/tcp/1.1.1.1/443' 2>/dev/null; }
lan(){ ping -q -c1 -W2 192.168.88.1 >/dev/null 2>&1; }
log "breaker armed (pid $$)"; START=0
while :; do
  s=$(state)
  case "$s" in
    Connecting) [ "$START" -eq 0 ] && START=$(date +%s)
      [ $(( $(date +%s) - START )) -gt 45 ] && { log "stuck Connecting -> disconnect"; piactl disconnect; } ;;
    Interrupted|Reconnecting) [ "$START" -eq 0 ] && START=$(date +%s)
      [ $(( $(date +%s) - START )) -gt 15 ] && { log "state $s >15s -> disconnect"; piactl disconnect; } ;;
    Connected) START=0
      if ! connected && ! lan; then FAILS=$((FAILS+1)); [ "$FAILS" -ge 3 ] && { log "lifeline lost -> disconnect"; piactl disconnect; FAILS=0; }; else FAILS=0; fi ;;
    *) START=0; FAILS=0 ;;
  esac
  [ -f "$DIR/stop" ] && { log "stop -> exiting"; rm -f "$PIDFILE"; exit 0; }
  sleep 3
done

Capture-while-connected (/tmp/pia_safe/capture.sh) — probe battery at +2s/+12s

#!/usr/bin/env bash
# Runs a probe battery at +2s and +12s after Connected; logs EVERYTHING to capture.out.
# Stop with: touch /tmp/pia_safe/stop_capture
OUT=/tmp/pia_safe/capture.out; : > "$OUT"
ts(){ date '+%H:%M:%S'; }
battery(){
  {
    echo "----- BATTERY @ $(ts) -----"
    echo "== resolv.conf:"; grep nameserver /etc/resolv.conf
    echo "== resolvectl:"; resolvectl status 2>/dev/null | grep -E 'Link |Current DNS|DNS Servers|Default Route' | head -12
    echo "== routes:"; ip route show
    echo "== tun0 mtu:"; cat /sys/class/net/tun0/mtu 2>/dev/null
    echo "== tun0 stats:"; ip -s link show tun0 2>/dev/null | sed -n '1,8p'
    echo "== dns per-link:"; resolvectl dns enp1s0 2>/dev/null; resolvectl dns tun0 2>/dev/null
    echo "== ping tunnel gw:"; ping -c2 -W3 10.24.128.1 2>&1 | tail -2
    echo "== ping 8.8.8.8:"; ping -q -c3 -W3 8.8.8.8 2>&1 | tail -2
    echo "== ping lan gw:"; ping -q -c1 -W2 192.168.88.1 2>&1 | tail -1
    echo "== http 1.1.1.1 (no DNS):"; curl -m4 -sS -o /dev/null -w 'code=%{http_code}\n' http://1.1.1.1 2>&1
    echo "== dns via tunnel 10.0.0.243:"; dig @10.0.0.243 google.com +time=2 +tries=1 +short 2>&1 | head -2
    echo "== dns via lerd 127.0.0.1:5300:"; dig @127.0.0.1 -p 5300 google.com +time=2 +tries=1 +short 2>&1 | head -2
    echo "== dns via router:"; dig @192.168.88.1 google.com +time=2 +tries=1 +short 2>&1 | head -2
    echo "== getent google.com:"; getent hosts google.com 2>&1 | head -1
  } >> "$OUT" 2>&1
}
LAST=""
while :; do
  s=$(piactl get connectionstate 2>/dev/null)
  if [ "$s" != "$LAST" ]; then
    echo "[$(ts)] STATE: ${LAST:-none} -> $s" >> "$OUT"; LAST="$s"
    case "$s" in Connecting) echo "--- at Connecting ---" >> "$OUT"; ip route show >> "$OUT";;
      Connected) ( sleep 2; battery ) & ( sleep 12; battery ) & ;; esac
  fi
  [ -f /tmp/pia_safe/stop_capture ] && exit 0
  sleep 1
done

Launch all three detached: setsid nohup bash /tmp/pia_safe/capture.sh >/dev/null 2>&1 & (and same for breaker.sh). Keeping piactl usable (world-readable daemon socket) is what makes the breaker able to recover even after the session's connectivity is gone.

---

Key learnings

1. "Connected but no Internet" is usually DNS, not routing. Prove it with one capture: tunnel pings/curl OK + getent/browser fail + direct dig @<tunnel-dns> answers.
2. Suspect UDP DNS poisoning when: a domain NXDOMAINs from the local resolver, the router, and 8.8.8.8/1.1.1.1 directly, but DNS-over-HTTPS resolves it. Only 53/UDP is tampered.
3. journalctl -u systemd-resolved is the forensic tool — it logs exactly which "Bus client" set which per-link DNS/domains, and in what order (10.0.0.243 → 127.0.0.1:5300).
4. PIA Linux facts: daemon logs at /opt/piavpn/var/daemon.log; daemon config at /opt/piavpn/etc/settings.json; piactl talks to the daemon over a world-writable socket (so piactl disconnect still works after connectivity is lost). The GUI re-asserts its cached settings (e.g. overrideDNS:pia) — file edits while the app runs get reverted.
5. Lerd owns 127.0.0.1:5300 via pasta, so any VPN that wants a local DNS resolver on 5300 (PIA's hnsd) can never bind; and Lerd's Go watcher re-claims VPN interfaces' per-link DNS regardless of NM dispatcher patches or NM unmanaged-devices rules. The reliable lever is to re-assert the tunnel DNS after the fact (helper), or change the app's own setting.
6. Rebooting the wrong thing can take down the sandbox networking (restarting Lerd's DNS container restarts the pasta netns). Prefer file-level/resolvectl changes; restart NM/Lerd only when necessary.
7. Containment first, connections second: always stage a breaker + capture before triggering a connect that could cut your own session.
8. After reboot, systemd-resolved can silently fall back to a per-link server's second entry when the first (127.0.0.1:5300) was marked degraded before the local resolver was up — the Current DNS Server line in resolvectl status says which one is live. Re-assert the link DNS list so the healthy resolver is the only option, then resolvectl flush-caches.
9. A service's own API calls returning 200 don't validate system DNS. PIA's daemon resolves its API independently, so it can look perfectly healthy while the system resolver is broken — verify system resolution (getent/resolvectl query) separately.
10. Never persist state/logs in /tmp from anything meant to survive reboots. A missing parent directory makes every cmd >> /tmp/... redirect fail, which silently skips the command while the unit keeps reporting "active". Use ~/.local/state/<app>/ and mkdir -p it at startup.

---

Per-machine values to adapt

Value	Where to find it	Notes
LAN router/gateway	ip route show default	Used for an alternate DNS probe (dig @<gw>) and breakers' LAN ping
PIA tunnel DNS 10.0.0.243	pia.ovpn (route 10.0.0.243 vpn_gateway) or grep dnsServers daemon.log	Verify while connected: dig @10.0.0.243 google.com
PIA zone IPs to pin	DoH query (see Fix 1)	Cloudflare-fronted; re-verify, they can change
PIA dirs /opt/piavpn	piactl --help, find	Version-dependent sometimes /opt/piavpn
Lerd dirs ~/.local/share/lerd/dnsmasq, ~/.config/lerd	find ~ -maxdepth 4 -iname '*lerd*'	User-level quadlet-managed
Passwordless resolvectl	sudo -n resolvectl dns ... (silent success = allowed)	Needed for the helper; else prompt or sudoers
Scanner for the helper service user	whoami	Unit ExecStart uses an absolute path

---

Cleanup / uninstall

# Stop + disable the DNS helper:
systemctl --user stop pia-dns-helper
systemctl --user disable pia-dns-helper
rm -f ~/.local/bin/pia-dns-helper.sh ~/.config/systemd/user/pia-dns-helper.service

# Remove the DNS zone pin:
rm -f ~/.local/share/lerd/dnsmasq/99-pia.conf
systemctl --user restart lerd-dns

# Revert root patches if desired:
sudo mv /etc/NetworkManager/conf.d/wgpia.conf.bak /etc/NetworkManager/conf.d/wgpia.conf
sudo mv /etc/NetworkManager/dispatcher.d/99-lerd-dns.bak /etc/NetworkManager/dispatcher.d/99-lerd-dns

# Reset PIA DNS to default if you ever want PIA-managed DNS again:
sudo sed -i 's/"overrideDNS":"system"/"overrideDNS":"pia"/' /opt/piavpn/etc/settings.json

---

Generated from a live debugging session — portable template; always re-verify per-machine values before applying on a new host.