HTTP/2 Thing

qb-http2.md

Upgrade Qbittorrent HTTP Announces to HTTP/2

Motivation

You probably don't want to do what is described in this document. Seriously consider just leaving things alone.

Because Qbittorrent, and most all torrent clients, still use HTTP/1.1, they typically do not reuse connections for multiple announces of different torrents to the same tracker announce URL. As the number of torrents goes up, there ends up being a lot of traffic back and forth just re-establishing a connection for each announce. This is highly inefficient, and while it is not really a problem for most users, or even most tracker operators, it is something that can start to be noticed at at the tracker level at a very large scale.

Using a more modern protocol, like HTTP/2, addresses this inefficiency by reusing an established connection. Unfortunately, Transmission is really the only torrent client that implements HTTP/2 right now. For Qbittorrent, the issue resides in an underlying library, libtorrent. The issue was raised in 2020, but work on this feature has not been a priority.

arvidn/libtorrent#4237 (comment)

Update: A pull request is being actively worked on! arvidn/libtorrent#8025

Description of HTTP/2 Upgrade Workaround

Reverse proxy software like Nginx or Caddy are usually used server-side as a front-end to an underlying website or web application. However, nothing (in theory) prevents use of the same pattern client-side. But, why would we do this? We can use this intermediary reverse proxy to upgrade the HTTP connection to a more modern protocol than the underlying client is capable of achieving. This way, the inefficiencies of HTTP/1.1 can stay local to the client and outbound traffic is sent using the higher protocol. Caddy upgrades connections when it can by default.

So, how do we get Qbittorrent to use Caddy as a reverse proxy? There are a few ways to achieve this behavior. I am going to describe a method meets the following goals:

• Run alongside a glutun container to allow VPN tunneling.
• Preserve the use of a secure protocol and certificate validation for trackers that are not proxied.
• Keep qbit attached to the (e.g.,) tun0 interface only.
• Avoid the use of a script to rewrite tracker announce URLs to an alternative local URL.
• Minimize the need for additional host configuration outside Docker.
• Reduce or eliminate any annoying maintenance of the workaround.

A glutun container is responsible for the networking of two attached containers: Caddy and Qbittorrent. I use a bind mount to override the glutun-provided /etc/hosts for Qbittorrent, adding entries to redirect a list of tracker announce URLs to the tun0 interface IP address. Since Caddy and Qbittorrent share a network namespace, this override directs traffic to Caddy, which is listening on the HTTPS TCP port. Caddy generates and manages self-signed certificates for each of the domains. The Caddy-generated certificate trust chain (as of writing, a root and intermediate certificate) is added to the Qbittorrent's trusted certificate data store.

Caddy upgrades and forwards the HTTP request to the destination tracker. If the connection is not upgraded, legacy keepalive headers are added to the request--one way to increase efficiency of HTTP/1.1 traffic.

There are three elements of this solution that need to be maintained, and this maintenance is performed using container automation:

1. Any time the Qbittorrent container is recreated, the Caddy-generated certificates need to be re-added to the trust data store. A post-start hook ensures this action is completed.
2. The IP address of the tun0 interface may change. A container health check monitors the tunnel interface IP address and updates the Qbittorrent hosts file if needed.
3. After a very long time, Caddy may need to regenerate its root certificate, which expires in 10 years. Up to you if you want to tie a bow on this automation step...

Step 1: Containers

Set up the containers as in the example below using Docker, with Qbittorrent and Caddy sharing a network namespace. This example used Docker Compose and shows Qbittorrent underneath a glutun container. Running Qbittorrent and Caddy this way means both containers reside at the same IP address. Qbittorrent will see caddy running on ports 80 and 443 on localhost. A similar principle is applicable if running both programs in an LXC container, for example.

When you network containers this way Docker shares their /etc/resolv.conf with the parent. This means if you change /etc/resolv.conf in any of the three containers, the changes will show up in all three. So, in order to override just the Qbittorrent name lookups, you need bind mount /etc/resolv.conf in the qbit container. Doing this overrides this behaviour, anow now you can redirect tracker announce domains in the bind mount to localhost. I'm not sure how someone would achieve this with LXC---there might be a way though.

Example Docker Compose

services:
  caddy:
    image: caddy:2
    user: 999:993    # your user may vary
    container_name: caddy
    restart: unless-stopped
    volumes:
      - ./caddy/conf:/etc/caddy
      - ./caddy/.data:/data
      - ./caddy/.config:/config
    network_mode: "service:qbittorrent"

  qbittorrent:
    image: qmcgaw/gluetun
    container_name: qbittorrent
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun:/dev/net/tun
    restart: unless-stopped
    networks:
      - torrent

  qbittorrent_:
    image: lscr.io/linuxserver/qbittorrent:libtorrentv1-release-5.1.2_v1.2.20-ls89.  # LSIO container
    container_name: qbittorrent_
    volumes:
      - ./qbittorrent/config:/config
      - /srv/downloads:/downloads
      - ./caddy/.data/caddy/pki/authorities/local/root.crt:/usr/local/share/ca-certificates/root.crt
      - ./qbittorrent/etc_hosts:/etc/hosts    # resolv.conf override
    environment:
        - DOCKER_MODS=linuxserver/mods:universal-cron
    restart: unless-stopped
    post_start:
      - command: update-ca-certificates
        user: root
    network_mode: "service:qbittorrent"


networks:
  torrent:
    name: torrent
    driver: bridge

Step 2: Generate Self-signed Certs

The easiest way to do this is to use Caddy's own internal tls mechanism. Caddy will generate its own root and intermediate certificate and (in the official container) place it in /data/caddy/pki/authorities/local. Caddy signs the domain certificates with this trust chain, so Qbittorrent needs to trust the root certificate. To ensure Qbittorrent trusts the root certificate:

1. Mount it into the Qbittorrent container in /usr/local/share/ca-certificates/. This is where the operating system expects to find local trusted certificates.
2. Run update-ca-certificates in the container. This adds any local certificates to the OS store of trusted certs. In practice, this is a file in /etc/ssl/certs/ca-certificates.

A script is provided below that does this automatically, run in crontab.

The Docker Compose file example also ensures the certificates are updated every time a new container is created.

Step 3: Configure Caddy

Now you need to configure Caddy with a Caddyfile to reverse proxy the tracker announce URLs. This config is actually pretty simple once you have the keys in place. Below is an example configuration file and a script to generate a Caddyfile.

Caddy Configuration

I am keeping the extra options pretty simple here---Caddy will upgrade the connection automatically and has mostly sensible defaults.

https://example.com {
        tls internal
        @incoming_conn_close {
                header Connection close
        }
        header @incoming_conn_close keep-alive
        reverse_proxy https://example.com {
                transport http {
                        dial_timeout 5s
                }
                header_up Host {upstream_hostport}
        }
}

Caddyfile Snippets

To shorten and simplify your Caddyfile, use snippets:

(tracker-reverse-proxy) {
        {args[0]} {
                tls internal
                @incoming_conn_close {
                        header Connection close
                }
                header @incoming_conn_close keep-alive
                reverse_proxy {args[0]} {
                        transport http {
                                dial_timeout 5s
                        }
                        header_up Host {upstream_hostport}
                }
        }
}

import tracker-reverse-proxy https://example1.com
import tracker-reverse-proxy https://example2.com
import tracker-reverse-proxy https://example3.com
import tracker-reverse-proxy https://example4.com

This makes it really easy to add domains by maintaining a domains.txt file.

#!/usr/bin/env bash

caddyfile="./caddy/conf/Caddyfile"
caddyfile="test"

if [ -f "$caddyfile" ]
then
    backupfile="$caddyfile".backup-$(date -u +%Y%m%d_%H%M%SZ)
    echo "Making backup to $backupfile"
    cp "$caddyfile" "$backupfile"
fi

printf "(tracker-reverse-proxy) {
        {args[0]} {
                tls internal
                @incoming_conn_close {
                        header Connection close
                }
                header @incoming_conn_close keep-alive
                reverse_proxy {args[0]} {
                        transport http {
                                dial_timeout 5s
                        }
                        header_up Host {upstream_hostport}
                }
        }
}\n\n" > "$caddyfile"

while IFS= read -r domain
do
    printf "import tracker-reverse-proxy %s\n" "$domain"
done < domains.txt >> "$caddyfile"

UDP Windows and HTTP/3

At some point in the future, maybe Caddy will by default go for HTTP/3 if available. It is not strictly needed to prepare for that, but if you want to, you can increase the UDP buffer sizes. This increase can/should be done on the host using sysctl.

https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes

/etc/sysctl.d/40-http2-udp-buffer-size

net.core.rmem_max = 7500000
net.core.wmem_max = 7500000

Step 4: Configure Qbittorrent

The /etc/hosts File

Grab the /etc/hosts out of the glutun container and put it in the file you are bind mounting to the qbit container. From that, make a ./qbittorrent/config/hosts.template file. Now you can append the tracker announce domains to this file. Use a placeholder for the tunnel IP address and use a cron job to keep it up to date.

Example /etc/hosts.template File

I have preferred to redirect the tracker announce domains to the tun0 IP address. I am not 100% sure that is needed, but it seems to be the best way to have it configured such that qbit is only attached to tun0 as well.

127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::  ip6-localnet
ff00::  ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

__tun_ip__ example1.com
__tun_ip__ example2.com

Remember when I said that we are essentially running a man-in-the-middle attack? Well, Qbittorrent will probably complain about that with default settings. You need to go into the advanced settings and turn off SSRF protection, which is specific mitigation for this exact kind of attack.

Also review what interfaces and IP addresses Qbittorrent is binding to and general routing when you are done.

Step 5: Setup cron

Setup cron in the qbittorrent container to regenerate the hosts file every 10 minutes. Also, update the root certificate every year.

*/5 *   * * *   /config/hosts.update.sh
0 1     1 1 1   /srv/docker/update-certificates.sh >/dev/null

Script to update hosts file from the template.

#!/usr/bin/env bash

tun_ip=$(/sbin/ip -4 -o addr show dev tun0 | /usr/bin/awk '{split($4,a,"/"); print a[1]}')
sed "s/__tun_ip__/$tun_ip/" /config/hosts.template > /etc/hosts

.