Skip to content

Instantly share code, notes, and snippets.

@ckerr

ckerr/bittorrent-client-tests.md

Created September 20, 2019 22:21
Show Gist options
  • Save ckerr/89882362c65b819614051935364b9eaa to your computer and use it in GitHub Desktop.
Save ckerr/89882362c65b819614051935364b9eaa to your computer and use it in GitHub Desktop.
Download ZIP
BitTorrent Client Performance Tests
Raw
bittorrent-client-tests.md

BitTorrent Client Performance Tests

Performance Constrained by Low-Speed 300 KB/s Down, 39 KB/s Up ADSL Provisioning

Benchmarks conducted 18-19, 22-24, 26-28 February; 01-07, 15-22 March 2010

BitTorrent Client Time to Download ubuntu-9.10-desktop i386.iso Memory Use While Downloading User CPU (MM:SS) Sys CPU (MM:SS) Engine GUI
BitTornado 0.3.18 1st run 53:33 19.3 MiB 2:15 0:53 Built-in Curses
BitTornado 0.3.18 2nd run 59:34 79.8 MiB 7:51 3:47 Built-in Curses
BitTornado 0.3.18 3rd run 59:05 90.3 MiB 8:36 3:37 Built-in Curses
Deluge 1.2.0 1st run 1 57:23 49.2 MiB 3:25 0:53 rasterbar 0.14.8 GTK+
Deluge 1.2.1 1st run 59:03 46.6 MiB 3:47 1:00 rasterbar 0.14.9 GTK+
Deluge 1.2.1 2nd run 58:49 46.6 MiB 3:24 0:58 rasterbar 0.14.9 GTK+
Deluge 1.2900-dev 1st run 2 58:29 38.6 MiB 3:20 0:54 rasterbar 0.14.8 GTK+
Deluge 1.2900-dev 2nd run 56:58 38.8 MiB 3:27 0:55 rasterbar 0.14.8 GTK+
Deluge 1.2900-dev 3rd run 3 57:00 38.3 MiB 3:12 0:55 rasterbar 0.15.0 GTK+
KTorrent 3.3.4 1st run 56:16 19.9 MiB N/A ^4 N/A Built-in Qt
KTorrent 3.3.4 2nd run 57:41 49.6 MiB 2:28 1:50 Built-in Qt
KTorrent 3.3.4 3rd run 58:47 55.5 MiB 2:23 1:38 Built-in Qt
KTorrent 3.3.4 4th run 57:57 51.9 MiB 2:12 1:35 Built-in Qt
Mainline 4.4.0 1st run 54:26 40.8 MiB 3:18 1:02 Built-in Curses
Mainline 4.4.0 2nd run 55:13 27.6 MiB 3:39 1:17 Built-in Curses
Mainline 4.4.0 3rd run 54:52 29.9 MiB 3:51 1:28 Built-in Curses
Monsoon 0.21 1st run ^5 913:59 34.6 MiB 158:57 19:18 MonoTorrent 0.72 GTK+
Monsoon 0.21 2nd run 75:39 30.4 MiB 14:34 2:52 MonoTorrent 0.72 GTK+
Monsoon 0.21 3rd run 55:30 33.2 MiB 11:05 2:31 MonoTorrent 0.72 GTK+
Monsoon 0.21 4th run 122.55 32.8 MiB 24:06 4:22 MonoTorrent 0.72 GTK+
qBittorrent 2.1.5 1st run 55:26 33.2 MiB 2:50 0:38 rasterbar 0.14.8 Qt
qBittorrent 2.1.5 2nd run 55:23 36.9 MiB 3:04 0:42 rasterbar 0.14.8 Qt
qBittorrent 2.1.6 1st run 56:28 41.0 MiB 3:07 0:44 rasterbar 0.14.8 Qt
qBittorrent 2.2.0 1st run 56:28 36.4 MiB 3:01 0:42 rasterbar 0.14.9 Qt
qBittorrent 2.2.0 2nd run 55:15 34.6 MiB 2:52 0:40 rasterbar 0.14.9 Qt
rTorrent 0.8.6 1st run 6 52:52 invalid: 16.1 MiB 0:16 0:19 rakshasa 0.12.6 Curses
rTorrent 0.8.6 2nd run 7 55:25 invalid: 9.2 MiB 0:47 0:22 rakshasa 0.12.6 Curses
rTorrent 0.8.6 3rd run 53:53 invalid: 5.8 MiB 0:49 0:22 rakshasa 0.12.6 Curses
rTorrent 0.8.6 4th run 53:44 invalid: 4.4 MiB 0:48 0:22 rakshasa 0.12.6 Curses
Transmission 1.76 1st run 55:29 10.4 MiB 1:30 0:31 libtransmission GTK+
Transmission 1.90 1st run 57:21 10.1 MiB 1:33 0:45 libtransmission GTK+
Transmission 1.91 1st run 60:20 N/A 1:15 0:47 libtransmission GTK+
Transmission 1.91 2nd run 60:40 11.2 MiB 1:15 0:45 libtransmission GTK+
Transmission 1.91 3rd run 59.46 10.6 MiB 1:40 0:47 libtransmission GTK+
Transmission 1.92 1st run 54:54 10.3 MiB 1:15 0:34 libtransmission GTK+
Transmission 1.92 2nd run 55:27 10.5 MiB 1:11 0:34 libtransmission GTK+
Transmission 1.92+ svn 10396 1st run 54:32 11.7 MiB 1:27 0:33 libtransmission GTK+
Transmission 1.92+ svn 10396 2nd run 54:35 10.8 MiB 1:13 0:33 libtransmission GTK+
uTorrent (Wine) 2.0 (18620) 1st run ^9 54:12 26.7 MiB 2:06 1:37 Built-in Wine
uTorrent (Wine) 2.0 (18620) 2nd run 53:52 20.8 MiB 2:03 1:25 Built-in Wine
uTorrent (Wine) 2.0 (18620) 3rd run 55:03 26.5 MiB 2:12 1:44 Built-in Wine
Vuze 4.3.1.4 1st run 53:40 246.5 MiB 5:44 0:51 Built-in Java SWT
Vuze 4.3.1.4 Classic View 1st run 53:58 243.5 MiB 5:01 0:47 Built-in Java SWT
Vuze 4.3.1.4 Classic View 2nd run 53:54 232.5 MiB 4:54 0:48 Built-in Java SWT

Methodology

System: Asus A8V Deluxe mainboard, AMD Athlon(tm) 64 Processor 3200+. 2 GB DDR RAM

Network: 10/100 Ethernet, IPv4 NAT, ClarkConnect linux gateway/firewall/router PC ^10, AT&T branded 2Wire 2701HG-B router/modem set to Bridged Ethernet mode, i.e., only the modem component is active

Broadband: AT&T ADSL provisioned as 300/39 KB/s; 240/39 KB/s typically best actual

Operating System: Fedora 12 Linux - #1 SMP i686 athlon i386 GNU/Linux

Kernels 11: kernel-PAE-2.6.32.8-58.fc12.i686; kernel-PAE-2.6.32.9-67.fc12.i686; kernel-PAE-2.6.32.9-70.fc12.i686

Conditions:

  • Only one client was running per test. All tests were run on the same workstation, described above.

  • No other systems in the LAN were active on the Internet during tests, except for IRC chat (no DCC) by one person.

  • All tests were performed between 09:00 and 01:00 EST.

  • Each client was tested at least three times, never in back-to-back sequence, and usually on different days.

  • Each test was run under the unix time utility as: time [program-name] [options] ubuntu-9.10-desktop-i386.iso.torrent.

  • Each client was tested using its initial setup defaults, with two exceptions, made during a previous launch to edit preferences where possible, or incorporated into the command line when required:

    (1) Each client was configured to use port 51421 (both TCP and UDP available); and
    (2) Each client was configured to save torrents in a client-named directory on the ext4 filesystem used.

  • Memory use ^12 was captured at 98% completion of each test run, by manually invoking: python ps_mem.py [client-name]

  • Immediately upon 100% completion and switch to seeding, each client was quit and its timed results recorded on exit.

  • Multiple client updates were released during the test span. I left all results in the table to show release variability in an active development environment.

Comments

These tests provide a decently rigorous picture of the capabilities and differences among several popular linux bittorrent clients, running on one system under one linux distribution over one type of ADSL broadband provisioning. Note that the tests are all for a single torrent, chosen for its large and well seeded swarm to provide as much reproducibility as any bittorrent swarm can likely offer. Among the things these tests do not provide are:\

(1) Scaling results for clients under multiple torrent loads;
(2) Performance differences between clients when each has been optimally configured beyond initial setup defaults;
(3) Performance differences between clients on more generously provisioned broadband;
(4) Features or performance of daemon, remote, web ui, or other variants on the standard client; and
(5) Effects (if any) of different linux distributions upon bittorrent client performance.

The single most significant performance constraint of these tests is the relatively low broadband speed as provisioned by the ISP. At such low speeds, choice of bittorrent client appears to make little difference in download metrics, leaving resource use and feature set as the primary salient factors. At this time, in this location, AT&T is not known to employ bittorrent throttling.

Thanks to Charles Kerr for posting an earlier version of these tests, upon which this version is based. Additional thanks to developers of various clients, whose help and suggestions improved these tests in many ways. All errors and inconsistencies in these tests are the fault of the author, Lacrocivious Acrophosist, who probably made the error specifically to annoy you, personally.

Special Note

[*] Memory use reports for rTorrent by this python script are not representative of actual memory use by rTorrent while downloading. Developers involved with multiple competing clients tested here noted this discrepancy. One stated, "rTorrent relies more upon the (kernel) page cache table than most other clients, and so memory usage reported in the table does not necessarily reflect how much free memory a system would need to run rTorrent with satisfactory results."

During tests, rTorrent's info screen confirms its own reported memory use differs significantly from that reported by the python script. That screen's "Memory usage:" field varies dynamically during download between ~22.0 MB and ~40.0 MB, cycling in 30-second intervals, most often from a "floor" of ~25.0 MB and increasing to ~36.0 MB before falling back to ~25.0 MB and repeating. Thus, a rough estimate of actual memory use in these tests is ~38 MB.

Footnotes

[1]: Deluge 1.2.0 had an overdownloading bug frowned upon by some trackers; this issue is fixed in 1.2.1 and later versions. The key to this fix is libtorrent-rasterbar 0.14.9, more than changes to Deluge itself.

[2] Awaiting solution to a trivial 1.2.1 build problem on the test system, two interim runs used 1.2900-dev with libtorrent-rasterbar 0.14.8.

[3] Deluge recently switched from Subversion to Git; this build is from a fresh git pull of the master branch, incorporating libtorrent-rasterbar 0.15.0.

[4] KTorrent forks on load, preventing the time utility from recording accurate start/end/CPU-use. Memory use difference on this run may be due to this fork (the python ps_mem.py ktorrent report was captured, but the time report was not). Subsequent tests were run with time using the ktorrent --nofork option, which enables time functionality.

[5] Monsoon has a poor endgame strategy. Repeatedly but not always, it would download to about 99% completion and then hang indefinitely on the last few pieces. Monsoon appears ill-suited for normal use, and was tested here only for comparison with prior tests.

[6] This test run used rTorrent from the Fedora 12 repository due to a naming conflict between libtorrent-rasterbar and libtorrent-rakshasa, before I found a workaround.

[7] This is the same libtorrent-rakshasa version as in the Fedora 12 repository, but both rTorrent and libtorrent-rakshasa were built from project source for this and subsequent test runs.

[8] PEPCAK attack. Terminal was in the wrong directory to find python ps_mem.py, and by the time I realized it the torrent had finished downloading.

[9] uTorrent memory use only. Wine components reported separate memory use: explorer.exe, 3.8 MiB; services.exe, 1.2 MiB; winedevice.exe, 2.8 MiB; and wineserver, 2.1 MiB; total 9.9 MiB. Wine version: wine-1.1.38-1.fc12.i686.

[10] ClarkConnect have rebranded themselves ClearOS. I have found their products exemplary since I began using them in 2003 (they don't pay me).

[11] Normal system updates, including the Fedora 12 updates-testing repository, were enabled during the test runs, and kernels changed during that span. Client tests were significantly staggered within the testing time span, and none of the operating system changes appeared to have any observable effect upon any client performance.

[12] I do not pretend to fully understand this python script. I am not a programmer. The script is well commented and used here for continuity with prior tests.

@GATOQSECOMIO
Copy link

Nice comparison! I'm looking forward to updating. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment