Adrien Rey-Jarthon jarthod

Example with wget:

wget --no-parent -rpk --wait=1 --random-wait https://wbsite.com/page

--no-parent prevents going up in the URLs --wait=1 --random-wait slows down request and add jitter to go easy on the server

If it needs a connected session:

--header="Cookie: session=xxxx"

Prepare (Ubuntu 22.04 Live):

sudo su
apt install pv

Dump disk (or partition) to local file

zstd < /dev/nvme0n1 | pv > laptop.img.zst

Dump disk (or partition) to compressed image over network (NAS)

Results

Server	Price*	CPU (1 thread)	CPU (4 threads)	IO
Scaleway Start1-XS Atom C3955, 25G NVMe	$4	21 sec P₉₅ = 2.1ms	21 sec P₉₅ = 14ms	🚗 653 IO/s, 10 Mb/sec P₉₅ = 0.40ms
Scaleway VC1-S Atom C2750, 50G SSD	$4	46 sec ⚠️ P₉₅ = 4.6ms	23 sec P₉₅ = 14ms	🚲 289 IO/s, 4.5 Mb/sec P₉₅ = 0.39ms
Hetzner CPX11 AMD EPYC, 40G SSD	$5	20 sec P₉₅ = 3.2ms	9.6 sec P₉₅ = 11ms	✈️ 3656 IO/s, 57 Mb/sec P₉₅ = 0.29ms
Hetzner CX21 Intel Xeon, 40G SSD	$6	24 sec P₉₅ = 2.8ms	12.5 sec P₉₅ = 11ms	🚤 1946 IO/s, 30 Mb/sec P₉₅ = 0.97ms
AWS t2.micro 1G*Intel E5-2676 v3 2.4

	# Examples:
	#
	# expect { code }.to change { finds("users") }.by(3)
	# expect { code }.to change { updates("contents") }.by(1)
	# expect { code }.not_to change { inserts }
	#
	# MongoSpy.flush
	# ..code..
	# expect(MongoSpy.queries).to match(
	# "find" => { "users" => 1, "contents" => 1 },

	#!/usr/bin/env ruby

	# Tries to detect when mongo is up but super slow (ex: IO issue)

	THRESHOLD = 20_000 # ms

	def test_mongo tries: 5
	out = `echo -e "db.isMaster()\ndb.getReplicationInfo()" \| mongo mongodb://localhost/?socketTimeoutMS=#{THRESHOLD} 2>&1`
	res = $?
	if res != 0 && !out['Connection refused']

	#!/usr/bin/env ruby

	# Provides a system wide file lock to ensure no more
	# than X process is running at the same time.

	# Example:
	#
	# process_semaphore!({
	# prefix: "/tmp/my_process",
	# limit: 2

	require 'zstd-ruby'

	class CompressedHash < Hash
	DICTIONARY = IO.read('config/dictionaries/1')
	def mongoize
	if size > 0 # only compress non-empty hash
	# BSON::Binary.new(Zstd.compress(self.to_bson.to_s))
	BSON::Binary.new(Zstd.compress_using_dict(self.to_bson.to_s, DICTIONARY))
	else
	self

	sudo cat /proc/sys/net/netfilter/nf_conntrack_max
	echo 500000 \| sudo tee /proc/sys/net/netfilter/nf_conntrack_max
	sudo cat /proc/sys/net/netfilter/nf_conntrack_max