Mike Perham mperham

git clone git://github.com/mperham/sidekiq
cd sidekiq && git checkout rails5
cd myapp && bundle
bundle exec sidekiq
# in another terminal tab
bundle exec rails s

Now hit http://localhost:3000/work to create a random set of jobs for sidekiq to perform.

	~/src/sidekiq (master *=)$ bundle exec bin/sidekiqload
	ruby 2.6.0preview3 (2018-11-06 trunk 65578) [x86_64-darwin17]
	2018-11-06T18:17:06.966Z 34703 TID-ovsoikzhj INFO: Booting Sidekiq 5.2.3 with redis options {:db=>13, :id=>"Sidekiq-server-PID-34703", :url=>nil}
	2018-11-06T18:17:10.046Z 34703 TID-ovsoikzhj ERROR: Created 100000 jobs
	2018-11-06T18:17:11.058Z 34703 TID-ovsofju03 ERROR: RSS: 56880 Pending: 94926
	2018-11-06T18:17:12.067Z 34703 TID-ovsofju03 ERROR: RSS: 57084 Pending: 89629
	2018-11-06T18:17:13.080Z 34703 TID-ovsofju03 ERROR: RSS: 57104 Pending: 84563
	2018-11-06T18:17:14.093Z 34703 TID-ovsofju03 ERROR: RSS: 57104 Pending: 79536
	2018-11-06T18:17:15.105Z 34703 TID-ovsofju03 ERROR: RSS: 57104 Pending: 74438
	2018-11-06T18:17:16.117Z 34703 TID-ovsofju03 ERROR: RSS: 57108 Pending: 69405

	=begin
	This script attempts to reproduce poor glibc allocator behavior within Ruby, leading
	to extreme memory fragmentation and process RSS bloat.

	glibc allocates memory using per-thread "arenas". These blocks can easily fragment when
	some objects are free'd and others are long-lived.

	Our script runs multiple threads, all allocating randomly sized "large" Strings between 4,000
	and 40,000 bytes in size. This simulates Rails views with ERB creating large chunks of HTML
	to output to the browser. Some of these strings are kept around and some are discarded.

	package storage

	import (
	"crypto/rand"
	"encoding/base64"
	"fmt"
	"os"
	"testing"
	"time"

	class Counter
	include Sidekiq::Worker

	# I'd use a separate Redis instance from Sidekiq
	REDIS = ConnectionPool.new(size: 5) { Redis.new(...) }

	# Call this API in your Rails app code to increment counters
	# as the user does things.
	def self.incr(name, amount=1)
	key = "counter-#{name}-#{current_window}"

	~/src/ent/myapp (master *=)$ bundle exec derailed bundle:objects
	Measuring objects created by gems in groups [:default, "production"]
	Total allocated: 4246025 bytes (34726 objects)
	Total retained: 437421 bytes (4179 objects)

	allocated memory by gem
	-----------------------------------
	1348174 puma-3.6.0
	749251 sidekiq/lib
	538517 pro/lib

	require "./src/sidekiq"

	`redis-cli flushdb`

	class LoadWorker
	include Sidekiq::Worker

	perform_types Int64
	def perform(idx)
	end

	require 'benchmark'
	require 'sidekiq-ent'
	require 'redis-lock'
	require 'redis-semaphore'
	require 'ruby_redis_lock'

	# monkey patch to remove exponential backoff in pmckee11-redis-lock,
	# otherwise this benchmark does not complete successfully.
	class Redis
	class Lock

	$ ruby -v
	ruby 2.0.0p598 (2014-11-13) [x86_64-linux]
	$ irb
	irb(main):001:0> require 'json'
	=> true
	irb(main):002:0> quit

	$ bundle exec irb
	rirb(main):001:0> require 'json'
	LoadError: cannot load such file -- json

	class MyCode
	def process
	do_something if leader?
	end

	def leader?
	Sidekiq::Senate.leader?
	end
	end