Michael Mahemoff mahemoff

iTunes categories in most popular feeds at https://player.fm/featured/programming

Technology
Technology,Tech News,Technology,Software How-To
Technology
Technology,Software How-To
Technology,Tech News,Technology,Software How-To
Technology
Technology,Tech News,Education,Education Technology,Technology,Software How-To

Put database in read-only mode and capture master log position:

mysqL> FLUSH TABLES WITH READ LOCK; SET GLOBAL read_only = ON; SHOW MASTER STATUS;`

Copy entire mysql data folder:

bash> cp -r /var/lib/mysql /home/me/prod_varlibmysql20170715 ; date`

When copy is complete, immediately turn off read-only mode:

The following MySQL command will show you long-running processes (exceeding 5 seconds in this case).

select * from information_schema.processlist where command <> 'sleep' and time > 5;

You can run it from console to view those happening now, or you could make a script to poll it periodically, with something like:

echo "select * from information_schema.processlist where command <> 'sleep' and time > 5;" | mysql > long-running.log

(See also https://serverfault.com/questions/416380/log-slow-queries-killed-before-completion/858093)

RATIONALE

It's important to have useful test data during development and debugging, resembling real-world data. It can be cumbersome to setup and maintain though. This is a mini-framework to help with seeding in Rails using a Domain Specific Language approach.

DESIGN APPROACH

I got tired of maintaining seeds.rb idempotently. Too dangerous to run this directly into production and overall, cumbersome to make it idempotent (ie work whether or not the objects already exist). So I decided to just make something that's just used for development, starting from an empty DB each time. I'm now maintaining a separate folder of individual data migrations to be performed in production. There's also separate fixtures for testing (separate data than this for performance reasons, and because tests need different kinds of data).
It's modular enough for parts to be invoked during testing.
The modularity is also part of a domain-specific language approach to seeding, as can be seen in the examples.

	### Install Python and aws

	[pip install awscli](https://docs.aws.amazon.com/cli/latest/userguide/installing.html)

	You may need to add it to your path, e.g. export PATH="$PATH:/home/player/.local/bin"

	### Setup AWS S3 bucket

	* In S3, create a new backup bucket. You may wish to set it up with versioning and lifecycle management rules so that you can just keep pushing to the same object and old versions will be deleted and/or moved to Glacier. Also recommended to establish tags and logging if cost is likely to be significant and therefore should be tracked.
	* In IAM, create a programmatic user and ensure it has an access key and secret access key

	Places Geek

	Unusual places

	https://en.wikipedia.org/wiki/Point_Roberts,_Washington - US coastal enclave in Canada

	https://en.wikipedia.org/wiki/Dahala_Khagrabari - Triple enclave until 2015 (Indian enclave inside Bangladeshi enclave inside Indian enclave inside Bangladesh)

	http://www.paulplowman.com/stuff/house-address-twins-proximity/ - UK house address twins, including 2 Manchester houses beside each other with same address

	# ruby 2.3.1p112 MacOS on MBP Retina late 2013
	# Benchmarking some techniques at:
	# https://stackoverflow.com/questions/12911869/is-there-a-faster-way-to-generate-random-string-in-ruby
	# https://stackoverflow.com/questions/88311/how-to-generate-a-random-string-in-ruby

	# Results show Object.hash is easily the fastest and shuffle is the slowest (as well as not being great as it doesn't support repeating characters)
	# SecureRandom is a good compromise, especially if the string is needed for security purposes. It's 5 tims slower than Object.hash, but Object.hash
	# is an int that would need converting to string to make an equivalent, smaller, representation.

	[35] pry(main)> all=[]; Benchmark.measure { 10000.times { all << ('a'..'z').to_a.shuffle[0,8].join} }

	[<Linode api_id=499946, label='psaa'>]
	[ { u'AVAIL': { u'10': 500,
	u'11': 500,
	u'2': 500,
	u'3': 500,
	u'4': 500,
	u'6': 500,
	u'7': 500,
	u'8': 500,
	u'9': 500},

	Class SidekiqUtils

	def self.queues
	::Sidekiq::Stats.new.queues.keys.map { \|name\| ::Sidekiq::Queue.new(name) }
	end

	def self.find_queue(name)
	self.queues.find { \|q\| q.name==name.to_s }
	end

	- name: Backup original my.cnf
	copy: remote_src=true src=/etc/mysql/my.cnf dest=/tmp/my.cnf.recent

	- name : Update my.cnf
	template: src=my.cnf.j2 dest=/etc/mysql/my.cnf owner=mysql mode=0644

	# diff returns error code if different, so we ignore "errors"
	- name: Check if my.cnf changed
	command: diff /etc/mysql/my.cnf /tmp/my.cnf.recent
	ignore_errors: true