My dev-ops "Learn from my mistakes" list

devops_LFMM.markdown

• Use a version control system that allows you to deploy exactly what you want to, not "everything up to and including a particular checkin", e.g., use Git, not Perforce.

• Have a centralized admin user account on servers, but also have individual user accounts and make people log in using their individual and then sudo. This strategy provides a better audit trail than logging in directly to the cetnral admin account.

• Don't build a job queue system from scratch using cron. There are so many better-behaved job queue systems in various languages.

  • Cron can't schedule more than some maximum number of jobs in a single minute, so it may miss running some jobs.
  • Cron can't re-run missed jobs, so you have to come up with a way yourself of identifying any missed jobs (either because of the above issue or because cron or the system was down) and running them.

• Databases sometimes need to be versioned in ways besides ActiveRecord-type migrations. For example, if two code branches differ in the columns they assume are present in a particular table, it may be too expensive to run the migrations that convert between the two versions of the table when switching to work on the other code branch. If sufficient storage space is available, keeping distinct copies of the database corresponding to the different versions of the code may be more time-efficient.

• Specifying default values for text-type columns via the database schema makes for a lot more work when converting the database from one character set to another (for example, MySQL's CHANGE or MODIFY commands don't preserve existing column attributes other than index attributes such as PRIMARY KEY or UNIQUE, so you have to gather up all the existing attributes yourself and apply them when modifying a column). So maybe it's better to keep that in the application's logic, even though database fans will shake their heads and tsk at the thought.

• If the Content-Type on a POST request is wrong (e.g., it says application/x-www-form-urlencoded but the request body is actually an XML document, not a list of name=value pair lines), the data in the body will probably be already discarded by the Web server before any filter (e.g., Rack middleware) you install has a chance to look at it.

• rake middleware == RAILS_ENV=development rake middleware