Hey Charles,
For some reason, your situation really resonates with me. I've read this whole thread from start to finish 3 times, so far. I mostly want to give my own answers and guidance to as many points as I can. Please ask follow-up questions if anything's unclear.
First I want to say John Woffindin's answer seems very interesting for your case. Definitely a +1.
And before I get into all of my points, sorry for the Markdown, but I really have to go to sleep now. No way I'm taking the time to format all that in Gmail :-)
Bundler or Gemsets?
If gemsets are working for your right now, then Bundler is not your highest priority. It's a better tool to prevent loading the wrong version of a gem than, gemsets, but only incrementally. That said, it's also a great tool to help you figure out how to upgrade your gems in a controlled manner. So definitely keep it on your shortlist of next steps.
When you decide to go with Bundler, the first step you should do, in my opinion, is to list all your gems in there, with the exact version of each gem you're using, then configure your Rails app to load your gems via Bundler.
Only when that's done successfully, should you start opening up those constraints as needed. If you want more info about the process of upgrading in a controlled manner, let us know.
Git
I think John's suggestion above would be a great first step to start getting everything a little more under control on the demo server. At least it's an approach that doesn't require you to convince and train N coworkers.
But that will only get you so far. So if you want to try introducing git at some point, do it via some of the visual tools: GitX, GitHub for Mac or for Windows.
Rsync
Until you can get a git deployment setup like John suggested, rsync can be a great help.
I've known about rsync for years. I knew it was this supposedly awesome and flexible tool to transfer files efficiently & so on. But every time I looked into it, I was faced with a wall of information and everything seemed so complicated. So many ways to sync, so powerful, so many things I'll have to understand first.
Well, here's a 5 minute primer.
Just sudo apt-get rsync on your server. It will not open up a new port, and shouldn't open up any security hole, you won't have to manage anything. By default, the data and the authentication will go through ssh, which I assume you're already using. That's it.
Now, a few examples:
rsync user@server:readme local_dir/ # copies readme from user's home dir to local_dir
rsync user@server:/var/www/readme local_dir/ # copies readme from absolute path /var/www
rsync -r user@server:/var/www local_dir/ # syncs all of /var/www to local_dir/www
rsync -r user@server:/var/www/* local_dir/ # syncs the content of /var/www to local_dir/, but not in www subdirectory
There's other interesting options like z (compress), a (preserve Linux user rights, useful for backups), --del (mirror), --exclude="*.sw?" (use vim?) and so on. rsync --help will surely give you 2-3 more interesting ideas. But don't let the power stop you. Just like with git, the simple case of rsync is simple.
rsync -rvmz --del --exclude="*.sw?" project/ server:/var/www/project/
Notes
I have a crappy memory. So I take a lot of notes.
notes mysql # opens Dropbox/documents/notes/mysql.md in my default editor
notes linux # same, but for Linux, you get the point
Here's a bash function and two aliases that can help you manage that, if you like this workflow.
alias cdnotes='cd ~/Dropbox/documents/notes/'
alias lsnotes='echo "in ~/Dropbox/documents/notes/" ; echo ; ls ~/Dropbox/documents/notes/'
function notes { pushd ~/Dropbox/documents/notes ; open `ls ./$@.*` ; popd ; }
Use at will. And ping me if you'd like to get the content of some of those :-)
Testing
Someone else said that already and I agree. Delete (or tuck away) all failing tests, or even all tests you don't trust. This seems crazy at first, but the point is to get you to a point where you run your tests and everything's green.
This way, if you break something that's under test, you'll see the failure right away. Because let's face it: you won't notice going from 42 errors, 35 failures to 43 errors, 35 failures, right? Going from 0 to 1 will, though!
Once you've got a trimmed down test suite that's at least green, now it's time to get back into the groove of testing. So if you're about to touch something that has no tests, add some that cover the basics, then do your actual modification, along with new tests to verify the behaviour of the new code.
I also like the idea of using Selenium. I've done that in the past when I've inherited a codebase in a Very Bad Shape™. Heck, I remember seeing Throw new "..." in there. It was supposed to be ruby. So yeah, needless to say, we wanted to ensure correct behaviour from a user's point of view (can log in / see the home page, etc), but we expected to throw away a lot of the code. So we tested from as high as possible, to ensure that what was working (or hobbling along) kept on doing so, exactly as expected. This gave us the confidence to slash away at a lot of problematic code. Of course, as we went along and did things better, we gradually added unit and functional tests to our shiny new code.
We only used Selenium as a local testing tool at the time, but I'm pretty sure you can point it at a domain as well, as a post-deployment smoke test.
Capistrano
Extremely powerful tool, used by a lot of people. But still somehow badly documented, for someone starting from zero with it, IMO.
The most fundamental point to understand about Cap is that it's a tool for your development machine. It doesn't need to be on your server. It's a tool to automate shell commands on your server. It also happens to come with a bunch of recipes that will help you deploy rails & such.
Let's start by getting a quick win.
gem install capistrano
mkdir ~/cap && cd ~/cap
echo "server '[email protected], :web" > Capfile
cap -vT # list available tasks (almost none)
If you've replaced the bogus user / IP address with a server you have access to, you're ready to rock already!
cap invoke COMMAND="ls -l"
cap invoke COMMAND="ps aux"
Now of course that's not super useful yet. But from this very simple base, you can start building tasks you need to do from time to time on your servers.
desc "Tail a file: defaults to /var/log/httpd.log, override with FILE=."
task :tail do
stream "tail -f #{ ENV['FILE'] || '/var/log/httpd.log' }"
end
cap tail
cap tail FILE=/path/to/rails.log
The real magic happens when you start listing multiple servers in your cap recipes.
echo "server '[email protected], :demo" >> Capfile
cap invoke COMMAND='ps aux'
cap tail FILE=/a/log/common/to/both.log # You'll get a stream displaying both
If you want to override where you run a cap task, you can do so with HOSTS:
HOSTS=42.42.42.41,myotherserver.com cap invoke COMMAND="ps aux"
Tasks can be assigned to roles, and can be hooked before/after other tasks:
task :deploy, :role => :app do
run "..."
end
task :tag_release do
run_locally "git tag ..."
end
after :deploy, :tag_release
Anyways, that's for the basics of Capistrano. Of course, a ton of value is alsoin the actual recipes to deploy Rails. If you can get that to work, great! If not (or not yet), Capistrano is still an extremely powerful tool at your disposal.
Process uptime monitoring
OSes come with a way to automatically start some daemons. But in a surprising turn of event, if said processes crash, they don't get restarted automatically.
For that you need to use process monitors such as runit or Bluepill. Being rubyists, let's talk about Bluepill. I find its readme pretty good to get started.
Once you install Bluepill and start monitoring your processes, they'll be automatically restarted if they crash. An important corollary is that now you can't do service mysql stop to stop MySQL, for instance. Bluepill will restart it :-) Now you have to go through Bluepill with bluepill mysql stop.
So just find where your critical process' pid file and log files are, and create a Bluepill configuration file for it:
# /etc/bluepill/mysql.pill
Bluepill.application("mysql") do |app|
app.process("mysql") do |process|
process.start_command = "/sbin/service mysqld start"
process.pid_file = "/var/run/mysqld/mysqld.pid"
end
end
And get Bluepill monitoring the process.
gem install bluepill
bluepill load /etc/bluepill/mysql.pill
bluepill status mysql
Just an aside: bluepill unmonitor mysql doesn't kill mysql, for example. It just stops monitoring it.
You can also use Bluepill to run any simple script in the background as a daemon, without having to implement the daemon feature (pid, detaching, etc.) Let's say you have a script that does something in a loop until you kill it. For instance, I have one to monitor the state of a MySQL slave. Here's an example of a Bluepill to manage such a script:
Bluepill.application("watch", :log_file => "/var/log/bluepill.log") do |app|
app.working_dir = "/u/apps/production/toolbox"
app.process("watch_slave") do |process|
process.start_command = "bin/watch slave #..."
process.stdout = process.stderr = "/var/log/watch_slave.log"
process.daemonize = true
end
end
Finally, Bluepill gets started when your server reboots. So all processes you manage with it will also be restarted.
Halfway conclusion
Ok, so in our private exchange I mentioned that I had a bunch of things I wanted to touch on. I got a little more than half done, I think. I'll cover some more points next time I have an hour or two. Probably next week.
I haven't covered as much of a plan of attack as I had hoped, but maybe that'll be easier to do if we can ever find half an hour to have a live chat.
Cheers,
Mat