Thinking Sphinx has some convenient tasks to manage searchd processes per-app. You can deploy multiple apps to a single server and have separate searchd processes for each one. This is great! What's not so great is figuring out how to monitor each one, while still retaining the normal functionality of the rake tasks.
"Just use [some monitoring software]!". Great idea. Except [some monitoring software] probably doesn't have a concept of "allow this service to be down while [some condition is true]". In our case, [some condition] is ts:rebuild
. ts:rebuild
is a convenience method to stop, index, and then restart searchd. It is a normal operation, necessary when you've made changes to your index schema (by adding, removing, or changing an attribute in your index definition).
The problematic flow that we're trying to solve looks like:
- God.rb is monitoring searchd.
- App user runs
ts:rebuild
, which stops searchd. - God.rb notices searchd has stopped running, and promptly restarts it (as it should, beca