riak-admin force-remove should not exist.
It's Friday evening, almost time to head out of the office for a nice long weekend. Your Riak cluster has been running along, everything fine. All of a sudden, the SSD in one of your Riak nodes decides to go up in a ball of flames. So you, being the good sysadmin that you are, immediately hop on the phone with your hardware vendor and order a new SSD. They tell you that you'll have it on Monday morning. Clearly you can't leave a broken node in your Riak environment, so you'll want to remove it from the cluster. You sit down at your terminal, hop on to a working Riak node and type
riak-admin force-remove [email protected]
NOOOOOOOOOOOOOOOOOOOOOOOOO!!!!
The only time you should ever run force-remove on a cluster is the moment when a node is unavailable, and —more importantly— you intend to reduce your node count. From our experience, this is almost never the case. A Riak cluster will handle a single node failure quite gracefully because of technologies like "read repair" and "hinted handoff".
But "Charlie", you say, "I need to take it out to put the new instance back in, right?" Again I say, "No." All you have to do to bring up a node in its former place is to bring up a new node with the same configuration and ring as its original. You have been backing up your configurations and ring files, haven't you? Even if not, we have a plan for that. But if you have, restore your config and ring files, start the node, and it will hop back into the cluster as though nothing were ever wrong. The cluster members who have been holding handoff data for it will start transferring it back to the prodigal node and all will be well. If replica rot is a concern (long lived, infrequently read data), you can either allow active anti-entropy to repair your data, or do it manually.
"So why is removing it bad? I'm trying to keep things tidy." Three words; Risk of rediagonalization. Removing a cluster node causes partition ownership changes. When there is no way for the partitions owned by the cluster member that you are removing to go to nodes that do not hold adjacent partitions, rediagonalization occurs. Rediagonalization simply takes the ownership as it sits, and completely rebuilds it so there are no adjacent partitions on the same node. This could, and surely would, involve the meaningless transfer of much data from one healthy node to another one just to satisfy this requirement. Similarly, once you go to add the node back in, ownership handoff will occur again to populate the node.