Rook v1.1.0 device/osd removal

gistfile1.txt

# This will use osd.5 as an example
# ceph commands are expected to be run in the rook-toolbox
1) disk fails
2) remove disk from node
3) mark out osd. `ceph osd out osd.5`
4) remove from crush map. `ceph osd crush remove osd.5`
5) delete caps. `ceph auth del osd.5`
6) remove osd. `ceph osd rm osd.5`
7) delete the deployment `kubectl delete deployment -n rook-ceph rook-ceph-osd-5`
8) delete osd data dir on node `rm -rf /var/lib/rook/osd5`
9) edit the osd configmap `kubectl edit configmap -n rook-ceph rook-ceph-osd-nodename-config`
9a) edit out the config section pertaining to your osd id and underlying device.
10) add new disk and verify node sees it.
11) restart the rook-operator pod by deleting the rook-operator pod
12) osd prepare pods run
13) new rook-ceph-osd-id-5 will be created
14) check health of your cluster `ceph -s; ceph osd tree`