Prepare Coreos for Ceph-Docker on Kubernetes

Assumptions

This guide relies on the osd-directory method of described in ceph-docker documentation, which assumes that osd dedicated disk is already formatted an mounted in '/home/core/data/ceph/osd'.

Pre-requistes

jinja2-cli on windows

Ceph docker relies on jinja2 for template to generate yaml templates to be deployed in kubernetes Install Python 2.7, then run the following command lines to install jinja2-cli.

export PATH=${PATH}:/c/Python27/Scripts

easy_install pyaml jinja2-cli

Install daemon set

cd ~/git/ceph-docker/examples/kubernetes-coreos
pushd ~/git/coreos-kubernetes/multi-node/vsphere-powercli/ && . ./init-kubectl.sh && popd
kubectl create namespace ceph
kubectl create -f install-ds.yaml

Tag node for storage

cd ~/git/ceph-docker/examples/kubernetes

kubectl label node 192.168.251.201 node-type=storage
kubectl label node 192.168.251.202 node-type=storage
kubectl label node 192.168.251.203 node-type=storage

Create Ceph secrets

cd generator
# ./generate_secrets.sh all `./generate_secrets.sh fsid`
./generate_secrets.sh all `./generate_secrets.sh fsid` osd_public_network=10.2.0.0/16 osd_cluster_network=10.2.0.0/16

kubectl create namespace ceph

kubectl create secret generic ceph-conf-combined --from-file=ceph.conf --from-file=ceph.client.admin.keyring --from-file=ceph.mon.keyring --namespace=ceph
kubectl create secret generic ceph-bootstrap-rgw-keyring --from-file=ceph.keyring=ceph.rgw.keyring --namespace=ceph
kubectl create secret generic ceph-bootstrap-mds-keyring --from-file=ceph.keyring=ceph.mds.keyring --namespace=ceph
kubectl create secret generic ceph-bootstrap-osd-keyring --from-file=ceph.keyring=ceph.osd.keyring --namespace=ceph
kubectl create secret generic ceph-client-key --from-file=ceph-client-key --namespace=ceph

cd ..

# Fix ceph-mon permission
kubectl create -f https://github.com/ReSearchITEng/kubeadm-playbook/raw/master/allow-all-all-rbac.yml

Initialize Ceph

kubectl create \
-f ceph-mds-v1-dp.yaml \
-f ceph-mon-v1-svc.yaml \
-f ceph-mon-v1-dp.yaml \
-f ceph-mon-check-v1-dp.yaml \
-f ceph-osd-v1-ds.yaml \
--namespace=ceph

Format disks

On each storage dedicated node. Open an SSH session to the host, then run the following commands to create a partition and format as xfs.

# Create partition
# sudo parted -s /dev/sdb mklabel gpt mkpart primary 0% 33% mkpart primary 34% 66% mkpart primary 67% 100%
sudo parted -s /dev/sdc mklabel gpt mkpart primary xfs 0% 85%

# Format partition
sudo mkfs.xfs /dev/sdc -f

# Check partition
sudo blkid -o value -s TYPE /dev/sdc

# Mount partion
sudo mkdir -p /home/core/data/ceph/osd
sudo mount /dev/sdc /home/core/data/ceph/osd

fjudith/ceph-docker_kubernetes.md