bzub bzub

Rook On Kubernetes (The Hard Way)

Why?

This only exists to help me learn about the inner workings of [Rook][rook]/[Ceph][ceph] and to test design ideas for Rook and [ceph-docker][ceph-docker] as I spin the wheels.

You should use [rook-operator][rook-operator] to try Rook on Kubernetes. Not this.

Status

Ideas I've implemented or want to implement.

TODO

Define defaults and override rules.
- Should there be a "default" DataPolicy that is assumed unless overridden?
- The DataPolicies, DataEndpoints, and StorageNodes lists are ordered. How can we use that ordering to make predictable results from selections?
  - Proposal for StorageNodes: The last group of selectors (all must match) to match a host is what applies.
  - Proposal for DataPolicies: The last policy to match a device or directory is what applies.
  - Proposal for DataEndpoints: Inclusive match. If any one rule describes a device or directory then it's added to the group.
Seems like it could be possible to combine DataPolicies and DataEndpoints into one resource type.

Hopefully there are few flaws in the logic and this proposal is a good compromise between simplicity and power.

Container Linux Config / Ignition

Here's the relevant bits that configure IPs on loopback via systemd-networkd. They are the same on all my nodes, so no variables.

networkd:
  units:
    - name: 00-vip-lo.network
      contents: |
        [Match]
        Name=lo
        [Network]

This workaround solves many issues where software in a Kubernetes pod expects a static, rarely changed IP address. Since we bind to a Kubernetes Service IP, it comes with many benefits such as automatic DNS entries within the cluster, health checking automation, and abstraction of network details. In fact these features of Services are used in the pod examples here to avoid hard-coding IP addresses and ports into the pod definitions.

TODO: More details.

Results

    cluster 9ece8946-1b9d-4bb5-8d51-34c968119795

Docs and manifests are now in the kube-router repository.

	The osd crush location config option is no longer supported. Please update your ceph.conf to use the crush location option instead.

	cephfs: tool/cephfs: displaying “list” in journal event mode (pr#11236, huanwen ren)
	cephfs: tools/cephfs: add pg_files command (issue#17249, pr#11026, John Spray)
	cephfs: tools/cephfs: add scan_links command which fixes linkages errors (pr#11446, Yan, Zheng)
	core,tools: Added append functionality to rados tool. (pr#11036, Tomy Cheru)
	core,tools: vstart: decrease pool size if <3 OSDs (pr#11528, John Spray)
	mon: if crushtool config is empty use internal crush test (pr#11765, Bassam Tabbara)
	performance,tools: rados: add hints to rados bench (pr#12169, Sage Weil)
	rbd,tools: rbd : make option –stripe-unit w/ B/K/M work (pr#12407, Jianpeng Ma)

	apiVersion: extensions/v1beta1
	kind: DaemonSet
	metadata:
	labels:
	app: rook
	role: osd
	rook_cluster: rookcluster
	name: osd
	spec:
	selector:

	# This ConfigMap is used to configure a self-hosted Calico installation.
	kind: ConfigMap
	apiVersion: v1
	metadata:
	name: calico-config
	namespace: kube-system
	data:
	# Configure this with the location of your etcd cluster.
	etcd_endpoints: "http://kube-etcd:2379"

	apiVersion: v1
	kind: ConfigMap
	metadata:
	name: kube-router-cfg
	namespace: kube-system
	labels:
	tier: node
	k8s-app: kube-router
	data:
	cni-conf.json: \|

	###
	### Stage 00-new-cluster complete. Logging to /tmp/np-test-logs/00-new-cluster.\{iptables,ipset\}
	###
	networkpolicy "test-network-policy" created
	###
	### Stage 01-np-created complete. Logging to /tmp/np-test-logs/01-np-created.\{iptables,ipset\}
	###
	deployment "test-np-server" created
	service "test-np-server" exposed
	###