danehans · February 3, 2017 17:30
diff --git a/Calico and KubeAdm b/Calico and KubeAdm
 Provisioning to use Calico
 1/20/2017
 First, tearing down the cluster, by doing this on each node:
 kubeadm reset
 rm -rf .kube
 systemctl start kubelet.service

 Did yum -y update on all nodes as well. Downloaded http://docs.projectcalico.org/v1.6/getting-started/kubernetes/installation/hosted/kubeadm/calico.yaml and then ran "kubectl create -f calico.yaml".

 Starting up cluster (with just the master node).

 kubeadm init

 Have:
 	kubeadm join --token=482742.64ff98647d01bf2c 10.87.49.77

 kubectl taint nodes --all dedicated-
 kubectl apply -f calico.yaml
 kubectl get pods --all-namespaces

 Issue: It is unable to download calico/ctl v0.23.1. It does not exist. Options are to use v1.0.0 (and use same for calico/node, to keep them similar?) or drop to v0.23.0 (and either keep calico/node at v0.23.1 or set it to v.23.0 too).  Trying v1.0.0. Stopped cluster and then restarting, after updating calico.yaml.

 Have:
 	kubeadm join --token=086198.58f8a53b0c2391be 10.87.49.77

 Looks like calico/configure is not starting up (message about calicoctl unknown command pool). Also dns pod not coming up, says CNI config is uninitialized. Will try v0.23.0 for both node and ctl…

 Have:
 	kubeadm join --token=b3be10.7356d594b9de79a2 10.87.49.77

 Looks like everything I running…
 [root@devstack-77 ~]# kubectl get pods --all-namespaces
 NAMESPACE     NAME                                  READY     STATUS    RESTARTS   AGE
 kube-system   calico-etcd-4zkxh                     1/1       Running   0          10m
 kube-system   calico-node-c8pkm                     2/2       Running   0          10m
 kube-system   calico-policy-controller-2swdt        1/1       Running   0          10m
 kube-system   dummy-2088944543-c5xlk                1/1       Running   0          12m
 kube-system   etcd-devstack-77                      1/1       Running   0          11m
 kube-system   kube-apiserver-devstack-77            1/1       Running   1          11m
 kube-system   kube-controller-manager-devstack-77   1/1       Running   0          11m
 kube-system   kube-discovery-1769846148-8fp6j       1/1       Running   0          12m
 kube-system   kube-dns-2924299975-r04zb             4/4       Running   0          12m
 kube-system   kube-proxy-3lmlh                      1/1       Running   0          12m
 kube-system   kube-scheduler-devstack-77            1/1       Running   0          11m

 No IPv6. Need to add IPAM setting to calico.yaml, with "assign_ipv6": "true". Doing that and then restarting the cluster (not sure if I can just reapply).

 Have:
 	kubeadm join --token=a68a48.184e49eb7ec5588e 10.87.49.77

 Note: Issue 455 is closed, they pushed a tag v0.23.1 out  for calico/ctl.

 Looks like DNS pod is not coming up, and is showing this error:
  14m	1s	372	{kubelet devstack-77}		Warning	FailedSync	Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-zxr50_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-zxr50_kube-system(6fb8a3ae-df38-11e6-ae00-003a7d69f73c)\" using network plugins \"cni\": No configured Calico ools; Skipping pod"

 01/23/2017
 Issue is that IPv6 I not set up on the nodes. Added this to /etc/sysconfig/network:
 NETWORKING_IPv6=yes

 For the bridge interface (using br_mgmt), added:
 IPV6INIT=yes
 IPV6ADDR=2001:1::71/64

 Used 71,77,78 for the three systems.

 Next, brought up cloud and started Ubuntu (w/o IPv6) in two containers and checked ping.
 	kubeadm join --token=228e0d.1b18edb2e1fe7159 10.87.49.77

 Did web.yml and redis-master-deployment.yml and then connected to latter and pinged the former with IPv4 address. Now, enabling IPv6 and trying again.


 Note: net-tools for route command, tcpdump installed too. Have "ip -6 addr add <address> dev <dev>" to add an IP address. Have "ip -6 route" or "route -A inet6" with package installed.
 	kubeadm join --token=3b13ce.433093a70c2e47a0 10.87.49.77

 The DNS pod is not running and we see the error:
  5m	3s	28	{kubelet devstack-77}		Warning	FailedSync	Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-1m2c5_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-1m2c5_kube-system(e76de7f4-e192-11e6-86d8-003a7d69f73c)\" using network plugins \"cni\": No configured Calico pools; Skipping pod"

 Q: Do I need to use the br_mgmt IP addresses?
 Q: Any other config?

 Trying the following (can try IPv6 later and maybe other settings, like DNS?):
 kubeadm init --api-advertise-addresses=172.18.96.26

 Control plane never became ready. Instead, trying this:
 kubeadm init --pod-network-cidr=2001:1::/64

 Seems stuck creating test deployment. Trying adding "subnet" with value "2001:2::/64" to calico.yaml IPAM area and then creating cluster. Still seeing DNS not coming up.

 Trying Again 1/26/2017
 Seeing notes on reference page will try these steps. First, set global IP on each node, with 2001:1::#/64, where # is 71,77,78. Did this on br_mgmt interface (can try on br_api, if this doesn't work). Verified pings.

 Bringing up cluster with calico yaml file, and only the assign_ipv6 flag set (no subnet).

 kubeadm init

 Have:
 	kubeadm join --token=0f1c24.963cd0640a394e46 10.87.49.77
 	
 kubectl taint nodes --all dedicated-
 kubectl apply -f calico.yaml
 kubectl get pods --all-namespaces

 The DNS pod is showing this:
  2m	2s	11	{kubelet devstack-77}		Warning	FailedSync	Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-nqm5n_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-nqm5n_kube-system(cff5cc0e-e3fd-11e6-a9ac-003a7d69f73c)\" using network plugins \"cni\": No configured Calico pools; Skipping pod"

 Getting calicoctl using these instructions: https://github.com/projectcalico/calicoctl

 git clone https://github.com/projectcalico/calicoctl.git $GOPATH/src/github.com/projectcalico/calicoctl

 Install glide:
 mkdir $GOPATH/bin
 curl https://glide.sh/get | sh

 cd /root/go/src/github.com/projectcalico/calicoctl
 glide install -strip-vendor
 make binary
 cd $GOPATH
 go build src/github.com/projectcalico/calicoctl/calicoctl/calicoctl.go
 mv calicoctl bin/

 cd
 ~/go/bin/calicoctl node --ip=10.87.49.77 --ipv6=2001:1::77 --libnetwork
 Set the Calico datastore access information in the environment variables or
 or supply details in a config file.

 Already have Calico node container running. How do I do the required changes? They mention about creating pools, but there is not pool command for calicoctl.

 Note: See this page (http://docs.projectcalico.org/v1.6/getting-started/kubernetes/installation/) has different steps to install calicoctl.

 1/27/2017
 Trying setting assign_ipv6 in addition to ipv6 in config file.
 	kubeadm join --token=6d8305.5794a9835accd11a 10.87.49.77

 Info from lrobson and epeterson…
 Need newer calico.yaml file from http://docs.projectcalico.org/v2.0/getting-started/kubernetes/installation/hosted/kubeadm/.

 They use BIRD, a software routing daemon that does BGP peering for nodes. http://bird.network.cz/

 BGP issues: BGP won't accept IPv6 router IDs. Need IPv4 config, even if doing IPv6 BGP.
 BIRD caveats: IPv6 ECMP doesn't work right in BIRD. IPv6 loopbacks also don't work correctly.

 BIRD6 templates in https://github.com/projectcalico/calicoctl/tree/master/calico_node/filesystem/etc/calico/confd/templates

 epeterson [11:48 AM] 
 I think that configuring v6 would just work as long as you got the prefixes the node was supposed to have into calico’s etcd.  Not 100% sure of the exact logic the template mechanism uses to generate templates but unless there’s something that’s parsing specific strings, it would work

 The tl;dr on v6netes is that the large majority of k8s users are doing things on cloud platforms.  AWS didn’t support v6 until very recently, azure and GCE still don’t.  I’m looking for a PR where a bunch of the issues are being discussed.

 [12:00]  
 One fundamental problem that is being worked for next release is that k8s doesn’t support multiple networks or address families.

 [12:00]  
 Well, not next release, 1.6 is a stability release.  But 1.7 multiple networks is on the agenda.

 Lrobson says need calico-ipam set to assign IPv6 addresses (http://docs.projectcalico.org/v2.0/reference/cni-plugin/configuration):
 cni_network_config: |-
    {
        "name": "k8s-pod-network",
        "type": "calico",
        "etcd_endpoints": "__ETCD_ENDPOINTS__",
        "log_level": "info",
        "ipam": {
            "type": "calico-ipam",
            "assign_ipv6": "true"
        },
        "policy": {
            "type": "k8s",
             "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
             "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
        },
        "kubernetes": {
            "kubeconfig": "/etc/cni/net.d/__KUBECONFIG_FILENAME__"

 And… add pool here:

 # The default IP Pool to be created for the cluster.
  # Pod IP addresses will be assigned from this pool.
  ippool.yaml: |
      apiVersion: v1
      kind: ipPool
      metadata:
        cidr: 2001:1::/64
      spec:
        ipip:
          enabled: true
        nat-outgoing: true

 Trying yaml file w/o changes. All pods came up. Trying with changes above:
 	kubeadm join --token=849d9a.7a5aeb672e07a210 10.87.49.77
 	
 The DNS pod is not coming up. It fails with a message saying: 
 	Failed to get IPv6 addresses for container veth

 It keeps restarting. Went into calico-node container (Using docker exec-it <container-name> sh), and then looked at:
 	cat /etc/calico/confd/config/bird6_ipam.cfg
 	cat /etc/calico/confd/config/bird6_aggr.cfg
 	
 There is a bird.cfg and bird6.cfg in /etc/calico/confd/config/ area. On this setup. There is some interesting things. It shows the router ID as 34.34.34.6, which is on the t interface and not br_api. I don't have global IPv6 on that I/F. May have just used that IP.

 So, DNS container is not getting an IPv6 address. See:
 	[root@devstack-77 ~]# ~/go/bin/calicoctl node status
 	Calico process is running.
 	
 	IPv4 BGP status
 	No IPv4 peers found.
 	
 	IPv6 BGP status
 	No IPv6 peers found.

 Eric Peterson, gave me a bird.cfg file to use. Need to create container/VM with BIRD so that have something for cluster to BGP peer with, so that calico nodes receive pod routes from other nodes.

 Q: Do I create a VM or container with BIRD running to provide the BGP peer info?
 Q: Could we use nexus9k as BGP peer?
 Q: Who's versed in BGP?

 1/30/2017 BIRD
 Created a Dockerfile in ~/bird. Downloaded the Fedora binary for bird (http://bird.network.cz/?download&tdir=redhat/) to Mac and copied to ds-77. Added to Docker file. Issue, install failed as needs ncurses-lib-6.0 and Centos has 5.0 as latest.

 Could not add bird repo to yum.repos.d/ as it uses FTP and cannot seem to access the site from within the lab.

 Trying Ubuntu (http://bird.network.cz/?download&tdir=debian/), and have this working, using this Dockerfile:

 	COPY ./apt.key /
 	RUN apt-key add /apt.key
 	RUN apt-get update -y && apt-get upgrade -y
 	RUN apt-get install -y bird
 	CMD /bin/bash

 Did "docker build ." and then "docker run -it <IMAGE#> bash". Need to label the image, I guess.

 Need to configure Bird. Eric says, because inside of a container, may need an external static NAT for docker0 IP to the internal Bird IP. His example had static route, because Bird host was on a secondary subnet and needed a route to primary VLAN IP to peer with it (BGP needs next hop reachability -doesn't use route table to establish peering).

 Q: Should I use VM instead? Could I use Bird running on host?
 Q: With Bird in container, it is using docker0 I/F. Does it need IPv6 enabled? Does docker need IPv6 enabled? How?

 In case need VM, can use virsh:
 yum install qemu-kvm qemu-img
 yum install virt-manager libvirt libvirt-python libvirt-client
 yum groupinstall virtualization-client virtualization-platform virtualization-tools

 restart libvirtd
 systemctl status libvirtd 

 Or can use Virtualbox.  Info? Decide on method, if do this.

 Issue with auto detect IP for calico with multiple nodes (https://github.com/projectcalico/calicoctl/issues/873).

 Ref: http://docs.projectcalico.org/v2.0/reference/private-cloud/l3-interconnect-fabric

 May be able to remove other networks from host, and calico may autodetect, on the remaining network. Instead of picking a network, and getting 34.34.34.6, as it did in my case with the t interface.

 ozdanborne [2:45 PM] 
 @pcm I think a few options are:
 - setup your interfaces so calico autodetects the right one
 - build a custom calicoctl.go / calico/node which autodetects the right one
 - update the `BGPPeer`s after they come up.

 a less "quick" solution would be for us to start putting together that `--can-reach` support (edited)

 Going to try the first option tomorrow… setting up interfaces so that auto-detect works.

 1/31/2017
 Took IP/Mask off of t and br_mgmt interfaces(how to shutdown too?), and added IPv6 address 2001:2::77/64 to brapi interface.Did the same on other nodes. Starting up kubeadm to see what happens.
 	kubeadm join --token=10f438.23104a16fdf4dd69 10.87.49.77

 Did taint and applied calico.yaml (after changed pool to 2001:2::/64. The calico-node container still has 34.34.34.6 IP and 2001:1::/64 for filter. Will tear down and then remove container images from docker and then retry. Had to delete stopped containers to get rid of the image.
 	kubeadm join --token=cdee1f.03c5873344ab69c1 10.87.49.77
 	
 Same issue with IP used.  Copied some config files in ~ to ~/save, in case they are being used. Removed containers:

 docker rm `docker ps --format "table {{.ID}}" -a | tail -n +2`

 Removed images and retrying kubeadm init.
 	kubeadm join --token=125b8a.5858c1c63bd9f5a4 10.87.49.77
 	
 It still is using 34.34.34.6. Trying Ipv6 for calico.yaml and rechecking.
 	kubeadm join --token=ff89b8.fa557582bb0ff77e 10.87.49.77
 	
 Looks like maybe info is in etcd (my theory). Deleting all containers, deleting all images, and deleting files in /var/lib/etcd/. Rebooted system and restarting up kubeadm. Instead of rebooting, may be able to do "ip route flush proto bird". Instead of deleting dir, may be able to do "curl -sL http://<IP_OF_ETCD>:2379/v2/keys/calico?recursive=true -XDELETE". Try some time.

 Still see 34.34.34.6 for ID, and 2001:1:: in bird6.cfg files.

 Lrobson/caseydavenport indicate that IP can be set manually. Can either delete DaemonSet that starts calico-node, and manually run calico-node with IP (http://docs.projectcalico.org/v2.0/reference/calicoctl/commands/node/run), or use calicoctl to edit the IP on the node (http://docs.projectcalico.org/v2.0/reference/calicoctl/resources/node). Suggested that I do:

 calicoctl get nodes -o yaml > nodes.txt
 calicoctl apply -f nodes.txt

 Tried. No node info listed. Need to set ETCD_AUTHORITY env variable to 10.87.49.77:6666 and then run it. Or can set this in /etc/calico/calicoctl.cfg:

 apiVersion: v1
 kind: calicoApiConfig
 metadata:
 spec:
  datastoreType: "etcdv2"
  etcdEndpoints: "http://10.87.49.77:6666"

 Ref: http://docs.projectcalico.org/v2.0/reference/calicoctl/setup/etcdv2

 Applied change, but container still shows 34.34.34.6. Should be in datastore, so will restart the calico-node. Did docker restart and the node has the correct IP now.

 Cleaning containers, reset kubeadm, clear etcd, delete .kube dir, flushed route on all nodes, and will restart with IPv6 configuration. Doing init:
 	kubeadm join --token=b29e05.9d010309f5113284 10.87.49.77

 Copied calico.yaml.v6 config file to calico.yaml and doing rest:
 kubectl taint nodes --all dedicated-
 kubectl apply -f calico.yaml
 kubectl get pods --all-namespaces

 Started up. See IPPools 192.168.0.0/16, 2001:1::/64, and 2001:2::/64. In calico-node, the router IP is 10.87.49.77 (correct). In bird6_aggr.cfg, it has blackhole, accept, and reject for 2001:1:…. In bird6_ipam.cfg, it has entries for both 2001:1::/64 and 2001:2::/64.

 Did "calicoctl delete ippool 2001:1::/64" and now that is gone. The bird6_aggr.cfg has the 2001:2::… addresses now, and  bird6_ipam.cfg now has the 2001:2::/64 network. So, it seems OK now.

 Q: Is it OK on the other nodes?
 Q: Why is DNS pod not able to get IPv6 address?
	Provisioning to use Calico
	1/20/2017
	First, tearing down the cluster, by doing this on each node:
	kubeadm reset
	rm -rf .kube
	systemctl start kubelet.service

	Did yum -y update on all nodes as well. Downloaded http://docs.projectcalico.org/v1.6/getting-started/kubernetes/installation/hosted/kubeadm/calico.yaml and then ran "kubectl create -f calico.yaml".

	Starting up cluster (with just the master node).

	kubeadm init

	Have:
	kubeadm join --token=482742.64ff98647d01bf2c 10.87.49.77

	kubectl taint nodes --all dedicated-
	kubectl apply -f calico.yaml
	kubectl get pods --all-namespaces

	Issue: It is unable to download calico/ctl v0.23.1. It does not exist. Options are to use v1.0.0 (and use same for calico/node, to keep them similar?) or drop to v0.23.0 (and either keep calico/node at v0.23.1 or set it to v.23.0 too). Trying v1.0.0. Stopped cluster and then restarting, after updating calico.yaml.

	Have:
	kubeadm join --token=086198.58f8a53b0c2391be 10.87.49.77

	Looks like calico/configure is not starting up (message about calicoctl unknown command pool). Also dns pod not coming up, says CNI config is uninitialized. Will try v0.23.0 for both node and ctl…

	Have:
	kubeadm join --token=b3be10.7356d594b9de79a2 10.87.49.77

	Looks like everything I running…
	[root@devstack-77 ~]# kubectl get pods --all-namespaces
	NAMESPACE NAME READY STATUS RESTARTS AGE
	kube-system calico-etcd-4zkxh 1/1 Running 0 10m
	kube-system calico-node-c8pkm 2/2 Running 0 10m
	kube-system calico-policy-controller-2swdt 1/1 Running 0 10m
	kube-system dummy-2088944543-c5xlk 1/1 Running 0 12m
	kube-system etcd-devstack-77 1/1 Running 0 11m
	kube-system kube-apiserver-devstack-77 1/1 Running 1 11m
	kube-system kube-controller-manager-devstack-77 1/1 Running 0 11m
	kube-system kube-discovery-1769846148-8fp6j 1/1 Running 0 12m
	kube-system kube-dns-2924299975-r04zb 4/4 Running 0 12m
	kube-system kube-proxy-3lmlh 1/1 Running 0 12m
	kube-system kube-scheduler-devstack-77 1/1 Running 0 11m

	No IPv6. Need to add IPAM setting to calico.yaml, with "assign_ipv6": "true". Doing that and then restarting the cluster (not sure if I can just reapply).

	Have:
	kubeadm join --token=a68a48.184e49eb7ec5588e 10.87.49.77

	Note: Issue 455 is closed, they pushed a tag v0.23.1 out for calico/ctl.

	Looks like DNS pod is not coming up, and is showing this error:
	14m 1s 372 {kubelet devstack-77} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-zxr50_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-zxr50_kube-system(6fb8a3ae-df38-11e6-ae00-003a7d69f73c)\" using network plugins \"cni\": No configured Calico ools; Skipping pod"

	01/23/2017
	Issue is that IPv6 I not set up on the nodes. Added this to /etc/sysconfig/network:
	NETWORKING_IPv6=yes

	For the bridge interface (using br_mgmt), added:
	IPV6INIT=yes
	IPV6ADDR=2001:1::71/64

	Used 71,77,78 for the three systems.

	Next, brought up cloud and started Ubuntu (w/o IPv6) in two containers and checked ping.
	kubeadm join --token=228e0d.1b18edb2e1fe7159 10.87.49.77

	Did web.yml and redis-master-deployment.yml and then connected to latter and pinged the former with IPv4 address. Now, enabling IPv6 and trying again.


	Note: net-tools for route command, tcpdump installed too. Have "ip -6 addr add <address> dev <dev>" to add an IP address. Have "ip -6 route" or "route -A inet6" with package installed.
	kubeadm join --token=3b13ce.433093a70c2e47a0 10.87.49.77

	The DNS pod is not running and we see the error:
	5m 3s 28 {kubelet devstack-77} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-1m2c5_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-1m2c5_kube-system(e76de7f4-e192-11e6-86d8-003a7d69f73c)\" using network plugins \"cni\": No configured Calico pools; Skipping pod"

	Q: Do I need to use the br_mgmt IP addresses?
	Q: Any other config?

	Trying the following (can try IPv6 later and maybe other settings, like DNS?):
	kubeadm init --api-advertise-addresses=172.18.96.26

	Control plane never became ready. Instead, trying this:
	kubeadm init --pod-network-cidr=2001:1::/64

	Seems stuck creating test deployment. Trying adding "subnet" with value "2001:2::/64" to calico.yaml IPAM area and then creating cluster. Still seeing DNS not coming up.

	Trying Again 1/26/2017
	Seeing notes on reference page will try these steps. First, set global IP on each node, with 2001:1::#/64, where # is 71,77,78. Did this on br_mgmt interface (can try on br_api, if this doesn't work). Verified pings.

	Bringing up cluster with calico yaml file, and only the assign_ipv6 flag set (no subnet).

	kubeadm init

	Have:
	kubeadm join --token=0f1c24.963cd0640a394e46 10.87.49.77

	kubectl taint nodes --all dedicated-
	kubectl apply -f calico.yaml
	kubectl get pods --all-namespaces

	The DNS pod is showing this:
	2m 2s 11 {kubelet devstack-77} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2924299975-nqm5n_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2924299975-nqm5n_kube-system(cff5cc0e-e3fd-11e6-a9ac-003a7d69f73c)\" using network plugins \"cni\": No configured Calico pools; Skipping pod"

	Getting calicoctl using these instructions: https://github.com/projectcalico/calicoctl

	git clone https://github.com/projectcalico/calicoctl.git $GOPATH/src/github.com/projectcalico/calicoctl

	Install glide:
	mkdir $GOPATH/bin
	curl https://glide.sh/get \| sh

	cd /root/go/src/github.com/projectcalico/calicoctl
	glide install -strip-vendor
	make binary
	cd $GOPATH
	go build src/github.com/projectcalico/calicoctl/calicoctl/calicoctl.go
	mv calicoctl bin/

	cd
	~/go/bin/calicoctl node --ip=10.87.49.77 --ipv6=2001:1::77 --libnetwork
	Set the Calico datastore access information in the environment variables or
	or supply details in a config file.

	Already have Calico node container running. How do I do the required changes? They mention about creating pools, but there is not pool command for calicoctl.

	Note: See this page (http://docs.projectcalico.org/v1.6/getting-started/kubernetes/installation/) has different steps to install calicoctl.

	1/27/2017
	Trying setting assign_ipv6 in addition to ipv6 in config file.
	kubeadm join --token=6d8305.5794a9835accd11a 10.87.49.77

	Info from lrobson and epeterson…
	Need newer calico.yaml file from http://docs.projectcalico.org/v2.0/getting-started/kubernetes/installation/hosted/kubeadm/.

	They use BIRD, a software routing daemon that does BGP peering for nodes. http://bird.network.cz/

	BGP issues: BGP won't accept IPv6 router IDs. Need IPv4 config, even if doing IPv6 BGP.
	BIRD caveats: IPv6 ECMP doesn't work right in BIRD. IPv6 loopbacks also don't work correctly.

	BIRD6 templates in https://github.com/projectcalico/calicoctl/tree/master/calico_node/filesystem/etc/calico/confd/templates

	epeterson [11:48 AM]
	I think that configuring v6 would just work as long as you got the prefixes the node was supposed to have into calico’s etcd. Not 100% sure of the exact logic the template mechanism uses to generate templates but unless there’s something that’s parsing specific strings, it would work

	The tl;dr on v6netes is that the large majority of k8s users are doing things on cloud platforms. AWS didn’t support v6 until very recently, azure and GCE still don’t. I’m looking for a PR where a bunch of the issues are being discussed.

	[12:00]
	One fundamental problem that is being worked for next release is that k8s doesn’t support multiple networks or address families.

	[12:00]
	Well, not next release, 1.6 is a stability release. But 1.7 multiple networks is on the agenda.

	Lrobson says need calico-ipam set to assign IPv6 addresses (http://docs.projectcalico.org/v2.0/reference/cni-plugin/configuration):
	cni_network_config: \|-
	{
	"name": "k8s-pod-network",
	"type": "calico",
	"etcd_endpoints": "__ETCD_ENDPOINTS__",
	"log_level": "info",
	"ipam": {
	"type": "calico-ipam",
	"assign_ipv6": "true"
	},
	"policy": {
	"type": "k8s",
	"k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
	"k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
	},
	"kubernetes": {
	"kubeconfig": "/etc/cni/net.d/__KUBECONFIG_FILENAME__"

	And… add pool here:

	# The default IP Pool to be created for the cluster.
	# Pod IP addresses will be assigned from this pool.
	ippool.yaml: \|
	apiVersion: v1
	kind: ipPool
	metadata:
	cidr: 2001:1::/64
	spec:
	ipip:
	enabled: true
	nat-outgoing: true

	Trying yaml file w/o changes. All pods came up. Trying with changes above:
	kubeadm join --token=849d9a.7a5aeb672e07a210 10.87.49.77

	The DNS pod is not coming up. It fails with a message saying:
	Failed to get IPv6 addresses for container veth

	It keeps restarting. Went into calico-node container (Using docker exec-it <container-name> sh), and then looked at:
	cat /etc/calico/confd/config/bird6_ipam.cfg
	cat /etc/calico/confd/config/bird6_aggr.cfg

	There is a bird.cfg and bird6.cfg in /etc/calico/confd/config/ area. On this setup. There is some interesting things. It shows the router ID as 34.34.34.6, which is on the t interface and not br_api. I don't have global IPv6 on that I/F. May have just used that IP.

	So, DNS container is not getting an IPv6 address. See:
	[root@devstack-77 ~]# ~/go/bin/calicoctl node status
	Calico process is running.

	IPv4 BGP status
	No IPv4 peers found.

	IPv6 BGP status
	No IPv6 peers found.

	Eric Peterson, gave me a bird.cfg file to use. Need to create container/VM with BIRD so that have something for cluster to BGP peer with, so that calico nodes receive pod routes from other nodes.

	Q: Do I create a VM or container with BIRD running to provide the BGP peer info?
	Q: Could we use nexus9k as BGP peer?
	Q: Who's versed in BGP?

	1/30/2017 BIRD
	Created a Dockerfile in ~/bird. Downloaded the Fedora binary for bird (http://bird.network.cz/?download&tdir=redhat/) to Mac and copied to ds-77. Added to Docker file. Issue, install failed as needs ncurses-lib-6.0 and Centos has 5.0 as latest.

	Could not add bird repo to yum.repos.d/ as it uses FTP and cannot seem to access the site from within the lab.

	Trying Ubuntu (http://bird.network.cz/?download&tdir=debian/), and have this working, using this Dockerfile:

	COPY ./apt.key /
	RUN apt-key add /apt.key
	RUN apt-get update -y && apt-get upgrade -y
	RUN apt-get install -y bird
	CMD /bin/bash

	Did "docker build ." and then "docker run -it <IMAGE#> bash". Need to label the image, I guess.

	Need to configure Bird. Eric says, because inside of a container, may need an external static NAT for docker0 IP to the internal Bird IP. His example had static route, because Bird host was on a secondary subnet and needed a route to primary VLAN IP to peer with it (BGP needs next hop reachability -doesn't use route table to establish peering).

	Q: Should I use VM instead? Could I use Bird running on host?
	Q: With Bird in container, it is using docker0 I/F. Does it need IPv6 enabled? Does docker need IPv6 enabled? How?

	In case need VM, can use virsh:
	yum install qemu-kvm qemu-img
	yum install virt-manager libvirt libvirt-python libvirt-client
	yum groupinstall virtualization-client virtualization-platform virtualization-tools

	restart libvirtd
	systemctl status libvirtd

	Or can use Virtualbox. Info? Decide on method, if do this.

	Issue with auto detect IP for calico with multiple nodes (https://github.com/projectcalico/calicoctl/issues/873).

	Ref: http://docs.projectcalico.org/v2.0/reference/private-cloud/l3-interconnect-fabric

	May be able to remove other networks from host, and calico may autodetect, on the remaining network. Instead of picking a network, and getting 34.34.34.6, as it did in my case with the t interface.

	ozdanborne [2:45 PM]
	@pcm I think a few options are:
	- setup your interfaces so calico autodetects the right one
	- build a custom calicoctl.go / calico/node which autodetects the right one
	- update the `BGPPeer`s after they come up.

	a less "quick" solution would be for us to start putting together that `--can-reach` support (edited)

	Going to try the first option tomorrow… setting up interfaces so that auto-detect works.

	1/31/2017
	Took IP/Mask off of t and br_mgmt interfaces(how to shutdown too?), and added IPv6 address 2001:2::77/64 to brapi interface.Did the same on other nodes. Starting up kubeadm to see what happens.
	kubeadm join --token=10f438.23104a16fdf4dd69 10.87.49.77

	Did taint and applied calico.yaml (after changed pool to 2001:2::/64. The calico-node container still has 34.34.34.6 IP and 2001:1::/64 for filter. Will tear down and then remove container images from docker and then retry. Had to delete stopped containers to get rid of the image.
	kubeadm join --token=cdee1f.03c5873344ab69c1 10.87.49.77

	Same issue with IP used. Copied some config files in ~ to ~/save, in case they are being used. Removed containers:

	docker rm `docker ps --format "table {{.ID}}" -a \| tail -n +2`

	Removed images and retrying kubeadm init.
	kubeadm join --token=125b8a.5858c1c63bd9f5a4 10.87.49.77

	It still is using 34.34.34.6. Trying Ipv6 for calico.yaml and rechecking.
	kubeadm join --token=ff89b8.fa557582bb0ff77e 10.87.49.77

	Looks like maybe info is in etcd (my theory). Deleting all containers, deleting all images, and deleting files in /var/lib/etcd/. Rebooted system and restarting up kubeadm. Instead of rebooting, may be able to do "ip route flush proto bird". Instead of deleting dir, may be able to do "curl -sL http://<IP_OF_ETCD>:2379/v2/keys/calico?recursive=true -XDELETE". Try some time.

	Still see 34.34.34.6 for ID, and 2001:1:: in bird6.cfg files.

	Lrobson/caseydavenport indicate that IP can be set manually. Can either delete DaemonSet that starts calico-node, and manually run calico-node with IP (http://docs.projectcalico.org/v2.0/reference/calicoctl/commands/node/run), or use calicoctl to edit the IP on the node (http://docs.projectcalico.org/v2.0/reference/calicoctl/resources/node). Suggested that I do:

	calicoctl get nodes -o yaml > nodes.txt
	calicoctl apply -f nodes.txt

	Tried. No node info listed. Need to set ETCD_AUTHORITY env variable to 10.87.49.77:6666 and then run it. Or can set this in /etc/calico/calicoctl.cfg:

	apiVersion: v1
	kind: calicoApiConfig
	metadata:
	spec:
	datastoreType: "etcdv2"
	etcdEndpoints: "http://10.87.49.77:6666"

	Ref: http://docs.projectcalico.org/v2.0/reference/calicoctl/setup/etcdv2

	Applied change, but container still shows 34.34.34.6. Should be in datastore, so will restart the calico-node. Did docker restart and the node has the correct IP now.

	Cleaning containers, reset kubeadm, clear etcd, delete .kube dir, flushed route on all nodes, and will restart with IPv6 configuration. Doing init:
	kubeadm join --token=b29e05.9d010309f5113284 10.87.49.77

	Copied calico.yaml.v6 config file to calico.yaml and doing rest:
	kubectl taint nodes --all dedicated-
	kubectl apply -f calico.yaml
	kubectl get pods --all-namespaces

	Started up. See IPPools 192.168.0.0/16, 2001:1::/64, and 2001:2::/64. In calico-node, the router IP is 10.87.49.77 (correct). In bird6_aggr.cfg, it has blackhole, accept, and reject for 2001:1:…. In bird6_ipam.cfg, it has entries for both 2001:1::/64 and 2001:2::/64.

	Did "calicoctl delete ippool 2001:1::/64" and now that is gone. The bird6_aggr.cfg has the 2001:2::… addresses now, and bird6_ipam.cfg now has the 2001:2::/64 network. So, it seems OK now.

	Q: Is it OK on the other nodes?
	Q: Why is DNS pod not able to get IPv6 address?