Magnum Testing

Magnum Testing

0. Create key
openstack --insecure keypair create feilong --public-key ~/.ssh/id_rsa.pub

1. Upload image
scp ~/Downloads/Fedora-Atomic-26-20170723.0.x86_64.qcow2 [email protected]:~/

2. Create image (NOTE: Be careful the disk-format)
openstack --insecure image create fedora-atomic --disk-format qcow2 --container-format bare --property os_distro=fedora-atomic --file ./Fedora-Atomic-26-20170723.0.x86_64.qcow2

3. Create cluster template

openstack coe cluster template create k8s-fc27-v1.12.7 --keypair feilong --flavor ds1G --master-flavor ds2G --coe kubernetes --external-network public --network-driver calico --docker-storage-driver=overlay2 --image=<> --labels=etcd_volume_size=5,kube_tag=v1.12.7

openstack coe cluster template create k8s-fc27-v1.13.10 --keypair feilong --flavor c1.c2r4 --master-flavor c1.c2r4 --coe kubernetes --external-network public --network-driver calico --docker-storage-driver=overlay2 --image=<> --labels=container_infra_prefix=docker.io/catalystcloud/,cloud_provider_enabled=true,cloud_provider_tag=1.14.0-catalyst,kube_tag=v1.13.10,ingress_controller=octavia,octavia_ingress_controller_tag=1.14.0-catalyst,heat_container_agent_tag=stein-stable,etcd_volume_size=20,prometheus_monitoring=true,keystone_auth_enabled=True,k8s_keystone_auth_tag=v1.15.0,auto_scaling_enabled=False,auto_healing_enabled=True,auto_healing_controller=magnum-auto-healer,magnum_auto_healer_tag=v1.15.0 --public

4. Set password for fedora image
/opt/cat/openstack/magnum/lib/python2.7/site-packages/magnum/drivers/common/templates/kubernetes/fragments

#cloud-config
password: atomic
ssh_pwauth: True
chpasswd: { expire: False }

4. Create cluster

openstack --insecure coe cluster create --name k8scluster --cluster-template k8s --node-count 1 --timeout 120

(22:27:29) strigazi: openstack stack update <stack_id> --existing -P minions_to_remove=<comma separated list with resources ids or private ips> -P number_of_minions=<integer>
(22:28:31) strigazi: the resource id can be found either from the name of the vms or by doing openstack stack resource list -n 2 <stack_id>

5. VNC

run command "ssh -N -L 5900:10.0.0.81:5901 [email protected]" on host. Then run "gvncviewer 127.0.0.1" to start VNC


6. All cluster information are in  /etc/sysconfig/heat-params, it can be sourced, then run scripts under /var/lib/cloud/instances/<instance_id>/scripts to debug

7. Run 'journalctl -u heat-container-agent --no-pager' to check the log about software deployment of heat agent

8. Busybox

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

9. Error found in devstack

{u'message': u'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 5bcaf6ce-3dde-43d2-a1ae-c9a217d7f658.', u'code': 500, u'details': u'  File "/opt/stack/nova/nova/conductor/manager.py", line 578, in build_instances\n    raise exception.MaxRetriesExceeded(reason=msg)\n', u'created': u'2018-02-22T00:52:15Z'}

It's because nova can't get ip assigned from Neutron, the workaround is restart all Neutron service by 'systemctl restart devstack@q-*'

10. dashboard logs 

[fedora@k8scluster-sc7jximdohmh-master-0 kubernetes]$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE       IP                NODE
default       alpine-1-75b69bb5cc-8k9qs                  1/1       Running   0          34m       192.168.160.139   k8scluster-sc7jximdohmh-minion-0
default       alpine-2-8699dd9657-ffc7x                  1/1       Running   0          34m       192.168.160.138   k8scluster-sc7jximdohmh-minion-0
default       alpine-3-7d56d867df-vcj2v                  1/1       Running   0          35m       192.168.160.137   k8scluster-sc7jximdohmh-minion-0
default       alpine-4-5f97677c-4nh4q                    1/1       Running   0          35m       192.168.160.136   k8scluster-sc7jximdohmh-minion-0
default       alpine-56cb6f6969-2bssh                    1/1       Running   0          8h        192.168.160.131   k8scluster-sc7jximdohmh-minion-0
kube-system   calico-kube-controllers-7c5c6f69c7-hx77z   1/1       Running   0          10h       10.0.0.4          k8scluster-sc7jximdohmh-minion-0
kube-system   calico-node-dktcd                          2/2       Running   0          33m       10.0.0.11         k8scluster-sc7jximdohmh-master-0
kube-system   calico-node-n85r6                          2/2       Running   0          10h       10.0.0.4          k8scluster-sc7jximdohmh-minion-0
kube-system   coredns-5864cfd79d-f2wjr                   1/1       Running   37         10h       192.168.160.130   k8scluster-sc7jximdohmh-minion-0
kube-system   heapster-68b976dd7-kfslh                   1/1       Running   0          10h       192.168.160.129   k8scluster-sc7jximdohmh-minion-0
kube-system   kubernetes-dashboard-846b8b6844-mchtk      1/1       Running   54         10h       192.168.160.128   k8scluster-sc7jximdohmh-minion-0
[fedora@k8scluster-sc7jximdohmh-master-0 kubernetes]$ kubectl logs kubernetes-dashboard-846b8b6844-mchtk -n kube-system
2018/03/05 11:19:48 Starting overwatch
2018/03/05 11:19:48 Using in-cluster config to connect to apiserver
2018/03/05 11:19:48 Using service account token for csrf signing
2018/03/05 11:19:48 No request provided. Skipping authorization
2018/03/05 11:19:49 Successful initial request to the apiserver, version: v1.9.3
2018/03/05 11:19:49 Generating JWE encryption key
2018/03/05 11:19:49 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2018/03/05 11:19:49 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2018/03/05 11:19:49 Initializing JWE encryption key from synchronized object
2018/03/05 11:19:49 Creating remote Heapster client for heapster:80
2018/03/05 11:19:49 Auto-generating certificates
2018/03/05 11:19:49 Successfully created certificates
2018/03/05 11:19:49 Serving securely on HTTPS port: 8443
2018/03/05 11:19:49 Successful request to heapster
[fedora@k8scluster-sc7jximdohmh-master-0 kubernetes]$ 

11. dns logs

[fedora@k8scluster-sc7jximdohmh-master-0 kubernetes]$ kubectl logs coredns-5864cfd79d-f2wjr -n kube-system
.:53
2018/03/05 11:04:26 [INFO] CoreDNS-1.0.1
2018/03/05 11:04:26 [INFO] linux/amd64, go1.9.2, 99e163c3
CoreDNS-1.0.1
linux/amd64, go1.9.2, 99e163c3
[fedora@k8scluster-sc7jximdohmh-master-0 kubernetes]$ 

.