Calico requires the node name to be lowercase. Maybe that's obvious knolwedge but all other components have no problems handling uppercase characters. Error:
[root@k8s-kube-w-8djLQqtkAHJZvaOa centos]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
327fa0492135 quay.io/calico/node "start_runit" 3 seconds ago Exited (1) 2 seconds ago k8s_calico-node_calico-node-k257k_kube-system_24149352-2567-11e8-94a0-02c64064058a_3
994fe7650e97 quay.io/calico/cni "/install-cni.sh" 41 seconds ago Up 40 seconds k8s_install-cni_calico-node-k257k_kube-system_24149352-2567-11e8-94a0-02c64064058a_0
94122e279a5a gcr.io/google_containers/pause-amd64:3.0 "/pause" About a minute ago Up About a minute k8s_POD_calico-node-k257k_kube-system_24149352-2567-11e8-94a0-02c64064058a_0
[root@k8s-kube-w-8djLQqtkAHJZvaOa centos]# docker logs 327fa0492135
2018-03-11 20:03:41.546 [INFO][7] startup.go 248: Early log level set to info
2018-03-11 20:03:41.546 [INFO][7] startup.go 259: NODENAME environment not specified - check HOSTNAME
2018-03-11 20:03:41.555 [INFO][7] startup.go 101: Skipping datastore connection test
2018-03-11 20:03:41.559 [INFO][7] startup.go 332: Building new node resource Name="k8s-kube-w-8djLQqtkAHJZvaOa"
2018-03-11 20:03:41.559 [INFO][7] startup.go 347: Initialize BGP data
2018-03-11 20:03:41.559 [INFO][7] startup.go 544: Using autodetected IPv4 address on interface eth0: 10.0.1.114/24
2018-03-11 20:03:41.559 [INFO][7] startup.go 412: Node IPv4 changed, will check for conflicts
2018-03-11 20:03:41.562 [INFO][7] startup.go 607: No AS number configured on node resource, using global value
2018-03-11 20:03:41.562 [ERROR][7] startup.go 159: Unable to set node resource configuration error=error with field Metadata.Name = 'k8s-kube-w-8djLQqtkAHJZvaOa' (name must consist of lower case alphanumeric characters, '-' or '.' (regex: [a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*))
2018-03-11 20:03:41.562 [WARNING][7] startup.go 1007: Terminating
The best is to make sure hostnames and dns names use no uppercase characters at all.
Seen such error:
Mar 11 23:05:35 k8s-kube-w-id-1520805773-scs5b6ri5sgld5hl kubelet[24342]: E0311 23:05:35.056619
24342 kubelet_node_status.go:106] Unable to register node "ip-10-0-1-5.eu-central-1.compute.internal"
with API server: Node "ip-10-0-1-5.eu-central-1.compute.internal" is invalid: metadata.labels: Invalid
value: "k8s-kube-w-id-1520805773-scs5b6ri5sgld5hl.internal.doc2track.com": must be no more than 63 characters
The fqdn must be 63 characters long.
A change to --cloud-provider=aws raises the bar to another level. Suddenly, the workers want to use aws.private_dns_name for authentication. So if you previously maintined a CA issuing certs for your own DNS names, forget that, change to use aws.private_dns_name. But only for authentication. Everywhere else the node still communicates your hostnames. So do not forget to set that --hostname-override. The API server talks to you over your own hostnames.
For example:
ip-10-0-1-155.eu-central-1.compute.internal
k8s-kube-w-...internal.doc2track.com
the certificate has to be like:
$ openssl x509 -in /var/folders/1v/dzstxkjx10q76ky0c5qxp9br0000gn/T/tmp.qH64yGNI/.certs/ip-10-0-1-114.eu-central-1.compute.internal.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
62:7e:36:4a:8d:10:b1:c3:c2:6c:b6:a9:15:1a:cf:62:65:1d:05:4d
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=DE, L=Aachen, O=k8s, OU=ops, CN=k8s ca intermediate
Validity
Not Before: Mar 11 19:55:00 2018 GMT
Not After : Mar 3 19:55:00 2048 GMT
Subject: C=DE, L=Aachen, O=system:nodes, OU=ops, CN=system:node:ip-10-0-1-114.eu-central-1.compute.internal
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (2048 bit)
Modulus (2048 bit):
00:b9:c3:08:ac:67:69:08:b9:82:5f:b6:5e:34:7a:
82:ba:d3:f5:c3:8c:e0:84:64:6f:08:a7:0f:33:f4:
d4:c7:43:71:92:4d:0b:f4:cc:47:d3:cb:f5:5c:91:
25:02:11:17:72:ba:c0:1a:92:ed:85:e3:7b:a9:a9:
5f:47:b6:4d:52:12:bd:8b:18:7f:d9:db:df:cc:2e:
50:3b:1f:06:3b:90:5e:60:97:20:b7:f2:c3:23:d1:
35:03:34:8a:ae:6b:18:ff:1a:ad:7e:f8:c0:79:51:
0e:22:55:73:87:8e:72:b6:73:73:c4:35:f2:d5:20:
5a:89:6f:aa:12:20:77:97:99:55:49:d0:c8:8f:83:
02:be:f1:53:4f:95:43:55:77:00:63:6f:e4:55:68:
8f:ec:72:28:5f:cb:25:f7:e0:6f:18:e8:27:a8:28:
6e:c3:eb:8c:7d:a7:44:3a:3c:11:3a:2e:4d:48:f4:
5e:1f:99:d9:d2:bd:a8:b6:b5:3f:eb:04:0d:b6:be:
3a:ed:46:e0:03:d7:fd:b2:e3:36:9c:b6:3f:f2:94:
69:6c:bf:ef:b4:02:1a:00:c3:8c:c3:d7:57:0e:c9:
57:97:72:37:26:bd:ac:64:11:2e:c7:63:24:b4:ed:
b0:ca:c2:47:36:4e:32:ab:9e:e4:ba:cf:4a:9b:c2:
2c:a7
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
47:58:36:F6:0B:E9:AC:FF:DE:F0:C9:5C:03:5D:5C:BC:FB:33:76:95
X509v3 Authority Key Identifier:
keyid:49:E5:EC:C7:85:D0:38:B1:FE:FE:0E:11:60:65:B1:F3:8A:DC:CC:6C
X509v3 Subject Alternative Name:
DNS:k8s-kube-w-8djLQqtkAHJZvaOa.internal.doc2track.com, DNS:k8s-kube-w.internal.doc2track.com, DNS:internal.doc2track.com, DNS:compute.internal, IP Address:127.0.0.1
Signature Algorithm: sha256WithRSAEncryption
8e:cb:83:92:dc:8f:5d:39:c8:cc:9e:cb:e9:a4:c3:1e:48:1b:
7b:44:25:31:95:57:1e:6e:3e:2f:f7:13:e1:4e:14:7e:f0:4d:
45:5b:63:bb:eb:82:dc:ed:96:6f:6b:fd:36:c8:ec:42:0e:61:
ac:ee:63:6d:d8:ac:29:52:18:d2:b8:ce:d4:08:79:a5:43:d0:
af:89:19:f1:c5:c7:fb:d2:ba:46:ea:1a:10:e9:79:ea:50:11:
fe:e3:d2:f7:ce:28:3e:c1:50:bc:04:b6:30:76:43:0b:20:da:
1a:92:80:7f:a2:8f:58:c7:d1:da:20:36:9c:6a:ff:e6:53:62:
49:17:4c:53:4e:76:20:ad:0a:dc:c0:e9:c6:d7:f9:2e:13:79:
cb:b6:3a:51:a5:cd:77:60:75:41:ad:c9:59:1d:b9:17:9d:ab:
fd:63:e7:0a:15:80:be:c8:ea:34:b0:9c:e4:54:6d:23:03:3b:
0c:97:85:bc:7d:75:b5:6c:6c:6b:a7:28:a5:cd:da:44:10:d0:
24:21:6e:cd:fa:b4:c0:cb:38:23:4c:8c:71:ae:b5:c3:ab:69:
f6:0f:ad:5f:d4:2e:6c:78:a8:cd:be:b2:4d:65:a5:14:b3:d5:
98:91:f1:bf:12:8b:b7:66:e1:f2:ed:87:fb:d0:c5:d8:14:c9:
6a:c2:6a:0e
Certificate's CN must have the value of aws.private_dns_name. If running without cloud privder, the common name must be the hostname you use.
I was expecting that the worker node will require AWS IAM access but the initial tests indicate that only the controller communicates with the cloud provider. Maybe that's different for other cloud providers.
As such, I did not have to add any IAM permissions to the workers. The controllers seem to be doing fine with:
{
"Sid": "k8sAws",
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"ec2:AttachVolume",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateRoute",
"ec2:CreateSecurityGroup",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:DeleteRoute",
"ec2:DeleteSecurityGroup",
"ec2:DeleteVolume",
"ec2:DescribeInstances",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVolume",
"ec2:DescribeVolumeModifications",
"ec2:DescribeVpcs",
"ec2:DetachVolume",
"ec2:ModifyInstanceAttribute",
"ec2:ModifyVolume",
"ec2:RevokeSecurityGroupIngress",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetHealth",
"kms:DescribeKey"
],
"Resource": [ "*" ]
}
This list is compiled by checking what AWS operations are being executed by the AWS kubernetes plugin. Some non-obvious, implied permissions may be reuired. Right now, it is chucking along with PersistentVolumeClaims. To be evaluated.