Skip to content

Instantly share code, notes, and snippets.

@danehans

danehans/ext-dns-deploy-trouble.md

Last active October 9, 2018 18:23
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save danehans/ca7f8ff8c22f5b37dc13f82fceafe2ae to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save danehans/ca7f8ff8c22f5b37dc13f82fceafe2ae to your computer and use it in GitHub Desktop.
Download ZIP
fedv2_ingress
Raw
ext-dns-deploy-trouble.md

I followed the steps in the nginx-ingress tutorial with one difference. I already had the nginx ingress controller deployed to my 2 k8s clusters using the official guide.

My Ingresses work by setting the host header for curl:

$ curl -I -HHost:nginx.external.daneyon.com http://35.233.196.28:30778
HTTP/1.1 200 OK
Server: nginx/1.15.3
Date: Wed, 03 Oct 2018 18:13:31 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 02 Oct 2018 14:49:27 GMT
ETag: "5bb38577-264"
Accept-Ranges: bytes

$ curl -I -HHost:nginx.external.daneyon.com http://35.233.152.40:30046/
HTTP/1.1 200 OK
Server: nginx/1.15.3
Date: Wed, 03 Oct 2018 18:15:14 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 02 Oct 2018 14:49:27 GMT
ETag: "5bb38577-264"
Accept-Ranges: bytes

But I see no A records for the external.daneyon.com zone:

$ gcloud dns record-sets list --zone="external-daneyon-com" --name "external.daneyon.com" --type A
Listed 0 items.

Here is the ingress spec from one of the k8s clusters:

# kubectl get ingress -n test-namespace -o yaml
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    creationTimestamp: 2018-10-03T18:07:07Z
    generation: 1
    name: test-ingress
    namespace: test-namespace
    resourceVersion: "110804"
    selfLink: /apis/extensions/v1beta1/namespaces/test-namespace/ingresses/test-ingress
    uid: 23b074ef-c737-11e8-8af7-42010a8a0002
  spec:
    rules:
    - host: nginx.external.daneyon.com
      http:
        paths:
        - backend:
            serviceName: test-service
            servicePort: 8088
          path: /
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Here are the zone details:

$ gcloud dns record-sets transaction describe -z "external.daneyon.com"
additions:
- kind: dns#resourceRecordSet
  name: external.daneyon.com.
  rrdatas:
  - ns-cloud-b1.googledomains.com. cloud-dns-hostmaster.google.com. 2 21600 3600 259200
    300
  ttl: 21600
  type: SOA
- kind: dns#resourceRecordSet
  name: external.daneyon.com.
  rrdatas:
  - ns-cloud-b1.googledomains.com.
  - ns-cloud-b2.googledomains.com.
  - ns-cloud-b3.googledomains.com.
  - ns-cloud-b4.googledomains.com.
  ttl: 300
  type: NS
deletions:
- kind: dns#resourceRecordSet
  name: external.daneyon.com.
  rrdatas:
  - ns-cloud-b1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200
    300
  ttl: 21600
  type: SOA

$ echo $POD
pod/external-dns-94cfb98c9-fzrz9

$ kubectl logs $POD 
time="2018-10-03T17:30:27Z" level=info msg="config: {Master: KubeConfig: RequestTimeout:30s IstioIngressGateway:istio-system/istio-ingressgateway Sources:[ingress] Namespace: AnnotationFilter: FQDNTemplate: CombineFQDNAndAnnotation:false Compatibility: PublishInternal:false PublishHostIP:false ConnectorSourceServer:localhost:8080 Provider:google GoogleProject:level-scheme-173421 DomainFilter:[external.daneyon.com] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSAssumeRole: AWSBatchChangeSize:4000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: CloudflareProxied:false InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:my-identifier TXTPrefix: Interval:1m0s Once:false DryRun:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha CRDSourceKind:DNSEndpoint ServiceTypeFilter:[]}"
time="2018-10-03T17:30:27Z" level=info msg="Created Kubernetes client https://10.96.0.1:443"
time="2018-10-03T17:30:52Z" level=error msg="Get https://www.googleapis.com/dns/v1/projects/level-scheme-173421/managedZones?alt=json: dial tcp: lookup www.googleapis.com on 10.96.0.10:53: read udp 172.17.0.10:43346->10.96.0.10:53: i/o timeout"
time="2018-10-03T17:31:28Z" level=info msg="All records are already up to date"
time="2018-10-03T17:32:27Z" level=info msg="All records are already up to date"
time="2018-10-03T17:33:27Z" level=info msg="All records are already up to date"
<SNIP>
Raw
ext-dns-gke.md

gcloud dns ref: https://cloud.google.com/dns/records/

Note: For some reason the ext-dns conrtoller is not removing a/txt records from g-dns when a svc is removed from a cluster. I am having to manually remove the records using gcloud:

$ gcloud dns record-sets transaction start -z=external-daneyon-com
$ gcloud dns record-sets transaction remove --zone "external-daneyon-com" \
--name nginx.external.daneyon.com. --type A --ttl 300 35.233.152.40
$ gcloud dns record-sets transaction remove --zone "external-daneyon-com" \
--name nginx.external.daneyon.com. --type TXT --ttl 300 \
"heritage=external-dns,external-dns/owner=my-identifier,external-dns/resource=service/test-namespace/test-service"
$ gcloud dns record-sets transaction execute -z=external-daneyon-com

If I change the svc/ing (i.e. delete/create with a new IP), then the ext-dns controller properly deletes the records and creates new ones:

time="2018-10-04T23:12:50Z" level=info msg="All records are already up to date"
time="2018-10-04T23:13:50Z" level=info msg="Change zone: external-daneyon-com"
time="2018-10-04T23:13:50Z" level=info msg="Del records: ing.external.daneyon.com. A [35.186.197.77] 300"
time="2018-10-04T23:13:50Z" level=info msg="Del records: ing.external.daneyon.com. TXT [\"heritage=external-dns,external-dns/owner=my-identifier,external-dns/resource=ingress/test-namespace/test-ingress\"] 300"
time="2018-10-04T23:13:50Z" level=info msg="Add records: ing.external.daneyon.com. A [35.241.21.55] 300"
time="2018-10-04T23:13:50Z" level=info msg="Add records: ing.external.daneyon.com. TXT [\"heritage=external-dns,external-dns/owner=my-identifier,external-dns/resource=ingress/test-namespace/test-ingress\"] 300"

Create the two k8s clusters:

$ gcloud container clusters create cluster1 --enable-kubernetes-alpha --cluster-version 1.11.2-gke.4 --no-enable-autorepair --no-enable-autoupgrade --num-nodes 1 --machine-type n1-highcpu-2 --scopes "https://www.googleapis.com/auth/ndev.clouddns.readwrite"

$ gcloud container clusters create cluster2 --enable-kubernetes-alpha --cluster-version 1.11.2-gke.4 --no-enable-autorepair --no-enable-autoupgrade --num-nodes 1 --machine-type n1-highcpu-2 --scopes "https://www.googleapis.com/auth/ndev.clouddns.readwrite"

Set kubectl creds:

$ gcloud container clusters get-credentials cluster1
$ gcloud container clusters get-credentials cluster2

Edit kubectl config file to match fed-v2 docs (i.e. use cluster1/2 for context, etc..).

$ gcloud dns record-sets list --zone="external-daneyon-com"
NAME                         TYPE  TTL    DATA
external.daneyon.com.        NS    21600  ns-cloud-b1.googledomains.com.,ns-cloud-b2.googledomains.com.,ns-cloud-b3.googledomains.com.,ns-cloud-b4.googledomains.com.
external.daneyon.com.        SOA   21600  ns-cloud-b1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
nginx.external.daneyon.com.  A     300    35.230.86.209
nginx.external.daneyon.com.  TXT   300    "heritage=external-dns,external-dns/owner=my-identifier,external-dns/resource=service/default/nginx"

To see just the A record:

$ gcloud dns record-sets list --zone="external-daneyon-com" --name "nginx.external.daneyon.com." --type A
NAME                         TYPE  TTL  DATA
nginx.external.daneyon.com.  A     300  35.230.86.209
Raw
ext-dns-note.md
Raw
fedv2-gke-deploy-issue.md

Update kube config context info to match fedv2 docs cluster1 and cluster2.

You must add bind your gke id to the admin role for fedv2 to create the necessary cr and crb in the host cluster: https://raw.githubusercontent.com/coreos/prometheus-operator/master/Documentation/troubleshooting.md

kubectl create clusterrolebinding daneyon-cluster-admin-binding --clusterrole=cluster-admin --user=daneyonhansen@gmail.com

Then you can run the join command:

$ kubefed2 join cluster1 --host-cluster-context cluster1 --add-to-registry --v=2
$ kubefed2 join cluster2 --host-cluster-context cluster1 --add-to-registry --v=2
Raw
fedv2_ingress-steps.md

Follow fed-v2 user guide to deploy until you get to the example k8s resources: https://github.com/kubernetes-sigs/federation-v2/blob/master/docs/userguide.md

Update the example federated svc:

# git diff -w example/sample1/federatedservice-template.yaml
diff --git a/example/sample1/federatedservice-template.yaml b/example/sample1/federatedservice-template.yaml
index f8f3f13..9dcd286 100644
--- a/example/sample1/federatedservice-template.yaml
+++ b/example/sample1/federatedservice-template.yaml
@@ -8,7 +8,8 @@ spec:
     spec:
       selector:
         app: nginx
-      type: NodePort
+      #type: NodePort
       ports:
         - name: http
-          port: 80
+          port: 8088
+          targetPort: 80

Then follow the docs to deploy the fed example k8s resources.

Follow the nginx docs to deploy the ingress controller: https://kubernetes.github.io/ingress-nginx/deploy/

Note: Use the bare-metal step for deployment (due to minikube).

Create a federated ingress template and placement for the test-service:

# cat federatedingress-template.yaml 
apiVersion: v1
kind: List
items:
- apiVersion: core.federation.k8s.io/v1alpha1
  kind: FederatedIngress
  metadata:
    name: test-ingress
    namespace: test-namespace
  spec:
    template:
      metadata:
        annotations:
          kubernetes.io/ingress.class: nginx
        name: test-ingress
        namespace: test-namespace
      spec:
        rules:
        - host: foo.bar.com
          http:
            paths:
            - path: /
              backend:
                serviceName: test-service
                servicePort: 8088

# cat federatedingress-placement.yaml 
apiVersion: v1
kind: List
items:
- apiVersion: core.federation.k8s.io/v1alpha1
  kind: FederatedIngressPlacement
  metadata:
    name: test-ingress
    namespace: test-namespace
  spec:
    clusterNames:
    - cluster2
    - cluster1

Test access to the example svc through the ingress for each minikube cluster. Since the nginx ingress controller for baremetal uses NodePort, get the Node IP of the nginx pod and the port of the nginx svc:

$ kubectl get po/nginx-ingress-controller-6bd7c597cb-r7fnb -n ingress-nginx -o yaml | grep hostIP
  hostIP: 10.138.0.2
$ kubectl get svc/ingress-nginx -n ingress-nginx
NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.103.166.128   <none>        80:30046/TCP,443:32221/TCP   54m

Test access:

$ curl -HHost:foo.bar.com http://10.138.0.2:30046/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Repeat for cluster 2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment