Skip to content

Instantly share code, notes, and snippets.

@khusseini
Last active August 25, 2022 04:04
Show Gist options
  • Save khusseini/1cd92e21fc0db2caafb5c65230dc22da to your computer and use it in GitHub Desktop.
Save khusseini/1cd92e21fc0db2caafb5c65230dc22da to your computer and use it in GitHub Desktop.
Adding traefik to LKS

Setup Traefik 2.2 with Let's Encrypt On LKE

If you searched for this article I chances are you know what LKE, Traefik and Let's Encrypt are, else here is a quick primer:

What is LKE?

"The Linode Kubernetes Engine (LKE) is a fully-managed container orchestration engine for deploying and managing containerized applications and workloads.". For more information please have a look at "Deploy and manage a cluster with Linode Kubernetes Engine"

What is Traefik?

Traefik "is an open source edge router" specifically designed for clusters in order to provide dynamic routing configuration and service discovery and is "natively compilant with every major cluster technology".

https://docs.traefik.io

What is Let's Encrypt?

Let's Encrypt was brought to live by the Internet Security Research Group in order to provide free and autoconfigurable SSL certificates for everyone.

https://letsencrypt.org/about/

In this guide

In this guide you will learn how you can provide Let's Encrypt SSL certificates for you web application in LKE using Traefik's dynamic configuration and routing.

Deploy Traefik CRDs and deployment

Traefik's documentation does already provide enough information to deploy working traefik routing onto your cluster. We can directly use their CRD and RBAC definitions which you can find in Kubernetes CRD or directly from Github:

  1. Deploy Custom Resource Definitions
    kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v2.2/docs/content/reference/dynamic-configuration/kubernetes-crd-definition.yml
  2. Deploy RBAC
    kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v2.2/docs/content/reference/dynamic-configuration/kubernetes-crd-rbac.yml

These resources configure Kubernetes to allow us to create new resources which handle routing for your services. You can find an example of available resources here.

Before we can use the new traefik resources we need to deploy traefik itself, but instead of using their example deployment we will create a new definition that is a little bit more elaborate.

Instead of configuring traefik with container arguments, we will deploy traefik with a ConfigMap.

  • manifests/configmap.yml:
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: traefik
      namespace: default
    data:
      traefik.yaml: |
        log:
          level: DEBUG
        providers:
          kubernetescrd: {}
        certificatesResolvers:
          myresolver:
            acme:
              email: [email protected]
              storage: /traefik/acme/acme.json
              caserver: https://acme-staging-v02.api.letsencrypt.org/directory
              tlsChallenge: {}
        entryPoints:
          web:
            address: ":8000"
            http:
              redirections:
                entryPoint:
                  to: ":443"
                  scheme: https
          websecure:
            address: ":4443"
    

This configuration tells traefik to create two endpoints, one named web and the other websecure. The web endpoint listens on port :8000 and redirects any http traffic to port :443 with the https schema. The second endpoint, websecure listens on port :4443 and accepts incoming traffic.

Note that the loglevel is set to DEBUG and the acme caserver is pointed to the staging server. Change the log.level to INFO and remove the caserver field once you know that everything is working. The staging server does not issue valid SSL certificates but has a much higher rate limit for testing purposes.

The redirect from web to port 443 is a little bit confusing here because we didn't talk about how traffic gets routed to your cluster anyway. As exaplained in What is Traefik?, traffic handled by trefik is routed based upon dynamic configuration which traefik receives from Kubernetes, however unlike general implementations of traefik, we do not deploy Traefik with a NodePort nor as a DaemonSet because we rely on the LoadBalancer resource to create a Linode Node Balancer to point a domain to.

Here is a small sketch of the flow:

                    |
  Public Web        |       Linode Cloud
                    |
+---------------+  |:80  +-----------------------+
|      http      +-------->                       |
+----------------+  |     |   TCP Load Balancer   |
|      https     +-------->                       |
+--------^-------+  |:443 +-----------------------+        
         ¦          |        ¦http:8000  ¦https:4443       
         ¦          |        ¦           ¦                   
         ¦          |--------¦-----------¦-------------------
         ¦          |     +--v--+   +----v--------+     LKE  |
         +----------------+ web |   |  websecure  |          |
                    |     +-----+   +----+--------+          |
                    |                    ¦                   |
                    |     +--------------v--------+          | 
                    |     |                       |          |
                    |     |   dynamic services    |          |
                    |     |                       |          |
                    |     +-----------------------+          |

In other words, we will create a Service resource of type LoadBalancer configured to proxy all TCP traffic from public port :80 to Traefik and all TCP traffic from port :443 to Traefik on port :4443.

  • manifests/service.yml:
    apiVersion: v1
    kind: Service
    metadata:
      name: traefik-lb
    spec:
      type: LoadBalancer
      ports:
        - name: web
          port: 80
          protocol: TCP
          targetPort: 8000
        - name: websecure
          port: 443
          protocol: TCP
          targetPort: 4443
      selector:
        app: traefik
      sessionAffinity: None
    

The Linode CCM will use this resource to create a Linode NodeBalancer which you can use later to point your domain(s) to.

Last but not least we will create the actual traefik deployment. For simplicity this example won't properly store assigned acme certificates. If you need to store those certificates you can either use a shared volume you created or an etcd key value backend.

  • manifests/deployment.yml:
    kind: Deployment
    apiVersion: apps/v1
    metadata:
      name: traefik
      namespace: default
      labels:
        app: traefik
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: traefik
      template:
        metadata:
          labels:
            app: traefik
        spec:
          serviceAccountName: traefik-ingress-controller
          containers:
            - image: traefik:v2.2
              name: traefik
              ports:
                - name: web
                  containerPort: 8000
                - name: websecure
                  containerPort: 4443
              args:
                - --configfile=/config/traefik.yaml
              volumeMounts:
                - mountPath: /config
                  name: config
                - mountPath: /traefik/acme
                  name: acme
          volumes:
            - name: config
              configMap:
                name: traefik
            - name: acme
              emptyDir: {}

Now just deploy the resources:

kubectl apply -f manifests/configmap.yaml
kubectl apply -f manifests/deployment.yaml
kubectl apply -f manifests/service.yaml

Once all the pods have been deployed we can move on to deploy a sample backend.

Deploy sample backend with Let's Encrypt SSL Certificates

If you do not already have an application at hand that you deployed to your cluster you can follow this handy guide.

You can follow the guide completely up until you create the Service resource. As we already have a LoadBalancer configured (which points to traefik), we only need to create a headless service and a traefik resource.

  • manifests/static-site-service.yml:
    apiVersion: v1
    kind: Service
    metadata:
      name: static-site-service
      labels:
        app: static-site
    spec:
      ports:
      - name: http
        port: 80
        protocol: TCP
        targetPort: 80
      selector:
        app: static-site
      clusterIP: None
  • manifests/ingress-route.yml:
    apiVersion: traefik.containo.us/v1alpha1
    kind: IngressRoute
    metadata:
      name: static-site-route
    spec:
      entryPoints:
      - websecure
      routes:
      - match: Host(`www.example.com`)
        kind: Rule
        services:
          - name: static-site-service
            passHostHeader: true
            port: 80
      tls:
        certResolver: myresolver
    

The IngressRoute resource will instruct traefik to use its settings for routing traffic from the outside to a kubernetes service. The tls.certResolver options instructs traefik to fetch an SSL certificate from Let's Encrypt as configured in traefik's ConfigMap.

Now when you open your site you should see your new certificate. Once you validated the certificate you can remove the caserver entry from the ConfigMap and redeploy manifests/deployment.yml

@stefandevo
Copy link

Hi, thanks for this gist. I have a couple of questions:

  • if I understand correctly, the load balancing features of Traefik cannot be used with LKE hence it has it's own LB?
  • 1 replica will cause that only this instance/pod will host the SSL certificates?

@stefandevo
Copy link

What extra benefits are there to use traefik over default nginx ingress and is the setup somehow “slowing” down the cluster ?

@khusseini
Copy link
Author

Hi, thanks for this gist. I have a couple of questions:

* if I understand correctly, the load balancing features of Traefik cannot be used with LKE hence it has it's own LB?
  • AFAIK traefik's lb feature should be still active within the LKE, however the Linode LB sits on the outside to point to traefik, so trafik decides which services on which node to call.
* 1 replica will cause that only this instance/pod will host the SSL certificates?
  • Yes, this is a simplified example ("For simplicity this example won't properly store assigned acme certificates. If you need to store those certificates you can either use a shared volume you created or an etcd key value backend.")

As for extra benefits, I found traefik to be more convenient than nginx when it comes to simply act as a proxy gateway/LB.
I am not sure about traefik's performance compared to nginx but as it is the case with most server software, the devil is in the details of configuration.

@stefandevo
Copy link

Thx for your response. About the LB. So LKE performs a load balance first between all available nodes; then hits Traefik and Traefik will choose another(or the same) node to execute. Right?

About Let's Encrypt. I am really confused about https://doc.traefik.io/traefik/v2.0/providers/kubernetes-crd/ It is saying If you are wanting to continue to run Traefik Community Edition, LetsEncrypt HA can be achieved by using a Certificate Controller such as Cert-Manager so not sure if the trick with the shared volume would actually work.

@khusseini
Copy link
Author

Oh that's a bummer. I missed that part about HA with traefik. The docs continue showing a workaround, however I am not sure I like that workaround.

@stefandevo
Copy link

@khusseini
Copy link
Author

Thank you for the link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment