Skip to content

Instantly share code, notes, and snippets.

@eramax
Last active August 10, 2022 11:40
Show Gist options
  • Save eramax/2824be061b1b43a941c8e4da8bd95cad to your computer and use it in GitHub Desktop.
Save eramax/2824be061b1b43a941c8e4da8bd95cad to your computer and use it in GitHub Desktop.
Single Kubernetes Nginx To Support Unlimited Static Web Apps With Their Subdomains

The Goal

I want to be able to publish every code commit of our react app.

Azure support deployment slots, which can be used but unfortunately, there is limited count of deployment slots that you can get.

The solution

By installing nginx as a deployment on kubernetes (single instance could be fine, you don't need to have an instance for each deployment) and make sure that nginx http folder is get mounted from azure file share

Moreover, we need to configure nginx to locate the project files based on the domain name.

By this we will end up with the following scenario:

the user go to a.eample.com

there is an nginx-ingress controller on the aks and its task to forward any request to our nginx service

nginx service is going to forward the request to one of the nginx pods

nginx pod will receive the request which is the user wants a.example.com and then nginx will go to its primary www location which is mounted from azure file share to find a directory called a.example.com

nginx pod will find the desired directory a.example.com since it has been created by our CI/CD pipeline and the files already been added into the file share directory.

nginx pod will response to the user will all needed files.

Creating AKS

subscription=9cd
rg=dev2
location=SwedenCentral
domain=ddddd.ml
AZ_AKS_NAME=k8s
sta=ssss$rg
SHARE_NAME=webapp

az account set --subscription $subscription
az group create -n $rg --location $location	

az network vnet create -n k8sVnet -g $rg --location $location --address-prefix "10.1.0.0/16"
az network vnet subnet create -n k8sSubnet -g $rg --vnet-name k8sVnet --address-prefix "10.1.0.0/22"
az network public-ip create -g $rg -n aks-public-ip --allocation-method Static --location $location --sku standard
AKSPublicIP=$(az network public-ip show -g $rg -n aks-public-ip | jq .ipAddress -r)

DNS_ID=$(az network dns zone create -g $rg -n $domain --query id -o tsv) 

az network dns zone create -g $rg -n $domain
az network dns zone show -g $rg -n $domain --query nameServers -o tsv

az aks create \
    -n $AZ_AKS_NAME \
    -g $rg \
    --enable-managed-identity \
    --location $location \
    --node-count 1 \
    --zones 1 2\
    --node-vm-size Standard_D4as_v5 \
    --network-plugin azure \
    --vnet-subnet-id /subscriptions/$subscription/resourceGroups/$rg/providers/Microsoft.Network/virtualNetworks/k8sVnet/subnets/k8sSubnet \
    --max-pods 150 \
    --generate-ssh-keys

az aks get-credentials -g $rg -n $AZ_AKS_NAME

aksidentityprid=$(az aks show -n $AZ_AKS_NAME -g $rg | jq -r .identity.principalId)
az role assignment create --role "Network Contributor" --assignee $aksidentityprid --scope /subscriptions/$subscription/resourceGroups/$rg

Creating azure storage - file share

# Create a storage account
az storage account create -n $sta -g $rg -l $location --sku Standard_LRS

# Export the connection string as an environment variable, this is used when creating the Azure file share
AZURE_STORAGE_CONNECTION_STRING=$(az storage account show-connection-string -n $sta -g $rg -o tsv)

# Create the file share
az storage share create -n $SHARE_NAME --connection-string $AZURE_STORAGE_CONNECTION_STRING

# Get storage account key
STORAGE_KEY=$(az storage account keys list --resource-group $rg --account-name $sta --query "[0].value" -o tsv)

# Echo storage account name and key
echo Storage account name: $sta
echo Storage account key: $STORAGE_KEY

kubectl create secret generic azure-secret --from-literal=azurestorageaccountname=$sta --from-literal=azurestorageaccountkey=$STORAGE_KEY

Mount azure file share to aks

apiVersion: v1
kind: PersistentVolume
metadata:
  name: azurefile
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: azurefile-csi
  csi:
    driver: file.csi.azure.com
    readOnly: true
    volumeHandle: 44830fa8-79b4-406b-8b58-621ba25353fd  # make sure this volumeid is unique in the cluster
    volumeAttributes:
      resourceGroup: dev2  # optional, only set this when storage account is not in the same resource group as agent node
      shareName: webapp
    nodeStageSecretRef:
      name: azure-secret
      namespace: default
  mountOptions:
    - dir_mode=0555
    - file_mode=0555
    - uid=0
    - gid=0
    - mfsymlinks
    - cache=strict
    - nosharesock
    - nobrl
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: azurefile
spec:
  accessModes:
    - ReadOnlyMany
  storageClassName: azurefile-csi
  volumeName: azurefile
  resources:
    requests:
      storage: 10Gi

Installing nginx-ingress controller on AKS

kubectl create namespace ngx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm upgrade  ingress-nginx ingress-nginx/ingress-nginx \
    --namespace ngx \
    --set controller.replicaCount=1 \
    --set controller.nodeSelector."kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."kubernetes\.io/os"=linux \
    --set controller.service.externalTrafficPolicy=Local \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-resource-group"="<Resource-Group-Name>" \
    --set controller.watchIngressWithoutClass=true \
    --set ingressClassResource.default=true \
    --set controller.service.loadBalancerIP="<PublicIP>"

Deploy nginx server on AKS

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |-
    user nginx;
    worker_processes  2;
    events {
      worker_connections  1024;
    }
    http {
      include    /etc/nginx/mime.types;
      index      index.html;
      server {
          root /usr/share/nginx/html/$http_host/;
          location / {
            try_files $uri /index.html;
            #try_files $uri $uri/ =404;
            #if (!-e $request_filename){
            #    rewrite ^(.*)$ /index.html break;
            #}
          }
      }
    }
  types: |
    types {
        text/html                             html htm shtml;
        text/css                              css;
        text/xml                              xml;
        image/gif                             gif;
        image/jpeg                            jpeg jpg;
        application/x-javascript              js;
        application/atom+xml                  atom;
        application/rss+xml                   rss;

        text/mathml                           mml;
        text/plain                            txt;
        text/vnd.sun.j2me.app-descriptor      jad;
        text/vnd.wap.wml                      wml;
        text/x-component                      htc;

        image/png                             png;
        image/tiff                            tif tiff;
        image/vnd.wap.wbmp                    wbmp;
        image/x-icon                          ico;
        image/x-jng                           jng;
        image/x-ms-bmp                        bmp;
        image/svg+xml                         svg svgz;
        image/webp                            webp;

        application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
        application/vnd.openxmlformats-officedocument.presentationml.slideshow    ppsx;
        application/vnd.openxmlformats-officedocument.presentationml.presentation    pptx;
        application/vnd.openxmlformats-officedocument.spreadsheetml.sheet        xlsx;

        application/java-archive              jar war ear;
        application/mac-binhex40              hqx;
        application/msword                    doc;
        application/pdf                       pdf;
        application/postscript                ps eps ai;
        application/rtf                       rtf;
        application/vnd.ms-excel              xls;
        application/vnd.ms-powerpoint         ppt;
        application/vnd.wap.wmlc              wmlc;
        application/vnd.google-earth.kml+xml  kml;
        application/vnd.google-earth.kmz      kmz;
        application/x-7z-compressed           7z;
        application/x-cocoa                   cco;
        application/x-java-archive-diff       jardiff;
        application/x-java-jnlp-file          jnlp;
        application/x-makeself                run;
        application/x-perl                    pl pm;
        application/x-pilot                   prc pdb;
        application/x-rar-compressed          rar;
        application/x-redhat-package-manager  rpm;
        application/x-sea                     sea;
        application/x-shockwave-flash         swf;
        application/x-stuffit                 sit;
        application/x-tcl                     tcl tk;
        application/x-x509-ca-cert            der pem crt;
        application/x-xpinstall               xpi;
        application/xhtml+xml                 xhtml;
        application/zip                       zip;

        application/octet-stream              bin exe dll;
        application/octet-stream              deb;
        application/octet-stream              dmg;
        application/octet-stream              eot;
        application/octet-stream              iso img;
        application/octet-stream              msi msp msm;

        audio/midi                            mid midi kar;
        audio/mpeg                            mp3;
        audio/ogg                             ogg;
        audio/x-realaudio                     ra;
        audio/x-m4a                           m4a;

        video/3gpp                            3gpp 3gp;
        video/mpeg                            mpeg mpg;
        video/quicktime                       mov;
        video/x-flv                           flv;
        video/x-mng                           mng;
        video/x-ms-asf                        asx asf;
        video/x-ms-wmv                        wmv;
        video/x-msvideo                       avi;

        video/mp4                             mp4;
        video/webm                            webm;
        video/x-m4v                           m4v;

    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-dep
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
          securityContext:
            allowPrivilegeEscalation: false
            #runAsUser: 0
          volumeMounts:
            - name: nginx-conf
              mountPath: /etc/nginx/
              #subPath: mime.types
              readOnly: true              
            - name: azure
              mountPath: "/usr/share/nginx/html"
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-conf
          items:
            - key: nginx.conf
              path: nginx.conf 
            - key: types
              path: mime.types      
      - name: azure
        persistentVolumeClaim:
          claimName: azurefile          
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  labels:
    app: nginx
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - http:
      paths:
      - path: /?(.*)
        pathType: Prefix
        backend:
          service:
            name: nginx-svc
            port:
              number: 80

Now you can upload multiple projects on the file share and each project is under a specific directory with directory name equals to a domain or any subdomain and then you will be able to reach out this domain directly

make sure to add an A DNS record of * to point to our public ip address.

Best.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment