Postal on Google Container Engine

assets-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: postal-assets
spec:
  template:
    metadata:
      name: postal-assets
      labels:
        name: postal-assets
    spec:
      restartPolicy: Never
      containers:
        - image: andyshinn/postal-app
          name: assets
          command: ["bundle", "exec", "rake", "assets:precompile"]
          volumeMounts:
            - mountPath: /opt/postal/config
              name: config
            - mountPath: /usr/src/app/public/assets
              name: assets
              readOnly: false
      restartPolicy: Never
      volumes:
        - name: config
          configMap:
            name: postal-config
        - name: assets
          gcePersistentDisk:
            pdName: postal-assets
            fsType: ext4

cron-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: cron
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        postal.app: cron
    spec:
      containers:
        - image: andyshinn/postal-app
          name: cron
          command: ["bundle", "exec", "rake", "postal:cron"]
          ports:
            - containerPort: 2525
          resources: {}
          volumeMounts:
            - mountPath: /opt/postal/config
              name: config
      restartPolicy: Always
      volumes:
        - name: config
          configMap:
            name: postal-config
status: {}

fast-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: fast
spec:
  replicas: 2
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        postal.app: fast
    spec:
      containers:
        - image: andyshinn/postal-app
          name: fast
          command: ["bundle", "exec", "rake", "postal:fast_server"]
          ports:
            - containerPort: 5010
              protocol: TCP
            - containerPort: 5011
              protocol: TCP
          volumeMounts:
            - mountPath: /opt/postal/config
              name: config
      restartPolicy: Always
      volumes:
        - name: config
          configMap:
            name: postal-config

fast-service.yaml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    postal.app: fast
  name: fast
spec:
  clusterIP: None
  ports:
  - name: headless
    port: 55555
    targetPort: 0
  selector:
    postal.app: fast
status:
  loadBalancer: {}

GCLOUD-EXAMPLE.md

Postal on Kubernetes in Google Cloud Platform

Requirements

1. Google Cloud Platform (GCP) account.
2. A project in the account (we're using postal-165921 in our example commands).

Infrastructure

1. Set the defailt zone to use with our gcloud commands: gcloud config set compute/zone us-central1-a
2. gcloud config set container/cluster postal
3. Navigate to https://console.cloud.google.com/ and create a new project for Postal (our example one will be called Postal and has an ID of postal-165921).
4. Navigate to the Container Engine section and create a new cluster called postal. The command we are using to create is gcloud container --project "postal-165921" clusters create "postal" --zone "us-central1-a" --machine-type "n1-standard-1" --image-type "COS" --disk-size "100" --scopes "https://www.googleapis.com/auth/compute","https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --num-nodes "3" --network "default" --enable-cloud-logging --no-enable-cloud-monitoring
5. In a moment you should be able to run gcloud container clusters get-credentials postal --zone us-central1-a --project postal-165921 to set up your local kubectl credentials.
6. Create a MySQL 5.7 instance in the SQL console. Our example is called postal.
7. In IAM and Admin create a service account. Role will be SQL Client and check to Furnish new private key (JSON type). Save the key somewhere locally.
8. Create the proxy user with gcloud beta sql users create postal cloudsqlproxy~% --instance=postal --password=p0st4l
9. Run gcloud sql instances describe postal to get the connectionName. Our example is postal-165921:us-central1:postal.
10. kubectl create secret generic cloudsql-instance-credentials --from-file=credentials.json=/Users/andy/Documents/Postal-d609301cc404.json
11. kubectl create secret generic cloudsql-db-credentials --from-literal=username=postal --from-literal=password=p0st4l
12. Create a grants.sql file with the following contents:

CREATE DATABASE `postal` CHARSET utf8mb4 COLLATE utf8mb4_unicode_ci;
GRANT ALL ON `postal`.* TO `postal`@`cloudsqlproxy~%`;
GRANT ALL PRIVILEGES ON `postal-%` . * to `postal`@`cloudsqlproxy~%`;

1. Create a new storage bucket and upload the grants.sql file to it.
2. Import this SQL file into the new postal database.
3. Enable Google Cloud SQL API at https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=postal-165921&duration=PT1H.
4. Create two external IP addresses (mx1 and mx2) for later use in SMTP load balancers.

Secrets

1. Generate a SSL certificate using openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=postaldemo/O=postaldemo".
2. kubectl create secret tls postal-tls --key tls.key --cert tls.crt

Application

1. Create your configuration map using kubectl create configmap postal-config --from-file ~/.postal --dry-run -o yaml. This assumes you already have the files from the postal initialize-config command living at ~/.postal.

DNS

We are using the domain postaldemo.com for our demo. We are going to host it in GCP.

1. Create the zone: gcloud dns managed-zones create postaldemo --dns-name postaldemo.com --description "Postal demonstration domain".
2. Get the name servers to use at the registrar using gcloud dns managed-zones describe postaldemo and update your registrar name server records.
3. Set static IPs for MX1 and MX2 to the instances in the SMTP server pool: gcloud compute addresses create mx1 mx2 --addresses $(gcloud compute instances list --filter='name:postal-smtp' --format='value[terminator=","](networkInterfaces[0].accessConfigs[0].natIP)')
4. Create MX1: gcloud dns record-sets transaction add --zone postaldemo --type A --name mx1.postaldemo.com. --ttl 300 $(gcloud compute addresses describe mx1 --format "value(address)").
5. Create MX2: gcloud dns record-sets transaction add --zone postaldemo --type A --name mx2.postaldemo.com. --ttl 300 $(gcloud compute addresses describe mx2 --format "value(address)").
6. Create SMTP endpoint: gcloud dns record-sets transaction add --zone postaldemo --type CNAME --name smtp.postaldemo.com. --ttl 300 $(gcloud compute addresses describe mx2 --format "value(address)") $(gcloud compute addresses describe mx1 --format "value(address)").

Future

• Convert all the initial GCP stuff to Terraform.
• Use https://github.com/kubernetes-incubator/external-dns instead of manual DNS.
• Figure out if we can use a PersistentVolume in a job for read/write and then read-only in the pods (for assets). Currently using a gcePersistentDisk for this instead.
• How to add 443 as Ingress controllers frontend ports.

initialize-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: postal-initialize
spec:
  template:
    metadata:
      name: postal-initialize
      labels:
        name: postal-initialize
    spec:
      restartPolicy: Never
      containers:
        - image: gcr.io/cloudsql-docker/gce-proxy:1.09
          name: cloudsql-proxy
          command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                    "-instances=postal-165921:us-central1:postal=tcp:3306",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
            - name: ssl-certs
              mountPath: /etc/ssl/certs
            - name: cloudsql
              mountPath: /cloudsql
        - image: andyshinn/postal-app
          name: initialize
          command: ["bash", "-c", "sleep 10 && bundle exec rake db:schema:load db:seed"]
          volumeMounts:
            - mountPath: /opt/postal/config
              name: config
      restartPolicy: Never
      volumes:
        - name: config
          configMap:
            name: postal-config
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir:

mx-services.yaml

apiVersion: v1
kind: Service
metadata:
  name: mx1
spec:
  ports:
    - port: 25
      protocol: TCP
      targetPort: 2525
  selector:
    postal.app: smtp
  type: LoadBalancer
  loadBalancerIP: 130.211.122.23
---
apiVersion: v1
kind: Service
metadata:
  name: mx2
spec:
  ports:
    - port: 25
      protocol: TCP
      targetPort: 2525
  selector:
    postal.app: smtp
  type: LoadBalancer
  loadBalancerIP: 104.198.75.93

rabbitmq-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  name: rabbitmq
spec:
  replicas: 3
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        postal.app: rabbitmq
    spec:
      containers:
        - image: andyshinn/postal-rabbitmq
          env:
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: RABBITMQ_NODENAME
              value: "rabbit@$(MY_POD_IP)"
            - name: RABBITMQ_USE_LONGNAME
              value: "true"
            - name: AUTOCLUSTER_TYPE
              value: k8s
            - name: RABBITMQ_ERLANG_COOKIE
              value: PHIOCHASOUQUIAXUFIETH
            - name: RABBITMQ_DEFAULT_PASS
              valueFrom:
                secretKeyRef:
                  name: postal-secrets
                  key: rabbitpassword
            - name: RABBITMQ_DEFAULT_USER
              value: postal
            - name: RABBITMQ_DEFAULT_VHOST
              value: postal
          name: rabbitmq
          ports:
            - name: management
              containerPort: 15672
            - name: amqp
              containerPort: 5672
          resources: {}
          livenessProbe:
            exec:
              command: ["rabbitmqctl", "node_health_check"]
            initialDelaySeconds: 15
            periodSeconds: 15
      restartPolicy: Always
status: {}

rabbitmq-service.yaml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    postal.app: rabbitmq
  name: rabbitmq
spec:
  ports:
  - name: "http"
    port: 15672
    targetPort: 15672
  - name: "amqp"
    port: 5672
    targetPort: 5672
  selector:
    postal.app: rabbitmq
status:
  loadBalancer: {}

requeuer-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: requeuer
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        postal.app: requeuer
    spec:
      containers:
        - image: gcr.io/cloudsql-docker/gce-proxy:1.09
          name: cloudsql-proxy
          command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                    "-instances=postal-165921:us-central1:postal=tcp:3306",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
            - name: ssl-certs
              mountPath: /etc/ssl/certs
            - name: cloudsql
              mountPath: /cloudsql
        - image: andyshinn/postal-app
          name: requeuer
          command: ["bundle", "exec", "rake", "postal:requeuer"]
          volumeMounts:
            - mountPath: /opt/postal/config
              name: config
      restartPolicy: Always
      volumes:
        - name: config
          configMap:
            name: postal-config
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir:
status: {}

smtp-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  name: smtp
spec:
  replicas: 2
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        postal.app: smtp
    spec:
      containers:
        - image: andyshinn/postal-app
          name: smtp
          command: ["bundle", "exec", "rake", "postal:smtp_server"]
          ports:
            - containerPort: 2525
              hostPort: 25
          resources: {}
          volumeMounts:
            - mountPath: /opt/postal/config
              name: config
      restartPolicy: Always
      volumes:
        - name: config
          configMap:
            name: postal-config
      nodeSelector:
        cloud.google.com/gke-nodepool: smtp
status: {}

smtp-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    postal.app: smtp
  name: smtp
spec:
  ports:
    - name: smtp
      protocol: TCP
      port: 25
      targetPort: 2525
  externalIPs:
    - 130.211.185.93
    - 104.154.77.75
  selector:
    postal.app: smtp

web-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  name: web
spec:
  replicas: 2
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        postal.app: web
    spec:
      containers:
        - image: gcr.io/cloudsql-docker/gce-proxy:1.09
          name: cloudsql-proxy
          command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                    "-instances=postal-165921:us-central1:postal=tcp:3306",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
            - name: ssl-certs
              mountPath: /etc/ssl/certs
            - name: cloudsql
              mountPath: /cloudsql
        - image: andyshinn/postal-app
          name: web
          command: ["bundle", "exec", "puma", "-C", "config/puma.rb"]
          env:
            - name: RAILS_SERVE_STATIC_FILES
              value: "true"
          ports:
            - containerPort: 5000
          resources: {}
          livenessProbe:
            httpGet:
              path: /login
              port: 5000
            initialDelaySeconds: 15
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /login
              port: 5000
            initialDelaySeconds: 15
            periodSeconds: 15
          volumeMounts:
            - mountPath: /opt/postal/config
              name: config
            - mountPath: /usr/src/app/public/assets
              name: assets
              readOnly: true
      restartPolicy: Always
      nodeSelector:
        cloud.google.com/gke-nodepool: default-pool
      volumes:
        - name: assets
          gcePersistentDisk:
            pdName: postal-assets
            fsType: ext4
            readOnly: true
        - name: config
          configMap:
            name: postal-config
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir:
status: {}

web-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web-external
  annotations:
    #kubernetes.io/ingress.global-static-ip-name: "test-ip"
    kubernetes.io/ingress.class: "gce"
spec:
  tls:
    - secretName: postal-secret
  backend:
    serviceName: web
    servicePort: 80

web-service.yaml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    postal.app: web
  name: web
spec:
  ports:
    - name: "web"
      port: 80
      targetPort: 5000
  selector:
    postal.app: web
  type: NodePort
status:
  loadBalancer: {}

worker-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: worker
spec:
  replicas: 2
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        postal.app: worker
    spec:
      containers:
        - image: gcr.io/cloudsql-docker/gce-proxy:1.09
          name: cloudsql-proxy
          command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                    "-instances=postal-165921:us-central1:postal=tcp:3306",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
            - name: ssl-certs
              mountPath: /etc/ssl/certs
            - name: cloudsql
              mountPath: /cloudsql
        - image: andyshinn/postal-app
          name: worker
          command: ["bundle", "exec", "rake", "postal:worker"]
          volumeMounts:
            - mountPath: /opt/postal/config
              name: config
      restartPolicy: Always
      volumes:
        - name: config
          configMap:
            name: postal-config
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir:
status: {}