Deploy a web app on a dedicated server with Kamal

DEPLOY_WITH_KAMAL_ON_DEDICATED_SERVER.md

Notes

This guide uses Kamal 1.8.1

Motivation

Kamal was designed with 1 service = 1 droplet/VPS in mind.
But I'm cheap and I want to be able to deploy multiple demo/poc apps apps on my $20/month dedicated server.
What the hell, I'll even host my private container registry on it.

Setup your dedicated server

• https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-22-04
• https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-22-04
• https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-22-04
• https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-22-04
• https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-22-04
• https://www.digitalocean.com/community/tutorials/how-to-set-up-minio-object-storage-server-in-standalone-mode-on-ubuntu-20-04
• https://min.io/docs/minio/linux/integrations/generate-lets-encrypt-certificate-using-certbot-for-minio.html

Setup your domain names

• Add A/AAAA record in your DNS records for server.mydomain.com, registry.mydomain.com, myapp1.mydomain.com and myapp2.mydomain.com to the IP of your dedicated server

Setup a private container registry

• https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-ubuntu-22-04

Note 1: This tutorial says you need a host server and a client server. In our case, we will use only one server to be both client and server.

Note 2: The port 5000 is already used by datadog-agent by default, so I prefer using 7000:5000 instead.

Install SSL certificates for the registry

• Run sudo certbot certonly --nginx -d registry.mydomain.com
• Add to /etc/nginx/sites-enabled/registry:

  server {
    ...
  
    listen 443 ssl;
    listen [::]:443 ssl;
  
    ssl_certificate /etc/letsencrypt/live/registry.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/registry.domain.com/privkey.pem;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!MD5;
    
    location / {
      ...
      
      proxy_pass                          http://localhost:7000;
      ...
    }
  }

Open postgresql access to the docker containers

• Add host all all 172.16.0.0/12 scram-sha-256 to /etc/postgresql/16/main/pg_hba.conf
• Add listen_addresses = '*' to /etc/postgresql/16/main/postgresql.conf

Open redis access to the docker containers

• Edit /etc/redis/redis.conf to set bind 0.0.0.0 and protected-mode no
• Allow access for docker containers sudo ufw allow proto tcp from 172.16.0.0/12 to any port 6379

Create a database

• sudo -u postgres createuser myapp1 --createdb --pwprompt --encrypted
• Update database.yml, add DB_HOST, DB_USERNAME, DB_PASSWORD, DB_PORT to your .env.erb file.

  production:
    <<: *default
    host: <%= ENV["DB_HOST"] %>
    database: myapp1_production
    username: <%= ENV["DB_USERNAME"] %>
    password: <%= ENV["DB_PASSWORD"] %>
    port: <%= ENV.fetch("DB_PORT") { 5432 } %>

Setup the traefik manager app with Kamal and Traefik

With the standard use of Kamal, a Traefik instance would be deployed for each app, which would end up with conflict between the Traefik instances. Instead, we are going to deploy an empty web app with a single Traefik instance, while Traefik will be disabled on the other apps.

• Create a new rails app (could also be sinatra or whatever) rails new traefik_manager --minimal --skip-active-record --skip-asset-pipeline --skip-test
• Run Docker locally
• Install Kamal. Follow: [https://kamal-deploy.org/docs/installation]
• Add your registry credentials in .env
• Run bin/kamal env push
• Create the file .kamal/hook/pre-deploy and make it executable chmod +x .kamal/hook/pre-deploy

#!/usr/bin/env bash

REMOTE_HOST="[email protected]"
NETWORK_NAME="traefik-network"
# SSH into the remote host and execute Docker commands
ssh $REMOTE_HOST << EOF
 # Check if the Docker network already exists
 if ! docker network inspect "$NETWORK_NAME" &>/dev/null; then
     # If it doesn't exist, create it
     docker network create "$NETWORK_NAME"
     echo "Created Docker network: $NETWORK_NAME"
 else
     echo "Docker network $NETWORK_NAME already exists, skipping creation."
 fi
EOF

• Choose a unique port for your apps, use sudo lsof -i -P -n | grep LISTEN to check which ports are already in use. For this guide I'll choose 7001 and 7002.
• Update config/deploy.yml

# Name of your application. Used to uniquely configure containers.
service: traefik-manager

# Name of the container image.
image: n-studio/traefik-manager

# Deploy to these servers.
servers:
  web:
    hosts:
      - server.mydomain.com
    options:
      network: "traefik-network"

# Credentials for your image host.
registry:
  # Specify the registry server, if you're not using Docker Hub
  server: registry.mydomain.com
  username:
    - KAMAL_REGISTRY_USERNAME
  # Always use an access token rather than real password when possible.
  password:
    - KAMAL_REGISTRY_PASSWORD

# Inject ENV variables into containers (secrets come from .env).
# Remember to run `kamal env push` after making changes!
env:
  secret:
    - RAILS_MASTER_KEY

# Use a different ssh user than root
ssh:
  user: ubuntu

# Configure custom arguments for Traefik
traefik:
  publish: false
  options:
    network: "traefik-network"
    publish:
      - "7001:7001"
      - "7002:7002"
  args:
    entryPoints.web-7001.address: ':7001'
    entryPoints.web-7002.address: ':7002'

# Configure a custom healthcheck (default is /up on port 3000)
healthcheck:
  cmd: echo "OK"

• Run bin/kamal setup
• Run bin/kamal traefik reboot after any change in the traefik configuration

Install Kamal in your web apps

• Run Docker locally
• Follow: [https://kamal-deploy.org/docs/installation]
• Add database/redis credentials
• Run bin/kamal env push
• Disable traefik with traefik: false in deploy.yml
• Add labels to link the app with port 7001 or 7002 in deploy.yml

The deploy.yml file should look like this:

# Name of your application. Used to uniquely configure containers.
service: my-app-1

# Name of the container image.
image: n-studio/my-app-1

# Deploy to these servers.
servers:
  web:
    hosts:
      - server.mydomain.com
    options:
      network: "traefik-network"
    traefik: false

# Credentials for your image host.
registry:
  # Specify the registry server, if you're not using Docker Hub
  server: registry.mydomain.com

  # Always use an access token rather than real password when possible.
  username:
    - KAMAL_REGISTRY_USERNAME
  password:
    - KAMAL_REGISTRY_PASSWORD

# Inject ENV variables into containers (secrets come from .env).
# Remember to run `kamal env push` after making changes!
env:
  clear:
    DB_HOST: 172.17.0.1
    DB_PORT: 5432
    REDIS_URL: redis://172.17.0.1:6379/1 # give a unique db number for each app
  secret:
    - RAILS_MASTER_KEY
    - DB_USERNAME
    - DB_PASSWORD

# Use a different ssh user than root
ssh:
  user: ubuntu

labels:
  traefik.http.routers.my-app-1-web.rule: PathPrefix(`/`)
  traefik.http.routers.my-app-1-web.entrypoints: web-7001
  traefik.http.routers.my-app-1-web.priority: 10
  traefik.http.services.my-app-1-web.loadbalancer.server.port: 3000

• Note: if you use destination with Kamal (eg. staging), you might want to write: traefik.http.routers.my-app-1-web-staging.rule: PathPrefix(`/`)
• Run bin/kamal setup
• Repeat the same for your other app with a different port (7002)

Create SSL certificates

• Run sudo certbot certonly --nginx -d myapp1.mydomain.com
• Edit production.rb in your app to set config.assume_ssl = true

Install reverse proxy with nginx

server {
    server_name myapp1.mydomain.com;
    listen 80;
    listen [::]:80;

    ## redirect http to https ##
    rewrite ^ https://$server_name$request_uri? permanent;
}

server {
    server_name myapp1.mydomain.com;
    listen 443 ssl;
    listen [::]:443 ssl;

    ssl_certificate /etc/letsencrypt/live/myapp1.mydomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/myapp1.mydomain.com/privkey.pem;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location / {
        proxy_pass                          http://localhost:7001;
        proxy_set_header  Host              $http_host;   # required for docker client's sake
        proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
        proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;
        proxy_set_header  X-Forwarded-Ssl   on;
        proxy_set_header  X-Forwarded-Port  $server_port;
        proxy_set_header  X-Forwarded-Host  $host;
        proxy_read_timeout                  900;
    }
    
    location /cable {
        proxy_pass                           http://localhost:7001/cable;
        proxy_http_version                   1.1;
        proxy_set_header   Upgrade           $http_upgrade;
        proxy_set_header   Connection        "Upgrade";
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   Host              $http_host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-Proto https;
        proxy_redirect                       off;
    }
    
    location ~ ^/assets/ {
        proxy_pass               http://localhost:7001;
    
        expires                  1y;
        add_header Cache-Control public;
        add_header ETag          "";
    }
}

• Run sudo service nginx restart

Visit website

• Open https://app1.mydomain.com in your browser

Run console (for Rails)

• kamal app exec -i 'bin/rails console'

Out of curiosity, if each app gets its own traefik instance which use a different host port, why the need for the routing (traefik.http.routers.my-app-1-web.entrypoints: my-app-1-web)? I would imagine that when Kamal deploys, it tells the traefik instance for app1 to bind to the exposed ports on app1, and same for app2.

In other words, I've seen the usage of traefik router when you have 1 traefik instance, and it needs to conditionally route to app1 or app2, but if you have 2 traefik instances, wouldn't they just know from how Kamal deploys them to route everything to/from the container they deploy with?

@gregschmit You're right! I updated the gist!

@superp I managed to make it work with the traefik: false configuration and updated the guide!

Note: Adding a worker service for solid queue will cause a gateway timeout error (no idea why). The only workaround I've found was to run solid queue with puma via plugin :solid_queue

Note for upgrading to Kamal 2:
Use port 8080 for registry because Kamal 2 needs the ports 80 and 443 to be completely available.