Skip to content

Instantly share code, notes, and snippets.

@cwant

cwant/dokku-mycc-presentation.md

Last active October 16, 2020 14:31
Show Gist options
  • Save cwant/42d53de4f31c7360bcfe6756f57a22b2 to your computer and use it in GitHub Desktop.
Save cwant/42d53de4f31c7360bcfe6756f57a22b2 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
dokku-mycc-presentation.md

Dokku on the Compute Canada cloud

First: lets provision a cloud VM

Lets spin up a cloud VM so we can talk about other stuff while we wait.

  • Deploy an Ubuntu 20.04 instance.
  • p4-3gb will work fine.
  • Make sure the keypair is injected
  • Make sure to associate the static IP to the one used for DNS (see below)
  • Don't forget to set a disk size!!!! (20GB will do)

DNS for public IP

I have set up three domain names (A records) pointing to the same floating IP in Google Cloud Platform (manages c3.ca):

  • whatever.uofa.c3.ca
  • whatever-rails.uofa.c3.ca
  • whatever-flask.uofa.c3.ca

Install Docker/Dokku on cloud VM

Lets SSH into our new app server VM (account ubuntu) and install Dokku:

wget https://raw.githubusercontent.com/dokku/dokku/v0.21.4/bootstrap.sh
sudo DOKKU_TAG=v0.21.4 bash bootstrap.sh

MTU!!!!

In /etc/docker/daemon.json:

{
  "mtu": 1450
}

Then service docker restart.

If you don't do this on East Cloud, pulling gems will fail with timeouts.

Acknowledgement

I saw a talk by Alan Vardy at the February 2019 YEGRB (Edmonton Ruby Meetup) on deploying Rails applications to Digital Ocean using Dokku. That talk inspired this presentation, and I am stealing a bunch of his stuff/steps. Here is a link to Alan's blog post on this subject:

https://www.alanvardy.com/posts/6

Goals

  • Set up Dokku on a cloud VM
  • Create a Ruby-on-Rails application and deploy to the cloud using Dokku
  • Create a flask application and deploy to the cloud using Dokku
  • Get an SSL cert
  • Play around with containers
  • Discuss some service implications

Dokku? What? Why?

https://github.com/dokku/dokku

19.7k stars, first commit was June 2013.

  • Dokku is a PaaS (platform-as-a-service) layer on top of IaaS (infrastucture-as-a-service, e.g., our cloud)
  • Dokku is inspired by the commercial hosting service Heroku
  • It uses Docker containers for managing services (e.g., application server, database server)
  • It uses git for deployment.
  • Magical container swapping ensures zero downtime during deployment (if new containers are broken, old containers keep serving application)
  • Ideally, we will be following the best practices of a '12 Factor App` (in particular, configuration through environment variables): https://12factor.net/

Note that on the commercial Heroku service, there is less setup required (the platform is all you have access to, not SSH access to a UNIX account). Setting up Dokku on an OpenStack cloud VM takes time so one has to ask 'Is it worth it?'.

Initial Dokku setup via HTTP

I visit http://whatever.uofa.c3.ca, which gives me a temporary setup webpage -- once the setup form is submitted, the page can't be accessed again.

(You can't visit it because of security rules, which will be loosened later.)

There is the opportunity to upload some public keys.

We can either run each application on a separate port, or as a subdomain of our server. We will choose the subdomain option.

I tell Dokku that the applications I run will be under uofa.c3.ca.

OPEN SECURITY RULES NOW!!! (note for self for during presentation)

Provision some services on cloud VM

On the command line of the cloud VM (as user ubunutu), create two applications, one for rails, one for flask.

dokku apps:create whatever-rails
dokku apps:create whatever-flask

(Despite the naming, Dokku figures out what kind of app it is based on what sort of files we push to it).

Configure dokku to use postgres:

sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git

We'll provision a database and hook it up to the Rails app

dokku postgres:create whateverdb
dokku postgres:link whateverdb whatever-rails

Notice that the environment variable DATABASE_URL is now set for the Rails app.

Create Rails app on development laptop

rails new

We want to create a new rails application that uses PostgreSQL by default, and we want to delay deploying the bundle.

rails new whatever-rails --skip-bundle --skip-webpack-install --skip-javascript --database=postgresql

Rails new also initializes a new git repository, so lets put what we have in git:

git add .
git commit -m "First commit on new rails app"

Add database and Dokku tools to Gemfile

We will run PostgreSQL in production, but SQLite3 for local development.

I had issues with the SQLite3 version, so need to be careful with the version

gem 'dokku-cli'

# Database-per-environment                                                      
group :development, :test do
  gem 'sqlite3', '~> 1.3.6'
end
group :production do
  gem 'pg', '>= 0.18', '< 2.0'
end

We need to also modify the development and production environments in config/database.yml. The development environment will use Sqlite3, and the production environment will be configured with the DATABASE_URL environment variable.

development:
  <<: *default
  adapter: sqlite3
  database: db/development.sqlite3

...

production:
  <<: *default
  url: <%= ENV['DATABASE_URL'] %>

Install the bundle

bundle install --path vendor/bundle

Important: I don't want to accidentally commit the local development bundle, or the database, so I'll add to .gitignore:

# Don't commit bundle
vendor/bundle

# Don't commit database
db/*.sqlite3

Let's ensure the web app runs:

bundle exec rails s

Now we can visit http://localhost:3000 in a browser to check out the web page.

Add and commit to Git:

git add .
git commit -m 'Web app works in development'

Set up a root web page

The Rails welcome page doesn't work in a production environment so we will create a root webpage that controls a database table:

bundle exec rails generate scaffold Thing name:string quantity:integer

Migrate our dev database:

bundle exec rails db:migrate

To the config/routes.rb we add a line in the Rails.application.routes.draw block to set this page as what gets loaded when people visit the root URL:

root 'things#index'

Confirm that this root page works by starting the webserver, then commit to git.

Setting up application container checks

The file CHECKS provides a test that our application is working when deployed to Dokku:

# CHECKS

WAIT=10  
ATTEMPTS=6  
/check.txt it_works

In order for the check to work, we add another route to config/routes.rb:

get '/check.txt', to: proc {[200, {}, ['it_works']]}

Telling Dokku that the database needs migration as part of the deployment pipeline

We need to create a file called app.json:

{
  "name": "whatever-rails",
  "description": "Whatever running on Dokku!",
  "keywords": [
    "dokku",
    "rails"
  ],
  "scripts": {
    "dokku": {
      "postdeploy": "bundle exec rails db:migrate"
    }
  }
}

Lets commit our app to git:

git add .
git commit -m 'Ready to deploy'

Deploying via git

Add a git remote pointing to our app server:

git remote add dokku [email protected]:whatever-rails

Our SSH key controls the authorization.

Finally, we deploy by pushing our master branch:

git push dokku master

While that's deploying, lets go to the OpenStack dashboard and change our cloud security group to allow public access. After deployment, we can visit our app at http://whatever-rails.uofa.c3.ca.

A Flask example

Create a directory somewhere on your work laptop for a flask project:

mkdir whatever-flask
cd whatever-flask

Create the python file whatever-flask.py

import os

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Like, whatever world!'

if __name__ == '__main__':
    # Bind to PORT if defined, otherwise default to 5000.
    port = int(os.environ.get('PORT', 5000))
    app.run(host='127.0.0.1', port=port)

We can now run our app locally: python whatever-flask.py (view it at http://localhost:5000).

We set up a requirements.txt file so that Dokku knows what python packages we need:

Flask==0.12.1
gunicorn==19.7.1

We create a Procfile so that Dokku knows how to start our application. We are using a Web-server gateway interface (WSGI) called 'Green Unicorn':

web: gunicorn whatever-flask:app --workers=4

Python dumps out lots of stuff we don't want in our repository, so update .gitignore:

__pycache__/
*.pyc

venv/

Create a git repository and commit to it:

git init
git add .
git commit -m "Deploy Flask with Dokku"

Set up our Dokku remote repository again:

git remote add dokku [email protected]:whatever-flask

Deploy by pushing:

git push dokku master

After deploy we visit http://whatever-flask.uofa.c3.ca.

SSL

IMPORTANT: Open VM security groups to public traffic first or cert request will fail

Dokku has a plugin to request and use a free SSL cert from Lets Encrypt. Via sudo on our app server, we install it:

sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git

Lets setup our an email to use when requesting certificates for our rails app:

dokku config:set --no-restart whatever-rails [email protected]

Now lets turn it on:

dokku letsencrypt whatever-rails

Visit: https://whatever-rails.uofa.c3.ca

Lets Encrypt certs have a short life (30 days?) so we set up a cron job on the app server to auto-renew our certificate before it expires:

dokku letsencrypt:cron-job --add

Extra stuff to try on the Dokku server

Show all applications running:

dokku apps

More verbose information:

dokku apps:report

Check out some logs:

dokku logs whatever-rails

Restart an application:

dokku ps:restart whatever-rails

Checking out the scaling of an application:

dokku ps:scale whatever-rails

Rescale the rails application to have four containers working:

dokku ps:scale whatever-rails web=4

List all containers:

sudo docker ps

Run bash in a container:

dokku enter whatever-rails web.1

Get a rails console:

dokku run whatever-rails console

Stop an application:

dokku ps:stop whatever-rails

Recover all applications after reboot:

dokku ps:restore

Database export and import:

dokku postgres:export [db_name] > [db_name].dump
dokku postgres:import [db_name] < [db_name].dump

dokku-cli

The rails app has a gem installed called dokku-cli which gives us access to some controls from the dev directory on the laptop:

bundle exec dokku ps
bundle exec dokku config

etc.

Summary

What did we see

  • Setup of dokku on a VM
  • Creation of apps
  • Nice deployment
  • Nice connection of services
  • Super-easy handling of SSL

What didn't we see

  • Lots of database creation, configuration stuff -- this was just easy
  • Lots of service configuration

Other thoughts

  • Would be super-nice if some of the Dokku stuff had a web interface in front of it. There is a project called 'wharf' on github that tries to do this -- tried it, and it almost worked (didn't have time to correct problems encountered).
  • Application deployment and SSL management is fantastic;
  • How to fix things when things go wrong? Probably needs some heavy container expertise.

Yeah, well so what?

So what do we want to do with this?

Is this something that we could offer as a service to our users? What would the service look like?

User would need to supply:

  • a pre-obtained hostname, or a subdomain of a domain we could provision for them;
  • whether they need a database (and/or other datastore, like Redis);
  • An email for LetsEncrypt.

We would give them:

  • the git remote, e.g. git remote add dokku [email protected]:super-science-thingy.
  • some assurances about database backup;
  • logging?
  • other admin tasks.

If we don't want to have a service offering, is this a path that we want to steer users down? Is it worth a workshop? Or have staff help with setup?

Or is this 'thanks, but no thanks' / 'not ready for prime time'? (or 'users can figure this out themselves'.)

Bonus: add database to flask app

REALLY DUMB STUFF:

  • the local dev instance will now be totally broken, some additional work is needed (e.g., install local database or hook it up to local rails database)
  • really, this is just an example that shows potential uses, not really a good recipe for anything

We'll modify the flask application to report the contents of the database connected to the rails application

On dokku VM:

dokku postgres:link whateverdb whatever-flask

Now flask app sees the DATABASE_URL environment variable

On local machine:

New requirements.txt contents:

Flask==0.12.1
gunicorn==19.7.1
psycopg2==2.7.7

That last line was added and interfaces with PostgreSQL.

New whatever-flask.py contents:

import os
import psycopg2
import urllib

from flask import Flask

url = urllib.parse.urlparse(os.environ.get('DATABASE_URL'))
db = "dbname=%s user=%s password=%s host=%s " % (url.path[1:], url.username, url.password, url.hostname)
conn = psycopg2.connect(db)
cur = conn.cursor()

app = Flask(__name__)

@app.route('/')
def hello():
    cur.execute('SELECT * from things')
    rows = cur.fetchall()

    response = '<pre>\n'
    for row in rows:
        response += ' -- '.join(map(str, row)) + '\n'
    response += '</pre>'
    return response

if __name__ == '__main__':
    # Bind to PORT if defined, otherwise default to 5000.
    port = int(os.environ.get('PORT', 5000))
    app.run(host='127.0.0.1', port=port)

Commit and push, should work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment