Heroku Release Phase script for managing Rails DB migrations, and playing nice with Review Apps and postdeploy scripts

00_Heroku-Release-Phase-Review-Apps-Rails_README.md

Heroku Release Phase + Review Apps + Rails

This is a simplified, but fairly thorough, set of scripts and configuration to enable Heroku Release Phase for Rails apps. Further, this particular set up plays nicely with Heroku Review Apps in that the release phase script will:

1. Fail, loudly, if the DB does not yet exist.
2. Load the DB schema if the current schema version (as determined by bin/rails db:version) is 0.
3. Run DB migrations otherwise.

For a "normal" app that usually means it will run the DB migrations. For a Review App, on the first deploy the release phase will bin/rails db:schema:load. And then the postdeploy script will seed data. During subsequent deploys to the Review App, the release phase will bin/rails db:migrate.

app.json

{
  "name": "your-app-name",
  "description": "Configuration for per-Pull-Request Reviews Apps on Heroku.",
  "scripts": {
    "postdeploy": "LOG_LEVEL=INFO bin/rake review_app:seed"
  },
  "stack": "heroku-16",
  "formation": [
    { "process": "web",    "quantity": 1 },
    { "process": "worker", "quantity": 1 }
  ],
  "addons": [
    "heroku-postgresql:hobby-dev"
  ],
  "buildpacks": [
    {
      "url": "heroku/ruby"
    }
  ],
  "env": {
    "Lots of other Env Vars...": "as you need",
    
    "HEROKU_APP_NAME": {
      "required": true
    },
    "HEROKU_PARENT_APP_NAME": {
      "required": true
    }
  }
}

bin-heroku_release.sh

#!/usr/bin/env bash
#
# Usage: bin/heroku_deploy

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
NO_COLOR='\033[0m'

set -euo pipefail

schema_version=$(bin/rails db:version | { grep "^Current version: [0-9]\\+$" || true; } | tr -s ' ' | cut -d ' ' -f3)

if [ -z "$schema_version" ]; then
  printf "💀${RED}   [Release Phase]: Database schema version could not be determined. Does the database exist?${NO_COLOR}\n"
  exit 1
fi

if [ "$schema_version" -eq "0" ]; then
  printf "\n⏳${YELLOW}   [Release Phase]: Loading the database schema.${NO_COLOR}\n"
  bin/rails db:schema:load
else
  printf "\n⏳${YELLOW}   [Release Phase]: Running database migrations.${NO_COLOR}\n"
  bin/rails db:migrate
fi

printf "\n🎉${GREEN}   [Release Phase]: Database is up to date.${NO_COLOR}\n"

Procfile

web: bundle exec puma -C ./config/puma.rb
worker: DB_POOL_SIZE=${WORKER_CONCURRENCY:-20} bundle exec sidekiq -c ${WORKER_CONCURRENCY:-20} -t ${WORKER_TIMEOUT:-25} -q default,1 -q mailers,1

# === Heroku Release Phase, ignored by `heroku local`. See: https://devcenter.heroku.com/articles/release-phase
release: bin/heroku_release

review_app.rake

# frozen_string_literal: true

namespace :review_app do
  desc 'Ensure environment is one we shish to spread seed in'
  task :ensure_review_app do
    abort 'This is not a Heroku Review App' unless review_app?
  end

  desc 'Seeds a review app with a subset of realistic-looking data'
  task :seed, [] => %w[
    ensure_review_app
    environment
    db:seed
    seed:administrator
    seed:widgets
  ] do
    Rails.logger.tagged('Seed App') { |l| l.info("Finished seeding new Review App: #{ENV['HEROKU_APP_NAME']}") }
  end

  def review_app?
    !!ENV['HEROKU_PARENT_APP_NAME']
  end
end

I have one issue that we run in quite frequently: We have two separated apps that share the same code and database. Now when we promote a new build simultaneously to both apps, we get an error sometimes because heroku runs the database migration at the same time, resulting the app deploy to fail on for the app that comes in a split second later (because the migrations are locked).
Any idea how to solve this? It might be possible to specify the app name to run the migration on, but that would be a bit static.

@pebneter My initial reaction is to dig into why you have two different apps, running the same code, sharing a database. Could you provide any context there?

Ignoring that for now, ultimately the setup you're describing is... unusual. And without more context as to the why, I can only offer generic suggestions, which might be Bad Ideas™, depending on the missing context. e.g., pick one App to be the "leader" which manages the DB, while the other is a "follower," totally dependent on the DB maintenance of the former. Something like an ENV Var to determine this is probably doable, with your Release Phase script checking that an NOOP-ing in the follower case.

Or you could look into something truly terrible like a global, distributed locking mechanism, complete with all of the sharp edges and broken corners that come with them. e.g., PostgreSQL Advisory Locks, or a Redis lock, etc...

I'm quite curious to hear more about the why, so please do share if you can!

So about the why: We are running a home automation platform (apilio.com), and we face the challenge of having to expose parts of the functionality to the public, while wanting to keep other parts restricted to certain sources to protect them from attacks.
Due to some restrictions of Heroku, we came up with an architecture that involves having two instances of the same app running as separate applications (so it's not one application scaled up, but two separate applications).

I think of the possible solutions (probably all with caution):

• Pick a "leader"
• Staged deployments: Always deploy to one first and to the other afterwards
• Make sure that the deployment script checks if a migration is already running and then wait for it
• Abandon the idea to automatically execute migration during deployments

while wanting to keep other parts restricted to certain sources to protect them from attacks

Meaning, some private networking? Heroku Private Spaces offer that kind of thing. Though it might be more complexity than is worth it for you right now depending on your needs, teams size, and run rate constraints.

But if you need to run two discrete instances, my gut says, in your case, the "pick a leader" strategy is probably the safest route. I'd go with something like an ENV var which simply says "yes, you are the leader" so that only that app actually does migrations during release phase. Then all other apps, by default, would no-op. Then you could add that magical ENV var to your Review App config so they'd still auto-migrate.

Yes, that was also my favorite strategy so far!
I guess in the long term, we will have to separate code and database changes completely, since it would also be nice to use the true no-downtime-deployment option of Heroku (preboot)

@stevenharman I really like this approach and currently using in an application of mine. One thing I'm wondering is why there is no need to run heroku restart after the db:migration has occured. Is the release phase somehow handling this for us?

After running a migration you’ll want to restart your app with heroku restart to reload the schema and pickup any schema changes.

Ref: https://devcenter.heroku.com/articles/rake

Good point. I see a warning from Rails caused by this sometimes and then have to do a manual restart.

why there is no need to run heroku restart after the db:migration has occured. Is the release phase somehow handling this for us?

That's right. The steps are:

1. Build new Slug
2. Spin up and run Release Phase Dyno using new Slug
3. Spin up new Dynos using new Slug
4. Take down old Dynos which were using the old Slug

Since the new Dynos aren't started until after Release Phase has run, they don't need to be restarted.

@pebneter My guess as to why you're seeing that warning is likely due to your particular deployment setup where you have multiple Apps pointing at a shared DB.

Since the new Dynos aren't started until after Release Phase has run, they don't need to be restarted.

Thank you @stevenharman ! That's what I was hoping happened during the release phase.

@stevenharman yes, fully agreed. It happens in my case due to the special setup I have.

Hi @stevenharman this is really helpful, thanks! Wondering if there's a reason you don't seed the database immediately after the schema load in heroku_release.sh?

👋 @eri-b. Thanks for the kind words!

Seeding the DB in so far as Heroku deploys are concerned is only done for Review Apps. And to handle that, we leverage the app.json's postdeploy script:

 "scripts": {
    "postdeploy": "LOG_LEVEL=INFO bin/rake review_app:seed"
  },

For a Review App, on the first deploy the release phase will bin/rails db:schema:load. And then the postdeploy script will seed data. During subsequent deploys to the Review App, the release phase will bin/rails db:migrate.

This ensures the seeding only happens once, only happens for Review Apps, and is consistent with how you'd seed a Review App DB without a Release Phase setup. Hope that helps. Please let me know if you have more questions.

@stevenharman I see, thanks!

This part helps clarify:

and is consistent with how you'd seed a Review App DB without a Release Phase setup

because it seemed that once you went through the trouble of ensuring the schema:load happens only once in the release phase (by checking the schema_version), why not seed the database at the same time (if it's a Review App)? Seems like packing them together helps readability a bit, but as you mention, this relies on a Release Phase setup.

I was thinking maybe it also made sense to keep the seeding out of the Release Phase, because a potential problem during seeding would cause the app creation to fail.

keep the seeding out of the Release Phase, because a potential problem during seeding would cause the app creation to fail.

That's an excellent point, and reason enough to keep schema loading and data seeding in the Release Phase and post-deploy, respectively. Thank you!

I wish Heroku would put the Release phase into app.json's scripts instead of Procfile. Feels dirty to show release phase in Procfile.

@toobulkeh I can understand that. Though since the release phase command runs in a one-off Dynoe, I also understand why it's in the Procfile.

@stevenharman is bin-heroku_release.sh saved in the bin/ directory or the root folder? Shouldn't it be named heroku_release.sh since the command in the Procfile is bin/heroku_release? I'm having trouble getting my review apps to correctly run the release phase and execute this file.

EDIT: I was finally able to get this to work. The filename has to match what is called in the Procfile release command.
i.e. file name and path: bin/heroku_release.sh, Procfile: release: bin/heroku_release.sh

The above config has the file named as bin-heroku_release.sh and then called with release: bin/heroku_release in the Procfile, this should probably be updated to reflect the correct naming. Thanks for all of this config, it's 💯!

@jcohenho Yes, sorry for the confusion. Gists only allow a flat file structure, so no directories. The actual file layout I use is:

./                            # repo root
./app.json
./Procfile
./bin/heroku_release          # be sure to `chmod +x` this file so its executable
./lib/tasks/review_app.rake

@stevenharman thanks for this gist

One issue i'm having though is in bin/heroku_release when run Heroku errors with

bin/heroku_release: 11: set: Illegal option -o pipefail

Just me?

@supairish

bin/heroku_release: 11: set: Illegal option -o pipefail

That's certainly a new one! Is your shebang line using bash? Like so: #!/usr/bin/env bash

@stevenharman yep its in there. I've opened a support ticket with Heroku, I'll post what they say and see if that flag isn't allowed anymore or something...

Just to circle back on my above problem... turned out I had a newline before the Shebang line and that was causing my issues... doh!

Is the shell script even necessary? Now app.json calls rake heroku:review_app:postdeploy and the Procfile calls rake heroku:release.

Personally, I think this has a few benefits

• Everything is in a single place making it easier to understand the order of operations
• It happens in a single rake call, cutting down on time spent calling it multiple disparate times
• It can be easily extended (For instance, we use the multi-procfile buildpack to separate review, staging, and production. They could each call their own heroku:<ENV>:release)

namespace :heroku do
  task ensure_schema: %w[environment] do
    # This determines if there's been a migration or not
    #   - `current_version` fails the same way `db:version` does if there's something seriously wrong
    #   - 0 means a newly created DB
    #   - Anything else means there have been migrations run
    Rake::Task['db:schema:load'].invoke if ActiveRecord::Base.connection.migration_context.current_version.zero?
  end
  task release: %w[environment ensure_schema db:migrate]

  namespace :review_app do
    task ensure_review_app: :environment do
      abort 'This must only be run in a review app' unless review_app?
    end

    ##
    # We're guaranteed that the schema has been loaded/migrated at this point since 'postdeploy' runs after 'release'
    # This is only run the first time an app is deployed
    #
    # For more information on the release process, see https://devcenter.heroku.com/articles/github-integration-review-apps#the-postdeploy-script
    desc 'This is run after the initial deploy (from app.json)'
    task postdeploy: %w[ensure_review_app db:seed]
  end

  def review_app?
    !!ENV['HEROKU_PARENT_APP_NAME']
  end
end

Also, for those asking about running heroku restart. I'm not sure that's strictly necessary. I did run into problems when running tasks that relied on the new DB schema (eg, %w[db:migrate db:seed]). There's no problem if you run them as discrete rake commands, since the application code is reloaded, but in the same task, you'll need something like

  ##
  # This needs to be run after db:migrate if you plan on running any other tasks in the same rake process
  #  It will force AR to re-read the columns from the DB; otherwise you'll get errors trying to access new attributes
  task reset_column_information: :environment do
    ActiveRecord::Base.descendants.each(&:reset_column_information)
  end

Interesting comment, @thefotios .
It's been a while since I have setup the bash script. Would it be worth migrating?

@thefotios A side comment. In your review_app? method, I don't have a HEROKU_PARENT_APP_NAME available in my env. Maybe this is an older stack? However, I do something very similar, I just use the presence of HEROKU_PR_NUMBER.

if ENV.has_key? "HEROKU_PR_NUMBER"

@pebneter It's up to you; they're functionally equivalent for this use case so it really all comes down to your use case and preference. We have a few more complicated things that run during our release and found it much easier to manage that in the Rakefile based approach for a few reasons

1. Everything is in Ruby already, so it just felt more natural
2. It was much easier to also tie it into our local development workflow (eg, something like rake local:setup can share the same steps)
3. It was more easily testable

@softwaregravy Thanks for pointing that out. We don't actually use that logic, I was just trying to keep it consistent with the original gist. But I think you're right that some of that changed between the "old" and "new" review apps.

@softwaregravy The HEROKU_PARENT_APP_NAME Config Var is defined as required in the app.json, meaning it inherited it from the parent app. So in the parent app (e.g., we always used our staging as the parent for Review Apps, so we'd set that Config Var there). As @thefotios said, all of this was build off of Review Apps v1 and I'm sure some things have changed in v2, but I've not updated this to reflect that.