Adding Rack-Based Health Checks Without curl or wget

Image: Lake View Cemetary, Cleveland OH by Wendy Bayer

You can not address a problem without first knowing that there is a problem. Having health checks in your application allows you to easily determine and monitor the status of your critical application or service and in many cases have it automatically self heal.

Kubernetes, docker compose, and many other container orchestration services rely on health checks to know when application services are alive and ready to serve traffic.

Additionally you can connect your website monitoring services to ensure that you know when your end users are not able to connect to your web-based services.

Here you will learn how to create, use, and test both a liveness and readiness health check in your Ruby Rack-based application without the need for curl or wget. While this post uses Rails and docker compose, this approach can translate to other Rack-based (Ruby) frameworks and orchestration services such as Kubernetes.

Overview of Health Checks

In general (and per Kubernetes) there are three basic types of health checks (or probes):

Liveness check which simply determines that the application is running (i.e. alive)
Readiness check which determines that the application is ready and capable of serving traffic/end users which often means that all of the application's dependencies are up and successfully connected and initialized (e.g. database, redis)
Startup check which determines that a long initializing application has successfully started

This post will focus on the two most common ones which are liveness and readiness as many applications can use their readiness check as a startup check.

/livez and /readyz

There is a convention of naming the health check endpoints /livez and /readyz which came from Google and its practices of standard "z-pages". The 'z' is added to avoid any existing name clashes. When engineers left Google, they took these conventions with them so it has somewhat spread throughout the IT industry.

👀 I learned this etymological trivia from Stack Overflow

Liveness and Readiness in Rails

Although this is somewhat specific to Rails, this section should also help to illustrate the practical difference between liveness and readiness.

Liveness

Generally for a Rails application, liveness should not depend on the state of dependent systems such as the application's database, redis, external APIs, etc. This is true even though a Rails application will not return non-error responses if it can not connect to the database or if there are pending migrations. Consider that in Kubernetes or docker compose/Swarm orchestration, the general mechanism of "self-healing" an application/service is to kill and restart the container. If the issue is with the dependency like the database, repeatedly killing and restarting the Rails application container is not going to help and just adds to the churn. Thus a liveness endpoint in Rails should simply return 200 (OK) if the application (more specifically the application server e.g. Puma) is running.

Readiness

Since readiness means that the application can provide its primary function i.e. serve traffic, it should depend on the state of its critical dependencies. Thus a Rails readiness endpoint often checks the database connection, redis connection, pending migrations, etc. and only returns 200 (OK) if all of those dependencies are available.

Implementing the Health Checks in Rack

At the time of this post (Rails 7.0.4.3), Rails is actually including a basic liveness health check /up in the next version (7.1?). Being a liveness check, this does not include the state of the database or other dependencies. Here is the Pull Request.

While this seems helpful, it is probably best if the health checks, especially liveness, are at a lower layer of the Rails middleware and in Rack itself. This excellent post by Nick Malcolm does a great job in explaining this.

Implementing the Liveness Check

Here is an example of a Rack-based liveness check. For Rails, you can put this in file lib/rack/live_check.rb...

# frozen_string_literal: true

module Rack
  # Most basic health check that simply indicates that the application is up
  # but not necessarily ready for traffic
  class LiveCheck
    def call(_env)
      # Must be alive to reach this
      [
        200,
        { 'Content-type' => 'application/json; charset=utf-8' },
        ['{ "status": 200, "message": "alive" }']
      ]
    end
  end
end

This example returns a basic informational JSON body, but you can customize it to suit your needs.

Configuring the Liveness Check at `/livez`

You will need to configure the liveness check in Rack, mapping it to the endpoint get /livez.

In Rails, you can add the following lines to the Rack configuration file config.ru...

require_relative 'lib/rack/live_check'
...

map '/livez' do
  run Rack::LiveCheck.new
end

Implementing the Readiness Check

Here is an example of a Rack-based readiness check. For Rails, you can put this in file lib/rack/ready_check.rb...

# frozen_string_literal: true

module Rack
  # Health check to ensure the database is ready for traffic
  class ReadyCheck
    def initialize
      @body_details = {}
    end

    def call(_env)
      response = ready? ? ready_response : error_response
      response.finish
    end

    def ready?
      database_connected?
      database_migrations?
    end

    def database_connected?
      ActiveRecord::Base.connection.execute('SELECT 1')
      @body_details[:database_connection] = 'ok'
    rescue StandardError
      @body_details[:database_connection] = 'error'
      false
    end

    def database_migrations?
      ActiveRecord::Migration.check_pending!
      @body_details[:database_migrations] = 'ok'
    rescue StandardError
      @body_details[:database_migrations] = 'pending'
      false
    end

    def ready_response
      status = 200
      response_body = { status:, message: 'ready' }.merge(@body_details)
      rack_response(response_body, status)
    end

    def error_response
      status = 503
      response_body = { status:, message: 'error' }.merge(@body_details)
      rack_response(response_body, status)
    end

    def rack_response(body, status)
      Rack::Response.new(
        body.to_json,
        status,
        response_header
      )
    end

    def response_header
      { 'Content-type' => 'application/json; charset=utf-8' }
    end
  end
end

This example checks the database connection and if there are pending migrations, but you can modify it to suit your needs, for instance by adding checks for redis and/or other dependencies or removing the pending migrations check. It also returns an informational JSON body which can be modified.

👀 The documentation of Rails health check gems like the shlima health-bit gem are a good source for how to check the status of common Rails dependencies

Configuring the Liveness Check at `/readyz`

Again, you will need to configure the readiness check in Rack, mapping it to the endpoint get /readyz.

In Rails, you can add the following lines to the Rack configuration file config.ru...

require_relative 'lib/rack/ready_check'
...

map '/readyz' do
  run Rack::ReadyCheck.new
end

Running Your Application Server

Now if you run your application server with the added health checks (e.g. bundle exec bin/rails server -p 3000 -b 0.0.0.0) and go to the /livez endpoint (e.g. http://localhost:3000/livez) using your browser, Postman, or curl, you should get a 200 response code and a body that looks something like this...

{ "status": 200, "message": "alive" }

Similarly for the /readyz endpoint (e.g. http://localhost:3000/readyz), you should get the 200 response code and a body that looks something like this...

{"status":200,"message":"ready","database_connection":"ok","database_migrations":"ok"}

CORS Considerations

If you are enabling Cross Origin Resource Sharing (CORS) in your Rails or other Rack-based application, be sure to configure CORS to be positioned above your health checks in the Rack middleware. For example in Rails, in the config/initializers/cors.rb initializer...

Rails.application.config.middleware.insert_before 0, Rack::Cors do
  allow do
     ...

Note the .middleware.insert_before 0 which configures CORS at the top (0) position.

👀 For more information, see Positioning in the Middleware Stack in the rack-cors documentation

Giving Credit Where Credit Is Due

The implementations for these Rack-based health checks are inspired and derived from the post Simple health-check for Ruby on Rails by Alejandro Abarca Rodríguez.

Writing a Simple Ruby Program To Check the Readiness

In a container orchestration system such as Kubernetes or docker compose, you need a way to know if your application is up and ready. For instance, in my Continuous Integration I use a docker compose framework to run E2E tests in a container against my application container and need to know that the application is ready before running the tests.

Most of the docker compose health check examples use curl to check the readiness endpoint, but you may not have curl available in your application's deployment/production image for security and/or image size reasons. Since you are running a Rack-based web application (e.g. Rails), you necessarily have Ruby available, so why not use that?

🤔 You can always convert your curl commands to Ruby using this awesome site https://jhawthorn.github.io/curl-to-ruby/

Here is a simple ruby program that checks your "z" endpoints (with /readyz the default) and returns a 0 return code (Unix standard for success) if the checked endpoint returns a 200 return code...

#!/usr/bin/env ruby
# ----------------------------------------------------------------------
# Simple script to query healthchecks of locally running Ruby
#  applications. Generally intended as a healthchecks for
# container orchestration so curl/wget are not needed
# ----------------------------------------------------------------------
require 'net/http'
require 'uri'

def z_endpoint(endpoint)
  puts "[#{endpoint}]"
  host = ENV.fetch('HOST', 'localhost')
  port = ENV.fetch('PORT', 3000)
  "http://#{host}:#{port}/#{endpoint}z"
end

endpoint = ARGV.shift || 'ready'
healthcheck = URI.parse(z_endpoint(endpoint))

begin
  response = Net::HTTP.get_response(healthcheck)
rescue StandardError
  exit(1)
end

puts "Healthcheck [#{healthcheck}] returned code: [#{response.code}] body: [#{response.body}]"
exit_code = (response.code == '200') ? 0 : 1
exit(exit_code)

You can put this program in file app_is in you application's root directory. This name is influenced by the PostgreSQL pg_isready health check. Be sure to set execute permissions e.g. chmod a+x app_is on this file.

Note that when running these health checks in a container orchestration system such as Kubernetes or docker compose, the checks run in the application container itself so the host islocalhost.

💡 You can use this same approach for your liveness health check as well

Configuring the Readiness Health Check in Docker Compose

Although this example is for docker compose, it is translatable to other container orchestration system such as Kubernetes.

Here is an example of using the Ruby app_is program in a docker-compose.yml file for the application...

services:
  app:
    healthcheck:
      test: ["CMD", "bash", "-c", "ruby", "app_is"]
      start_period: 10s
      interval: 10s
      timeout: 5s
      retries: 10

👉 Note the test: line, this is very important. This is the only way that I could get this to work. This executes the bash command with the -c option to run the ruby command with the app_is program/script. Running just the app_is program even with the "shebang" line at the top did not work.

You should adjust your timing configurations (e.g. start_period, interval, etc.) to suit your application's startup characteristics.

Testing the Health Check Endpoints

Now that you have implemented your health checks, you should ensure that they continue to work as you make changes to your application by adding automated tests.

Because the health checks are implemented in the Rack middleware and not in the Rails applications, you will not be able to test them with controller or request/integration specs/tests. Since they are simple API endpoints, they are easy enough to test in your End-to-End (E2E) tests.

Here is an example using the Faraday gem with RSpec and a custom RSpec matcher for the health check response.

Implementing the Tests

In file spec/health_checks_spec.rb...

# frozen_string_literal: true

require 'spec_helper'

RSpec.describe 'Health Checks' do

  describe '/livez' do
    subject(:livez_request) { app_connection.get(endpoint_url('/livez')) }

    it { expect(livez_request).to be_request_response(200, livez_response) }
  end

  describe '/readyz' do
    subject(:readyz_request) { app_connection.get(endpoint_url('/readyz')) }

    it { expect(readyz_request).to be_request_response(200, readyz_response) }
  end

  private

  def livez_response
    {
      'status' => 200,
      'message' => 'alive'
    }
  end

    def readyz_response
    {
      'status' => 200,
      'message' => 'ready',
      'database_connection' => 'ok',
      'database_migrations' => 'ok'
    }
  end

  def app_connection
    Faraday.new(url: app_base_url) do |conn|
      conn.request :json
      conn.response :json
    end
  end

  def endpoint_url(path)
    "#{app_base_url}#{path}"
  end

  def app_base_url
    ENV.fetch('E2E_BASE_URL')
  end
end

Implementing the Custom Response Matcher

Although it is not necessary, implementing a custom RSpec matcher allows you to test and see all of the response at once which is helpful for debugging when the tests fail.

In file spec/support/matchers/be_request_response.rb...

# frozen_string_literal: true

# Custom RSpec Matcher to match an
# expected request response
module BeRequestResponse
  class BeRequestResponse
    def initialize(status, body)
      @status = status
      @body = body
    end

    def matches?(actual)
      @actual_status = actual.status
      @actual_body = actual.body

      @actual_status == @status &&
        @actual_body == @body
    end

    def failure_message
      "expected that actual status code [#{@actual_status}] would match " \
        "expected status code [#{@status}] and that actual body " \
        "[#{pretty(@actual_body)}] would match expected body " \
        "[#{pretty(@body)}]"
    end

    def failure_message_when_negated
      "expected that actual status code [#{@actual_status}] would not match " \
        "expected status code [#{@status}] and that actual body " \
        "[#{pretty(@actual_body)}] would match expected body " \
        "[#{pretty(@body)}]"
    end

    def description
      "have response status code [#{@status}] and body [#{@body}]"
    end

    private

    def pretty(json)
      JSON.pretty_generate(json)
    end
  end

  def be_request_response(status, body)
    BeRequestResponse.new(status, body)
  end
end

# Include the custom matcher in RSpec
RSpec.configure do |config|
  config.include BeRequestResponse
end

Adding Faraday and Your Custom Matcher to `spec_helper.rb`

Finally, add Faraday and your custom RSpec matcher to spec/spec_helper.rb...

...
# --- CUSTOM ADDITIONAL CONFIGURATION ---
require 'faraday'
require_relative 'support/matchers/be_request_response'

Running the Tests Against the Server With Healthchecks

Now if you run your application server with the health checks (e.g. bundle exec bin/rails server -p 3000 -b 0.0.0.0) and then run your E2E tests supplying the E2E_BASE_URL environment variable set to your running server (e.g. E2E_BASE_URL=http://localhost:3000 bundle exec rspec --format documentation), your output should look something like the following

Health Checks
  /livez
    is expected to have response status code [200] and body [{"status"=>200, "message"=>"alive"}]
  /readyz
    is expected to have response status code [200] and body [{"status"=>200, "message"=>"ready", "database_connection"=>"ok", "database_migrations"=>"ok"}]

Finished in 0.08437 seconds (files took 0.43692 seconds to load)
2 examples, 0 failures

Conclusion

And that's it. Now you have added liveness and readiness health checks to your application and a custom Ruby program to test if your app is ready along with the automated E2E tests to ensure your health checks are working.

brianjbayer/gist-curlless-rack-based-healthchecks.md