Skip to content

Instantly share code, notes, and snippets.

@andyjeffries
Last active November 19, 2024 16:44
Show Gist options
  • Save andyjeffries/2264d5ffaf232ef1748af4edceb51590 to your computer and use it in GitHub Desktop.
Save andyjeffries/2264d5ffaf232ef1748af4edceb51590 to your computer and use it in GitHub Desktop.
kamal-issue.md

Running it once:

  INFO [9824c617] Running docker run --detach --restart unless-stopped --name some-web-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012 --network kamal --hostname 1.2.3.4-49ac32e40b0e -e KAMAL_CONTAINER_NAME="some-web-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012" -e KAMAL_VERSION="92d82dbc0250bcfa1d86bed17685f67dc0c87012" --env RAILS_ENV="staging" --env-file .kamal/apps/some-website-staging/env/roles/web.env --log-opt max-size="10m" --volume /storage:/app/storage --volume $(pwd)/.kamal/apps/some-website-staging/assets/volumes/web-92d82dbc0250bcfa1d86bed17685f67dc0c87012:/app/public/assets --label service="some-website" --label role="web" --label destination="staging" example/repo:92d82dbc0250bcfa1d86bed17685f67dc0c87012 on 1.2.3.4
  INFO [9824c617] Finished in 0.785 seconds with exit status 0 (successful).
  INFO [03e088fc] Running docker container ls --all --filter name=^some-web-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012$ --quiet on 1.2.3.4
  INFO [03e088fc] Finished in 0.122 seconds with exit status 0 (successful).
  INFO [fdadabb6] Running docker exec kamal-proxy kamal-proxy deploy some-web-process --target="041b991b033d:80" --host="staging.example.com" --tls --deploy-timeout="30s" --drain-timeout="30s" --buffer-requests --buffer-responses --log-request-header="Cache-Control" --log-request-header="Last-Modified" --log-request-header="User-Agent" on 1.2.3.4
 ERROR Failed to boot web on 1.2.3.4
  INFO First web container is unhealthy on 1.2.3.4, not booting any other roles
  INFO [2d4322d6] Running docker container ls --all --filter name=^some-web-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012$ --quiet | xargs docker logs --timestamps 2>&1 on 1.2.3.4
  INFO [2d4322d6] Finished in 0.141 seconds with exit status 0 (successful).
 ERROR 2024-11-19T16:33:04.345976401Z Preparing database...
2024-11-19T16:33:28.148046954Z Migrating to ChangeSolidQueueRecurringTasksStaticToNotNull (20241112163631)
2024-11-19T16:33:28.148053486Z == 20241112163631 ChangeSolidQueueRecurringTasksStaticToNotNull: migrating ====
2024-11-19T16:33:28.148059957Z -- change_column_null(:solid_queue_recurring_tasks, :static, false, true)
2024-11-19T16:33:28.148067478Z    (0.8ms)  UPDATE `solid_queue_recurring_tasks` SET `static`=TRUE WHERE `static` IS NULL
2024-11-19T16:33:28.148075016Z    (102.3ms)  ALTER TABLE `solid_queue_recurring_tasks` CHANGE `static` `static` tinyint(1) DEFAULT TRUE NOT NULL
2024-11-19T16:33:28.148082927Z    -> 0.1196s
2024-11-19T16:33:28.148089155Z == 20241112163631 ChangeSolidQueueRecurringTasksStaticToNotNull: migrated (0.1197s)
2024-11-19T16:33:28.148096184Z
2024-11-19T16:33:28.148102192Z   ActiveRecord::SchemaMigration Create (3.1ms)  INSERT INTO `schema_migrations` (`version`) VALUES ('20241112163631') RETURNING `version`
2024-11-19T16:33:28.148109922Z   SQL (0.6ms)  SELECT RELEASE_LOCK('854892912238153740')
2024-11-19T16:33:28.213839089Z Migrations finished!
2024-11-19T16:33:30.585219324Z => Booting Puma
2024-11-19T16:33:30.585274647Z => Rails 8.0.0 application starting in staging
2024-11-19T16:33:30.585284970Z => Run `bin/rails server --help` for more startup options
  INFO [3ab882dc] Running docker container ls --all --filter name=^some-web-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012$ --quiet | xargs docker inspect --format '{{json .State.Health}}' on 1.2.3.4
  INFO [3ab882dc] Finished in 0.147 seconds with exit status 0 (successful).
 ERROR null
  INFO [3a3dc019] Running docker container ls --all --filter name=^some-web-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012$ --quiet | xargs docker stop on 1.2.3.4
  INFO [3a3dc019] Finished in 12.695 seconds with exit status 0 (successful).
Releasing the deploy lock...
  Finished all in 1211.4 seconds
  ERROR (SSHKit::Command::Failed): Exception while executing on host 1.2.3.4: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy

Running it again (with no changes, just literally kamal deploy -d staging again):

  INFO [72eba711] Running docker run --detach --restart unless-stopped --name some-web-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012 --network kamal --hostname 1.2.3.4-5ebe6e12730d -e KAMAL_CONTAINER_NAME="some-web-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012" -e KAMAL_VERSION="92d82dbc0250bcfa1d86bed17685f67dc0c87012" --env RAILS_ENV="staging" --env-file .kamal/apps/some-website-staging/env/roles/web.env --log-opt max-size="10m" --volume /storage:/app/storage --volume $(pwd)/.kamal/apps/some-website-staging/assets/volumes/web-92d82dbc0250bcfa1d86bed17685f67dc0c87012:/app/public/assets --label service="some-website" --label role="web" --label destination="staging" example/repo:92d82dbc0250bcfa1d86bed17685f67dc0c87012 on 1.2.3.4
  INFO [72eba711] Finished in 0.597 seconds with exit status 0 (successful).
  INFO [0705af3c] Running docker container ls --all --filter name=^some-web-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012$ --quiet on 1.2.3.4
  INFO [0705af3c] Finished in 0.127 seconds with exit status 0 (successful).
  INFO [d01b5057] Running docker exec kamal-proxy kamal-proxy deploy some-web-process --target="26e2c5fd3632:80" --host="staging.example.com" --tls --deploy-timeout="30s" --drain-timeout="30s" --buffer-requests --buffer-responses --log-request-header="Cache-Control" --log-request-header="Last-Modified" --log-request-header="User-Agent" on 1.2.3.4
  INFO [d01b5057] Finished in 29.253 seconds with exit status 0 (successful).
  INFO First web container is healthy on 1.2.3.4, booting any other roles
  INFO [c83b27b8] Running docker container ls --all --filter name=^some-web-process-6de88e986e0166d52cd33b8f2267109bcaf18706$ --quiet | xargs docker stop on 1.2.3.4
  INFO [c83b27b8] Finished in 12.604 seconds with exit status 0 (successful).
  INFO [9c24391a] Running /usr/bin/env find .kamal/apps/some-website-staging/assets/extracted -maxdepth 1 -name 'web-*' ! -name web-92d82dbc0250bcfa1d86bed17685f67dc0c87012 -exec rm -rf "{}" + ; find .kamal/apps/some-website-staging/assets/volumes -maxdepth 1 -name 'web-*' ! -name web-92d82dbc0250bcfa1d86bed17685f67dc0c87012 -exec rm -rf "{}" + on 1.2.3.4
  INFO [9c24391a] Finished in 0.152 seconds with exit status 0 (successful).
  INFO [d0c68f09] Running docker container ls --all --filter name=^some-website-job-staging-92d82dbc0250bcfa1d86bed17685f67dc0c87012$ --quiet on 1.2.3.4
  INFO [d0c68f09] Finished in 0.141 seconds with exit status 0 (successful).
  INFO [3b8e456f] Running docker run --detach --restart unless-stopped --name some-job-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012 --network kamal --hostname 1.2.3.4-ff99e19a3d1e -e KAMAL_CONTAINER_NAME="some-job-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012" -e KAMAL_VERSION="92d82dbc0250bcfa1d86bed17685f67dc0c87012" --env RAILS_ENV="staging" --env-file .kamal/apps/some-website-staging/env/roles/job.env --log-opt max-size="10m" --volume /storage:/app/storage --label service="some-website" --label role="job" --label destination="staging" example/repo:92d82dbc0250bcfa1d86bed17685f67dc0c87012 bundle exec rake solid_queue:start on 1.2.3.4
  INFO [3b8e456f] Finished in 0.670 seconds with exit status 0 (successful).
  INFO [da1a4311] Running docker container ls --all --filter name=^some-job-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 1.2.3.4
  INFO [da1a4311] Finished in 0.158 seconds with exit status 0 (successful).
  INFO Container is running, waiting for readiness delay of 7 seconds
  INFO [fdde3a28] Running docker container ls --all --filter name=^some-job-process-92d82dbc0250bcfa1d86bed17685f67dc0c87012$ --quiet | xargs docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' on 1.2.3.4
  INFO [fdde3a28] Finished in 0.110 seconds with exit status 0 (successful).
  INFO Container is healthy!
  INFO [9746814a] Running docker container ls --all --filter name=^some-job-process-6de88e986e0166d52cd33b8f2267109bcaf18706$ --quiet | xargs docker stop -t 30 on 1.2.3.4
  INFO [9746814a] Finished in 12.628 seconds with exit status 0 (successful).
  INFO [94ff4444] Running docker tag example/repo:92d82dbc0250bcfa1d86bed17685f67dc0c87012 example/repo:latest-staging on 1.2.3.4
  INFO [94ff4444] Finished in 0.161 seconds with exit status 0 (successful).
Prune old containers and images...
  INFO [ce6e1be4] Running docker ps -q -a --filter label=service=some-website --filter status=created --filter status=exited --filter status=dead | tail -n +6 | while read container_id; do docker rm $container_id; done on 1.2.3.4
  INFO [ce6e1be4] Finished in 2.600 seconds with exit status 0 (successful).
  INFO [79a52def] Running docker image prune --force --filter label=service=some-website on 1.2.3.4
  INFO [79a52def] Finished in 0.163 seconds with exit status 0 (successful).
  INFO [6dbf34e6] Running docker image ls --filter label=service=some-website --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w "$(docker container ls -a --format '{{.Image}}\|' --filter label=service=some-website | tr -d '\n')example/repo:latest-staging\|example/repo:<none>" | while read image tag; do docker rmi $tag; done on 1.2.3.4
  INFO [6dbf34e6] Finished in 3.862 seconds with exit status 0 (successful).
Releasing the deploy lock...
  Finished all in 102.6 seconds

The entrypoint is:

#!/bin/bash -e

# If running the rails server then create or migrate existing database
if [ "${1}" == "./bin/rails" ] && [ "${2}" == "server" ]; then
  echo "Preparing database..."
  ./bin/rails db:prepare
  echo "Migrations finished!"
fi

exec "${@}"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment