Using AWS EC2 spot instances with hibernation

aws_ec2_spot_hibernation.md

Following has been tested on Ubuntu 16.04 - higher versions will require similar steps

Prepare AMI

• Install hibernation supported kernel and other required utils

sudo apt install -y linux-aws-hwe
sudo apt-get install -y uswsusp
sudo apt-get install -y pm-utils

• Ensure disk space equivalent to RAM is free. Increase disk if needed
• Append nokaslr to GRUB_CMDLINE_LINUX_DEFAULT in following files
  • /etc/default/grub.d/50-cloudimg-settings.cfg
  • /etc/default/grub
  • Update grub using: update-grub; update-grub2
• Reboot once to verify if everything works correctly.
• After reboot verify nokaslr is set in boot options using cat /proc/cmdline
• Create an AMI of this instance

Launch Spot instance

• During launch select interruption mode as "hibernation"
• Add following lines to user data

#!/bin/bash
/usr/bin/enable-ec2-spot-hibernation

• After launching, connect via SSH to the system. Verify swap setup has been successful

sudo service hibagent status

• Verify swap-offset matches in following commands

sudo swap-offset /swap
cat /etc/default/grub.d/99-set-swap.cfg

• Create /etc/pm/sleep.d/10_send-message - this allows you to add run custom commands when hibernating/resuming

#!/bin/sh

# Action script ensure that unattended-upgrades is finished 
# before a hibernate 
#
# Copyright: Copyright (c) 2009 Michael Vogt
# License:   GPL-2
#

PATH=/sbin:/usr/sbin:/bin:/usr/bin

case "${1}" in
        hibernate)
                echo "Spot interrupted on $(hostname)"
                # Do something else
                ;;
        resume|thaw)
                echo "Spot restored on $(hostname)"
                # Do something else
                ;;
esac
###############

• sudo chmod a+x /etc/pm/sleep.d/10_send-message

• Enable hibernation

  • Change ExecStart in /lib/systemd/system/systemd-hibernate.service
    • ExecStart=/usr/sbin/pm-hibernate
    • This is needed as resume does not work with the default hibernate method
  • Update the service: sudo systemctl daemon-reload

• Reboot and verify if swap offset in following commands:

sudo service hibagent status
sudo swap-offset /swap
cat /etc/default/grub.d/99-set-swap.cfg
cat /proc/cmdline

• Verify nokaslr is present in cat /proc/cmdline
• If swap-offset does not match, mostly diskspace is a problem. Increase it as needed and repeat above steps.