detect all ephemeral disks on EC2 then stripe together in a raid-0 vol mounted at /mnt

raid_ephemeral.sh

#!/bin/bash
#
# this script will attempt to detect any ephemeral drives on an EC2 node and create a RAID-0 stripe
# mounted at /mnt. It should be run early on the first boot of the system.
#
# Beware, This script is NOT fully idempotent.
#

METADATA_URL_BASE="http://169.254.169.254/2012-01-12"

yum -y -d0 install mdadm curl

# Configure Raid - take into account xvdb or sdb
root_drive=`df -h | grep -v grep | awk 'NR==2{print $1}'`

if [ "$root_drive" == "/dev/xvda1" ]; then
  echo "Detected 'xvd' drive naming scheme (root: $root_drive)"
  DRIVE_SCHEME='xvd'
else
  echo "Detected 'sd' drive naming scheme (root: $root_drive)"
  DRIVE_SCHEME='sd'
fi

# figure out how many ephemerals we have by querying the metadata API, and then:
#  - convert the drive name returned from the API to the hosts DRIVE_SCHEME, if necessary
#  - verify a matching device is available in /dev/
drives=""
ephemeral_count=0
ephemerals=$(curl --silent $METADATA_URL_BASE/meta-data/block-device-mapping/ | grep ephemeral)
for e in $ephemerals; do
  echo "Probing $e .."
  device_name=$(curl --silent $METADATA_URL_BASE/meta-data/block-device-mapping/$e)
  # might have to convert 'sdb' -> 'xvdb'
  device_name=$(echo $device_name | sed "s/sd/$DRIVE_SCHEME/")
  device_path="/dev/$device_name"

  # test that the device actually exists since you can request more ephemeral drives than are available
  # for an instance type and the meta-data API will happily tell you it exists when it really does not.
  if [ -b $device_path ]; then
    echo "Detected ephemeral disk: $device_path"
    drives="$drives $device_path"
    ephemeral_count=$((ephemeral_count + 1 ))
  else
    echo "Ephemeral disk $e, $device_path is not present. skipping"
  fi
done

if [ "$ephemeral_count" = 0 ]; then
  echo "No ephemeral disk detected. exiting"
  exit 0
fi

# ephemeral0 is typically mounted for us already. umount it here
umount /mnt

# overwrite first few blocks in case there is a filesystem, otherwise mdadm will prompt for input
for drive in $drives; do
  dd if=/dev/zero of=$drive bs=4096 count=1024
done

partprobe
mdadm --create --verbose /dev/md0 --level=0 -c256 --raid-devices=$ephemeral_count $drives
echo DEVICE $drives | tee /etc/mdadm.conf
mdadm --detail --scan | tee -a /etc/mdadm.conf
blockdev --setra 65536 /dev/md0
mkfs -t ext3 /dev/md0
mount -t ext3 -o noatime /dev/md0 /mnt

# Remove xvdb/sdb from fstab
chmod 777 /etc/fstab
sed -i "/${DRIVE_SCHEME}b/d" /etc/fstab

# Make raid appear on reboot
echo "/dev/md0 /mnt ext3 noatime 0 0" | tee -a /etc/fstab

thanks

Ya this was awesome thank you

Awesome script. One thing to add:
mdadm --create --verbose /dev/md0 --level=0 -c256 --raid-devices=$ephemeral_count $drives

May given a warning if you only have 1 ephemeral drive, just add --force before --raid-devices.

Awesome. Thanks, you've just saved me a day.

It has critical issue when reboot. I saw this message in EC2 system log. and I cannot connect this instance.
( c3.xlarge, Ubuntu 14.04 )

 * Starting enable remaining boot-time encrypted block devices�[74G[ OK ]
The disk drive for /mnt is not ready yet or not present.
keys:Continue to wait, or Press S to skip mounting or M for manual recovery

I think that needs to add in /etc/fstab the ‘nobootwait’ option.
http://askubuntu.com/questions/120/how-do-i-avoid-the-s-to-skip-message-on-boot
But it can only continue boot process, not fix (reformat md0 device). I have no idea to fix on boot time, but it really needed.

I'm curious, why bother RAID-ing the ephemeral devices together to mount /foo instead of simply using them as swap and mounting the same /foo as tmpfs? One would think, the swap-handling code in kernel is much more mature than the RAID0...

Thanks hey-jude! I had the same issue with my ephemeral RAID0. Adding the nobootwait option in fstab fixed the issue for me too.

For some images I'm getting 'udev' error due to the awk command hitting the first line looking for root. It looks like the root drive is the 3rd line down for some images, so I changed the script as follows:
root_drive=df -h | grep -v grep | awk 'NR==4{print $1}'`
which was finally finding the correct root partition and the script is working as intended. I'm using this as a user_data snippet on boot.

I think the issue of the drive not being ready at boot is because there is a difference in the state of the ephemeral drives on a reboot versus a stop and then start. You get new unformatted drives when using a stop then restart but you get the same drives on a reboot. I'm trying to figure out how to detect that on boot and run the mkfs if that is the case.

Thanks! Was getting 'resource busy' at create RAID step but your umounting of the cloud-config mounted ephemeral drive solved my problem.

If you trust the docs, there's a much easier way to find all the ephemeral volumes:

find /dev -regextype posix-egrep -regex '.*/(sd|xvd)[b-e]'

On Ubuntu 14.04

root_drive=df -h | grep -v grep | awk 'NR==2{print $1}'`

was pulling the first drive, not necessarily the root drive, for me until I changed it to

root_drive=df -h | grep -P '^.*/$' | awk 'NF==6{print $1}'`.

On Ubuntu 16.04 I am using this to get the ephemeral volumes:

root@ip-10-228-67-215:~# lsblk -I 202 -d -n --output NAME,MAJ:MIN | grep -v "202:0" | awk '{print $1}'
xvdf
xvdg

Note that Ubuntu 16.04+ uses systemd and it does not support the nobootwait mount option; use nofail instead.

Great script. Saved me some time. thanks! - I did add some code for this to run efficiently as a startup script... details below. Cheers! - Joseph P.

PS: The correct startup log on EC2 (As of this date) is: "/var/log/boot.log"

Additions:

1. created as 'startup' script on redhat / centos
2. added code to verify script is running on startup
3. added code so that this script will not execute on reboot if the raid disk is already mounted. (In case of a shutdown it will typically execute as ephemeral disks are wiped out.)
4. added code to prevent /etc/fstab mount point entry from being written multiple times.

additional code

#######  To use as startup script  ########
### 1. Copy script to directory: /opt/aws/
# mkdir -p /opt/aws/
# cp 'myscript.sh' /opt/aws/
# chmod 755 /opt/aws/myscript.sh
# 
### 2: Add script path to 'rc.local' to execute at startup:
# echo "/opt/aws/myscript.sh" | tee -a /etc/rc.d/rc.local
#
### Intended use: 
# EC2 reboot: raid volume and data will be persistent on "/dev/md0"
# EC2 shutdown: all ephemeral storage is wiped. This script will initialize all instance stores and mount raid disk on boot.
#

# checksum to verify script is executing at reboot
DATE=$(date +'%F %H:%M:%S')
DIR=/tmp
echo "Current date and time: $DATE" > $DIR/ephem_bootscript_lastrun.txt

# check if Raid 0 disk is mounted | IF mounted then exit - if not continue
if grep '/dev/md0' /etc/mtab > /dev/null 2>&1; then
echo "'/dev/md0' is mounted...exiting" && exit
else
echo "'/dev/md0' not mounted"
echo "Continuing..."
fi

# Make raid volume mount on reboot (Prevent writing entry to /etc/fstab multiple times)
MP=/yourmountpoint
MOUNT_PATH="/dev/md0 ${MP} ext3 noatime 0 0"
FILE="/etc/fstab"
grep -qF "$MOUNT_PATH" "$FILE" || echo "$MOUNT_PATH" >> "$FILE"