Setup AWS GPU instance

setup_aws_gpu.sh

#!/bin/bash

####################################################################################
## AWS GPU Instance Setup Script                                                  ##
## Michelle L. Gill                                                               ##
## [email protected]                                                  ##
## Gist location: https://gist.github.com/mlgill/63114af65d9dd3e1c386b03a02c199b7 ##
## Date updated: 2017/05/27                                                       ##
####################################################################################

# BASH prompt colors
BLUE='\033[1;34m'
RED='\033[1;31m'
NC='\033[0m'

echo ""
echo -e $BLUE"############################################################################"$NC
echo -e $BLUE"# MUST use Ubuntu Server 16.04 LTS (HVM), SSD Volume Type, ami-80861296    #"$NC
echo -e $BLUE"# MUST set region to US-EAST-1A under 'Configure Instance Details, Subnet' #"$NC
echo -e $BLUE"############################################################################"$NC
echo ""
sleep 2

nvidia_cuda_version="8.0.61"
anaconda_python_version="Anaconda3-4.3.1"
anaconda_short_python_version="anaconda3"
theano_version="0.9.0"
keras_version="2.0.2"

echo ""
echo -e $BLUE"Using: nvidia_cuda_version           = $nvidia_cuda_version"$NC
echo -e $BLUE"       anaconda_python_version       = $anaconda_python_version"$NC
echo -e $BLUE"       anaconda_short_python_version = $anaconda_short_python_version"$NC
echo -e $BLUE"       theano_version                = $theano_version"$NC
echo -e $BLUE"       keras_version                 = $keras_version"$NC
echo ""
sleep 2

echo ""
echo -e $BLUE"Installing apt-get packages"$NC
echo ""
sleep 1

# ensure system is updated and has basic build tools
sudo apt-get update
#sudo apt-get --assume-yes upgrade
sudo apt-get --assume-yes install tmux build-essential gcc g++ make binutils graphviz
sudo apt-get --assume-yes install software-properties-common

mkdir downloads
cd downloads

echo ""
echo -e $BLUE"Installing cuda drivers"$NC
echo ""
sleep 1

# download and install GPU drivers
wget "http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_${nvidia_cuda_version}-1_amd64.deb" \
      -O "cuda-repo-ubuntu1604_${nvidia_cuda_version}_amd64.deb"
sudo dpkg -i cuda-repo-ubuntu1604_${nvidia_cuda_version}_amd64.deb
sudo apt-get update
sudo apt-get -y install cuda
sudo modprobe nvidia
nvidia-smi
echo "export PATH=\"/usr/local/cuda/bin:\$PATH\"" >> ~/.bashrc
source ~/.bashrc

# # install cudnn libraries
# NOTE: this is now installed by the keras-gpu package
# echo ""
# echo -e $BLUE"Installing cuDNN drivers"$NC
# echo ""

# wget "http://mlgill.co/MmyqEBGyHQ.tgz" -O "cudnn.tgz"
# tar -zxf cudnn.tgz
# cd cuda
# sudo cp lib64/* /usr/local/cuda/lib64/
# sudo cp include/* /usr/local/cuda/include/

echo ""
echo -e $BLUE"Installing Anaconda"$NC
echo ""
sleep 1

# install Anaconda for current user
if [[ ! -e "$HOME/$anaconda_short_python_version" ]]; then
    # rm -rf "$HOME/$anaconda_short_python_version"
    wget "https://repo.continuum.io/archive/${anaconda_python_version}-Linux-x86_64.sh" \
     -O "${anaconda_python_version}-Linux-x86_64.sh"
    bash "${anaconda_python_version}-Linux-x86_64.sh" -b
fi

echo "export PATH=\"$HOME/$anaconda_short_python_version/bin:\$PATH\"" >> ~/.bashrc
export PATH="$HOME/$anaconda_short_python_version/bin:$PATH"
source ~/.bashrc
conda install -y bcolz scipy numpy pandas scikit-learn matplotlib seaborn jupyter notebook gensim nltk
conda upgrade -y --all
pip install pydot pydot-ng

echo ""
echo -e $BLUE"Installing NLTK corpora"$NC
echo ""
sleep 1

# Setup NLTK corpora
python -m nltk.downloader -d $HOME/nltk_data all
echo "export NLTK_HOME=\"$HOME/nltk_data\"" >> ~/.bashrc
source ~/.bashrc

echo ""
echo -e $BLUE"Installing Keras"$NC
echo ""
sleep 1

# Install and configure keras
conda install -y keras-gpu=$keras_version
#sudo ln -s /usr/local/cuda-8.0/targets/x86_64-linux/include/* /usr/local/cuda/include/
sudo ln -s $HOME/$anaconda_short_python_version/include/cudnn.h /usr/local/cuda/include/
sudo ln -s $HOME/$anaconda_short_python_version/lib/libcudnn* /usr/local/cuda/lib64/

mkdir ~/.keras
echo '{
    "image_dim_ordering": "th",
    "epsilon": 1e-07,
    "floatx": "float32",
    "backend": "theano"
}' > ~/.keras/keras.json

# Install and configure theano
echo "[global]
device = cuda
floatX = float32
[cuda]
root = $HOME/$anaconda_short_python_version
[dnn]
enabled = True
include_path = /usr/local/cuda/include
library_path = /usr/local/cuda/lib64" > ~/.theanorc

# include_path = $HOME/$anaconda_short_python_version/include:/usr/local/cuda/include
# library_path = $HOME/$anaconda_short_python_version/lib:/usr/local/cuda/lib64" > ~/.theanorc

echo "export LIBRARY_PATH=\"/usr/local/cuda/lib64:\$LIBRARY_PATH\"" >> ~/.bashrc
source ~/.bashrc

echo ""
echo -e $BLUE"Configuring Jupyter notebook"$NC
echo ""
sleep 1

# configure jupyter
jupyter notebook --generate-config --y

cat $HOME/.jupyter/jupyter_notebook_config.py \
| sed "s/#c.NotebookApp.password = ''/c.NotebookApp.password = ''/g" \
| sed "s/#c.NotebookApp.token = '<generated>'/c.NotebookApp.token = ''/g" \
| sed "s/#c.NotebookApp.open_browser = True/c.NotebookApp.open_browser = False/g" \
> $HOME/.jupyter/jupyter_notebook_config_new.py

mv -f $HOME/.jupyter/jupyter_notebook_config_new.py $HOME/.jupyter/jupyter_notebook_config.py

echo ""
echo -e $BLUE"******************************************************"$NC
echo -e $BLUE"** NOTE: this instance MUST be rebooted before use. **"$NC
echo -e $BLUE"******************************************************"$NC
echo ""

setup_aws_gpu_instructions.md

Setting Up an AWS GPU Instance

1. Navigate to the AWS EC2 dashboard. Ensure your location is "US East, N. Virginia" in the black bar at the top of the window.
2. Create a new instance on AWS by selecting "Launch Instance" from the EC2 dashboard.
3. Select "Ubuntu Server 16.04 LTS (HVM), SSD Volume Type - ami-80861296" for the operating system. The AMI number (ami-80861296) can be entered manually under "Community AMIs" if Amazon has changed the default instance since this guide was created.
4. Select a p2.xlarge GPU compute instance.
5. Ensure the "Subnet" says 'use-east-1a'.
6. Add 128 GB of storage to the instance so there is room for your files.
7. Click through the remaining screens until you get to "Security Group". Ensure that only SSH (port 22) is open. Select "Review and Launch".
8. When the instance has launched, copy the setup script (setup_aws_gpu.sh) onto the instance, which will look something like this:

   scp -i ~/.ssh/id_rsa setup_aws_gpu.sh [email protected]:~

9. SSH onto the instance and run the setup script. It will take care of the entire install and will provide status updates in blue as it does so.
10. Once complete, the instance will have to be rebooted for the CUDA GPU drivers to be completely loaded:

    sudo shutdown -r now

11. SSH back onto the instance and it's ready for use. Note that the IP address may have changed.

This entire setup has been saved as ami-79d5996f, and can be loaded directly from AWS rather than running the above setup script. However, it's good to have the script as a reference for future endeavors.

This is fantastic work!