Download, Build and Deploy MPICH 3.1 and PLDA 3.1 on Computers in a Cluster

install_mpich_and_plda.bash

# This script download, build and deploy MPICH 3.1
# (http://www.mpich.org) and pLDA 3.1 (https://code.google.com/p/plda)
# to computers in a cluster.
#
# Usage:
#
# You download and run this script from any directory.  For example,
# consider that Alice is running this script to install MPICH and pLDA
# on David's workstation in the role of Steven, she should do:
#
#  alice:> install_mpich_and_plda.sh [email protected]
#
# This script then
#
#  1. downloads MPICH and pLDA to the working directory of Alice
#  2. scp packages to Steven's home directory on david.workstation.com
#  3. unpack packages and build them at
#     david.workstation.com:/$HOME_OF_STEVEN/
#  4. install MPICH to david.workstation.com:$HOME_OF_STEVEN/usr
#
# Prerequistics:
#
# In above case, Alice must be able to log in to
# [email protected] in a passwordless way.  Or, Alice must
# has her public key enlisted in Steven's ~/.ssh/authorized_keys file.
# To do so, Alice can run the following command:
#
#  alice:> cat ~/.ssh/id_rsa.pub | ssh [email protected]:~/.ssh/authorized_keys
#
# where ~/.ssh/id_rsa.pub is Alice's public key.  If Alice does not
# yet has this file, she could run the following command to generate a
# pair of private/public keys:
#
#  alice:> ssh-keygen -t rsa
#
# Install on Multiple Nodes:
#
# Alice might want to deploy MPICH and pLDA on more than one computers
# for parallel training of pLDA.  Suppose she also want to use Bob's
# computer (still in the role of Steven), she can do:
#
#  alice:> install_mpich_and_plda.sh [email protected]
#
# Run pLDA using MPI
#
# Suppose that Alice wants to start a parallel pLDA training job using
# 5 processes running two computers (david.workstation.com and
# bob.workstation.com), she should login to either computer, say
# Steven's:
#
#  alice:> ssh [email protected]
#  steven@david:> ~/usr/bin/mpiexec -n 5 -f ~/hosts \
#                 ~/plda/mpi_lda \
#                 --num_topics 2 --alpha 0.1 --beta 0.01 \
#                 --training_data_file ~/plda/testdata/test_data.txt \
#                 --model_file /tmp/lda_model.txt \
#                 --total_iterations 150
#
# where the flag '-f ~/hosts' refers to a text file on
# david.workstation.com:~/hosts, which is a text file whose each line
# is a computer name.  In this case, the content of ~/hosts should be:
#
#  david.workstation.com
#  bob.wordstation.com
#
# Also notice that, before you start the training process, you must
# copy the input file (training data), in the case,
# ~/plda/testdata/test_data.txt, to all computers invoked.

mpich_pkg_name=mpich-3.1
plda_pkg_name=plda
mpich_pkg=$mpich_pkg_name.tar.gz
plda_pkg=$plda_pkg_name-3.1.tar.gz
mpich_remote=http://www.mpich.org/static/downloads/3.1/mpich-3.1.tar.gz
plda_remote=https://plda.googlecode.com/files/plda-3.1.tar.gz

# CheckPrerequisits wyi@u64a checks whether g++ and make were
# installed.  It must be OK to do passwordless ssh to wyi@u64a.
function CheckPrerequisits() {
    if [ "$(ssh $1 'which g++')" = "" ]; then
        echo "No g++ installed on $1"
        exit
    fi

    if [ "$(ssh $1 'which make')" = "" ]; then
        echo "No make installed on $1"
        exit
    fi
}

# FindOrDownloadPackage ~/a.txt http://somewhere.com/a.txt checks if
# the file ~/a.txt exists; if not, it downloads
# http://somewhere.com/a.txt and save it as ~/a.txt.
function FindOrDownloadPackage() {
    if [ -f $1 ]; then
	echo "Found MPICH pakcage: $1"
    else
	echo "Cannot find $1. Download it ..."
	wget "$2" -O $1
	if [ $? -eq 0 ]; then
	    echo "Done."
	else
	    echo "Failed downloading $2.  Exit."
	    exit
	fi
    fi
}

# CopyPackageToTarget ~/a.tgz [email protected] copies local
# file ~/a.tgz to [email protected]:~/a.tgz.
function CopyPackageToTarget() {
    echo "Copy $1 to target computer ..."
    scp $1 $2:~/
    if [ $? -eq 0 ]; then
	echo "Done."
    else
	echo "Failed."
	exit
    fi
}

if [ $# != 1 ]; then
    echo "Usage: install_mpich_and_plda.sh <target>"
    echo "  where <target> has the form [email protected] or computer.com."
    exit
fi

target=$1
echo "Installing to computer $target"

CheckPrerequisits $target

FindOrDownloadPackage ~/$mpich_pkg $mpich_remote
FindOrDownloadPackage ~/$plda_pkg $plda_remote

CopyPackageToTarget ~/$mpich_pkg $target
CopyPackageToTarget ~/$plda_pkg $target

echo "Copy MPICH installation script."
target_home=$(ssh $target 'pwd ~')
cat | ssh $target 'cat > ~/install_mpich.sh' <<endmsg
cd ~
rm -rf $mpich_pkg_name
tar xzf $mpich_pkg
cd $mpich_pkg_name
./configure --disable-f77 --disable-fc --enable-static --disable-shared -prefix=$target_home/usr && make -j12 install 2>&1
endmsg

echo "Build and install MPICH."
ssh $target 'source ~/install_mpich.sh 2>&1'

echo "Copy PLDA installation script."
cat | ssh $target 'cat > ~/install_plda.sh' <<endmsg
cd ~
rm -rf $plda_pkg_name
tar xzf $plda_pkg
cd $plda_pkg_name
cat Makefile | sed 's/mpicxx/~\/usr\/bin\/mpicxx/' > Makefile.new
make -f Makefile.new -j12 all 2>&1
endmsg

ssh $target 'source ~/install_plda.sh 2>&1'
echo "Installation completed."

Hello, so where is the script install_mpich.sh and install_plda.sh?