Scientific Computing Environment for archaeology: Installing R, Rstudio and other things on a lightweight Linux in a VirtualBox virtual machine

a-scientific-computing-environment-for-archaeology.txt

Short instructions to setup a Lubuntu Virtual Machine with
R & RStudio:

1. Download these: 
http://lubuntu.net/ (Intel x86 desktop cd)
https://www.virtualbox.org/wiki/Downloads (Oracle VM VirtualBox)

2. Install Oracle VM VirtualBox, open it (if using windows,
run as administrator), click 'New' button, at 
'Name' put Lubuntu, 'Type' choose Linux, 'Version' choose
Ubuntu (32-bit), then click 'Next' and change memory size 
to 2048 MB (or more, but as a rule of thumb not more 
than a quarter of your total RAM), Click 'Next', select 
'create virtual drive now', click 'Next',
select 'VDI', click 'Next', select 'Fixed size', 
click 'Next', set disk size to 8GB, click 'Create'. Feel
free to make those values larger if you like. If you can't 
set it to 8GB, set it as large as you can. 

3. At the main Oracle VM VirtualBox window, double-click on 
'Lubuntu' the VM you just set up. You will then get a little
window that asks you to 'select start-up disk', click on the
little folder icon to the right of the drop-down field, then
find where you downloaded the lubuntu desktop cd. It should
be a file that ends in iso and is about 700 MB. Once you've 
selected that file, click 'start' on the 'select start-up disk'
window. If you get an error, try turning off your antivirus
software, if that doesn't help, uninstall VirtualBox and 
download and install a new copy of VirtualBox
from here: https://www.virtualbox.org/wiki/Testbuilds

4. You will then be taken to the Lubuntu install menu, 
choose 'English' by pressing enter, then on the next
screen 'Install Lubuntu' (use the arrow keys to select, 
then press enter). Look in the lower right corner of the
window to see what button you need to press to free the
mouse (for me it's the right control key). It will take 
a few minutes to install, keep an eye on it, you will 
need to click 'continue', then 'download updates while
installing', then 'continue' again, then 'erase disk 
and install Lubuntu', then 'continue', then edit your location
then 'continue', then 'continue again (the keyboard
layout is usually fine), then type your name and password,
keep it easy and simple as this is a very low security
use (we're not using this for online banking!) and 
you'll be typing the password often. Select 'log in
automatically' then click 'continue'. Then wait a few
minutes and watch the 'copying files' progress bar.
Once that's done, it will probably want to restart,
click to accept a restart (you may have to hit enter 
a few times to move the process along), or if you 
don't get the option, click on the
little circular shutdown icon in the lower right
corner of the window, and choose 'reboot', and wait
for it to restart (hitting enter a few times if
it looks like it's stopped doing things).

5. Now open a terminal emulator window, which you 
can do by clicking on the blue button in the lower
left corner, then click on 'Accessories' then click
on 'LXTerminal'. Or you can press CTRL-ALT-t

6. At the terminal prompt, type these, one line
at a time, exactly as printed here, and press 
enter after each line (you will be prompted to 
type your password, note that you wont see 
anything on screen when you type in 
your password, no dots or asterixes - this is
normal). Once you've typed your password, press
enter. Here are the lines to type in:

  sudo apt-get update
  sudo apt-get install build-essential module-assistant -y
  sudo m-a prepare
  sudo apt-get install virtualbox-guest-x11 -y

Wait for things to scroll past in the console,
it will take a few minutes. When you get your
prompt back (ie stuff stops scrolling by and 
the console is ready for a new command), restart 
your VM by clicking on the little circular 
shutdown icon in the lower right corner of 
the window, and choosing 'reboot' or you can 
type this at the terminal:

  sudo reboot

Once you're going again, look to the main 
Oracle VM VirtualBox menu bar, click on 
'Devices', then 'Shared clipboard', and set 
'Shared Clipboard' to bidirectional. 
Now you can copy and paste text to and from 
your VM.

Go back to the 'Devices' menu you were just at
and click on 'Shared folder settings...' then 
click on 'Machine Folders', then click on the
little button that is a picture of a folder
with a plus sign, that will make a little box
appear. In this box, click on the little down 
arrow at the right of the 'Folder Path' field,
then click on 'Other...' and choose a folder 
on your computer that contains files you want 
to work with in your VM. Check the boxes for 
'auto-mount' and 'make permanent', then click
OK, and click OK again. In your VM, open a terminal
and type:
 
sudo usermod -aG vboxsf MYUSERNAME
 
Where MYUSERNAME is the username you set up for
yourself when you made the VM. Enter your password
when prompted, then reboot the VM again. This will
give you access to the folder on your computer
from the VM, so you can easily transfer files between
the two environments. The shared folder can be 
found on the VM in a directory called 'media',
which you can access by typing /media in the 
file manager address bar. 
  
7. Open a terminal and run these three lines 
(you can now copy and paste, so do that):
 
  wget https://gist.githubusercontent.com/benmarwick/11204658/raw/install_things.sh
  chmod +x install_things.sh
  sudo ./install_things.sh
 
then press enter and wait, watching things 
scroll by. This step downloads a small script
from github and runs it. The script installs
R, RStudio, and a bunch of other things.
This will take 30+ mins, the exact time 
depends on the speed of your internet 
connection. Keep an eye on it, you'll 
need to enter your password here and 
there, and press enter once or twice. If 
know what you're doing you can edit the script
to add or subtract items. But as it is will be
fine for most purposes. 
 
8. Now you're ready to compute! If you've
never used linux or the terminal before,
do check out the video lessons at:
https://www.youtube.com/playlist?list=PLA86D04D6E0BFD2E0
and the text at:
http://software-carpentry.org/v5/novice/shell/index.html
Here are a few tips to get started:
 
  sudo firefox # opens a web browser
  sudo rstudio # opens rstudio
  sudo gedit # opens a text editor
  sudo abiword # opens a word processor
  sudo gnumeric # opens a spreadsheet

install_things.sh

#!/bin/bash

# Marwick Lab Computational Environment Setup
 
# this script can be run from the terminal with the next line (minus the #)
# bash install_things.sh

# you will need to enter your password at a few points, so keep an eye on it while it runs.

# in case we need to step through:
# set -x
# trap read debug
 
## Need this for Debian but not Lubuntu 
## Make self su: http://stackoverflow.com/a/15067345/1036500
# install sudo to ensure we get access to everything
# su -c "aptitude install sudo"
# su root
## Here we assume your username is ben ... do edit this line if you chose a different username!
# sudo adduser ben sudo
 
echo "install a few dependancies for our workflow"
sudo apt-get update  -y
# sudo apt-get upgrade  -y
sudo apt-get install libgstreamer0.10-0 -y
sudo apt-get install libgstreamer-plugins-base0.10-dev -y
sudo apt-get install libcurl4-openssl-dev -y
sudo apt-get install libssl-dev libssh2-1-dev  -y
sudo apt-get install libopenblas-base -y
sudo apt-get install libxml2-dev -y
sudo apt-get install make -y
sudo apt-get install gcc -y
sudo apt-get install git -y
sudo apt-get install pandoc -y
sudo apt-get install libjpeg62 -y
sudo apt-get install unzip -y
sudo apt-get install curl -y
sudo apt-get install littler -y
sudo apt-get install openjdk-7-* -y
sudo apt-get install gedit -y
sudo apt-get install jags -y
sudo apt-get install imagemagick -y
sudo apt-get install docker-engine -y
sudo apt-get install gdebi-core
# sudo apt-get install mysql-server -y

# for Texlive with Lubuntu (making PDFs)
# these are BIG, not recommended if you're short on disk space!
# sudo add-apt-repository ppa:texlive-backports/ppa
# sudo apt-get update
# sudo apt-get install texlive-latex-base -y
# sudo apt-get install texlive-fonts-recommended -y
# sudo apt-get install texlive-latex-extra -y

# a few handy FOSS items for drawing, graphics and maps
# sudo apt-get install inkscape -y
# sudo apt-get install gimp -y
# sudo apt-add-repository ppa:ubuntugis/ppa
# sudo apt-get update
# sudo apt-get install qgis -y

# also a few things in case we use python 
# from http://faculty.washington.edu/rjl/uwhpsc-coursera/vm.html
#sudo apt-get install liblzma-dev -y
#sudo apt-get install ipython -y
#sudo apt-get install ipython-notebook -y 
#sudo apt-get install python-pandas -y
#sudo apt-get install python-numpy -y
#sudo apt-get install python-scipy -y
#sudo apt-get install python-matplotlib -y
#sudo apt-get install python-dev -y
#sudo apt-get install python-sphinx -y
#sudo apt-get install gfortran -y
#sudo apt-get install openmpi-bin -y
#sudo apt-get install liblapack-dev -y
#sudo apt-get install thunar -y
#sudo apt-get install gitk -y
#sudo apt-get install docdiff -y
#sudo apt-get install python-setuptools -y

echo "edit the sources file to prepare to install R"
# see http://cran.r-project.org/bin/linux/ubuntu/README
sudo sh -c 'echo "deb http://cran.rstudio.com/bin/linux/ubuntu yakkety/" >> /etc/apt/sources.list' # if Lubuntu

echo "get keys to install R"
# sudo apt-key adv --keyserver keys.gnupg.net --recv-key 381BA480 # if Debian
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9 # if Lubuntu
 
echo "install R and some helpers"
sudo apt-get update
sudo apt-get install r-base  -y
sudo apt-get install r-base-dev -y
sudo apt-get install r-cran-xml  -y
sudo apt-get install r-cran-rjava -y
sudo R CMD javareconf # for rJava

echo "install RStudio from the web"
# use daily build to get rmarkdown & latest goodies
# http://stackoverflow.com/a/15046782/1036500
# check if 32 or 64 bit and install appropriate version... 
# http://stackoverflow.com/a/106416/1036500

# currently these steps are needed to get RStudio installed https://stackoverflow.com/a/38646838/1036500

MACHINE_TYPE=`uname -m`
if [ ${MACHINE_TYPE} == 'x86_64' ]; then
  # 64-bit stuff here

URL=$(wget -q -O -  http://www.rstudio.org/download/daily/desktop/ubuntu64 | grep -o -m 1 "https[^\']*" )

FILE=`mktemp`; sudo wget "$URL" -qO $FILE && sudo gdebi $FILE --non-interactive; rm $FILE

else
  # 32-bit stuff here

URL=$(wget -q -O -  http://www.rstudio.org/download/daily/desktop/ubuntu32 | grep -o -m 1 "https[^\']*" )

FILE=`mktemp`; sudo wget "$URL" -qO $FILE && sudo gdebi $FILE --non-interactive; rm $FILE
fi


echo "start R and install commonly used packages"
# http://stackoverflow.com/q/4090169/1036500
# Make an R script file to use in a moment...
LOADSTUFF="options(repos=structure(c(CRAN='http://cran.rstudio.com/')))
update.packages(checkBuilt = TRUE, ask = FALSE)
packages <- c('codetools', 'Rcpp', 'devtools', 'knitr', 'ggplot2',  'plyr', 'dplyr', 'XML', 'RCurl', 'readxl', 'tidyr') 
# just some of my most often used ones
install.packages(packages)
# and some from github
devtools::install_git(c('https://github.com/rstudio/rmarkdown.git',  'https://github.com/hadley/reshape'))" # close the R script

# put that R code into an R script file
FILENAME1="loadstuff.r"
sudo echo "$LOADSTUFF" > /tmp/$FILENAME1

# Make a shell file that contains instructions in bash for running that R script file
# from the command line. There may be a simpler way, but nothing I tried worked.
NEWBASH='#!/usr/bin/env 
sudo Rscript /tmp/loadstuff.r'
FILENAME2="loadstuff.sh"

# put that bash code into a shell script file
sudo echo "$NEWBASH" > /tmp/$FILENAME2

# run the bash file to exectute the R code and install the packages
sh /tmp/loadstuff.sh
# clean up by deleting the temp file
rm /tmp/loadstuff.sh 
 
# done.
echo "all done"
echo "type 'sudo rstudio' in the terminal to start RStudio"

loadstuff.r

# set cran repo so the user doesn't have to
options(repos=structure(c(CRAN='http://cran.rstudio.com/')))
# update installed pacakges
update.packages(checkBuilt = TRUE, ask = FALSE)
# make a list of additional packages to install
packages <- c('codetools', 'Rcpp', 'devtools', 'knitr', 'ggplot2', 'plyr', 'dplyr', 'XML', 'RCurl', 'readxl', 'tidyr')
# install that list
install.packages(packages)
# and some from github
devtools::install_git(c('https://github.com/rstudio/rmarkdown.git', 'https://github.com/hadley/reshape'))" 
message("All done installing additional packages")
# end
 

notes-to-self.txt

############# More detailed bits and pieces ##############

If archeos https://github.com/archeos/ArcheOS gets things right it could replace all of this, just run that in the VM

The Berkeley Computational Enviroment http://collaboratool.berkeley.edu/ is similar but a bit heavy

## First download a Linux distribution: 
# Debian internet-installer image file: http://www.debian.org/distrib/netinst
# or
# Lubuntu, if looking for something smaller and using less memory, use the desktop cd from http://lubuntu.net/

# then download and install Oracle VM VirtualBox: https://www.virtualbox.org/wiki/Downloads

## start VirtualBox
# Make new VM, follow prompts, set "Type: Linux" and "Version: Debian" or if using Lubuntu, "Version: Ubuntu" then ok
# at 'select start-up disk' choose the downloaded iso to make the VM from, then if using Debian, use the Debian graphic installer
# many fields can be left blank, try them and see, but DO be sure to select “Debian desktop environment”
# There are many useful guides for this (stop following when you get to the desktop, don't run any of the code in these how-tos)
# Here's a selection of how-tos (you might also search youtube)
# Mac: http://williamjturkel.net/2013/05/31/installing-debian-linux-in-a-vm/ 
# http://www.brianlinkletter.com/installing-debian-linux-in-a-virtualbox-virtual-machine/
# Windows: https://jtreminio.com/2012/07/setting-up-a-debian-vm-step-by-step/

## install guest additions (GA) to allow for copy and pasting between VM and host
# http://askubuntu.com/questions/22743/how-do-i-install-guest-additions-in-a-virtualbox-vm
# first open terminal and type (without #)
# sudo apt-get update
# sudo apt-get install make gcc
# then at the terminal, change to the GA directory and type:
# sudo ./VBoxLinuxAdditions.run 
# then 
# sudo reboot

# Once VM has finished installing and Linux is up and running on your screen, 
# open a terminal to continue, in Debian do:

Applications -> Accessories ->Tterminal

# Or in Lubuntu do:

Button in lower left of screen -> Accessories -> LXTerminal

# see shell script below for getting things installed, read through it before running to 
# know what to expect, then copy-paste the shell script below into the text editor (gedit, leafpad etc.) on your  VM, 
# save locally on your VM, then run the shell script from the terminal by 
# typing the next line in the terminal (without the #):
# bash install_things.sh

## Other things we want to do with a VM:

# to share the entire Linux VM with others
# http://superuser.com/questions/235376/how-to-share-a-virtualbox-image
file -> export appliance

# to swap files between host OS and guest OS: https://www.virtualbox.org/manual/ch04.html
# and don't forget to add self to share folder: https://www.virtualbox.org/ticket/7951

# Debian: getting in and out of su: 
# in:  su -
# out: su yourusername

# Docker 
# this seems to be important to set variables and make Docker work on windows
# eval "$(docker-machine.exe env docker)"

# To do:

# add right-click or keyboard shortcut to 'open terminal'  cf. http://askubuntu.com/a/207448/119774 http://www.cyberciti.biz/faq/linux-gnome-open-terminal-shell-prompt-here/

# see http://cran.r-project.org/bin/linux/ubuntu/README
# sudo sh -c 'echo "deb http://cran.rstudio.com/bin/linux/debian wheezy-cran3/" >> /etc/apt/sources.list' # if Debian
sudo sh -c 'echo "deb http://cran.rstudio.com/bin/linux/ubuntu trusty/" >> /etc/apt/sources.list' # if Lubuntu
# sudo sh -c 'echo "deb http://cran.rstudio.com/ raring-backports main restricted universe" >> /etc/apt/sources.list' # if Lubuntu

# sudo easy_install rpy2 # if using python: now we can sudo ipython notebook & use R within: http://nbviewer.ipython.org/github/ipython/ipython/blob/3607712653c66d63e0d7f13f073bde8c0f209ba8/docs/examples/notebooks/rmagic_extension.ipynb#Rmagic-Functions-Extension
 


From https://github.com/ropensci/docker/blob/master/ropensci/Dockerfile

## Add CRAN binaries and update
RUN add-apt-repository -y ppa:marutter/c2d4u && apt-get update -qq && apt-get upgrade -qy 


## From the Build-Depends of the Debian R package: 968 MB!
RUN apt-get install -y --no-install-recommends gcc g++ gfortran libblas-dev liblapack-dev tcl8.5-dev tk8.5-dev bison groff-base libncurses5-dev libreadline-dev debhelper texinfo libbz2-dev liblzma-dev libpcre3-dev xdg-utils zlib1g-dev libpng-dev libjpeg-dev libx11-dev libxt-dev x11proto-core-dev libpango1.0-dev libcairo2-dev libtiff5-dev xvfb xauth xfonts-base texlive-base texlive-latex-base texlive-generic-recommended texlive-fonts-recommended texlive-fonts-extra texlive-extra-utils texlive-latex-recommended texlive-latex-extra default-jdk mpack bash-completion 

## rmarkdown-related dependencies: 146 MB
RUN apt-get install -y --no-install-recommends texlive-humanities lmodern texlive-fonts-recommended texlive-latex-extra imagemagick pandoc pandoc-citeproc 

## R binaries: 149 MB 
RUN apt-get install -y r-cran-dplyr r-cran-devtools r-cran-rcpp r-cran-xml r-cran-rjava r-cran-rmysql r-cran-rsqlite libxslt1-dev r-cran-rsqlite.extfuns r-cran-rmysql r-cran-data.table r-cran-testthat r-cran-knitr r-cran-ggplot2 r-cran-mgcv r-cran-mass r-cran-tidyr r-cran-shiny r-cran-monetdb.r libxslt1-dev libpq-dev 

## Not avaiable on marutter/c2d4u
# r-cran-rstudioapi r-cran-microbenchmark r-cran-rpostgresql r-cran-roxygen2

RUN echo 'options("repos"="http://cran.rstudio.com", encoding="UTF-8")' > /.Rprofile
RUN Rscript -e 'install.packages(c("RPostgreSQL", "roxygen2", "microbenchmark", "rstudioapi"))'
RUN Rscript -e 'source("http://bioconductor.org/biocLite.R"); biocLite("rhdf5", ask=FALSE); biocLite("BiocInstaller")'
RUN Rscript -e 'devtools::install_github(c("rstudio/rmarkdown", "rstudio/packrat", "hadley/bigrquery", "hadley/reshape"))'
RUN Rscript -e 'install.packages(c("SSOAP", "XMLSchema", "Sxslt"), repo="http://www.omegahat.org/R", type="source")'
RUN Rscript -e 'devtools::install_github("egonw/rrdf", subdir="rrdflibs", build_vignettes=FALSE); devtools::install_github("egonw/rrdf", subdir="rrdf", build_vignettes=FALSE)'

###
ubuntu mini, https://help.ubuntu.com/community/Installation/MinimalCD
http://ask.xmodulo.com/install-minimal-ubuntu-desktop.html

## if buggy, get the pre-built VM from 
http://www.osboxes.org/lubuntu/