MATLAB on sherlock

backup_sherlock.md

How to backup data from sherlock to google drive

1. Download and install rclone to your $HOME/bin directory. Setup up rclone to connect to your google drive. Suppose you call your remote google drive drive.

2. Create a bash script called backup_gdrive.sh with the following information

#!/bin/sh

STUDY_DIR=STUDY
LOCAL_DIR=${SCRATCH}/${STUDY_DIR}
GDRIVE_DIR=Data/${STUDY_DIR}/derivatives/

## Replace `drive` with whatever you call your remote backup volume in rclone. 
rclone sync --copy-links --update ${LOCAL_DIR} drive:${GDRIVE_DIR}

## Uncomment and use copy instead of sync if you don't want files on google drive to be overwritten. 
# rclone copy --copy-links --update ${LOCAL_DIR} drive:${GDRIVE_DIR}

## Using checksum slows down the operation as it is intensive to use checksum to compare if the local and remote files are identical 
# rclone copy --checksum --update ${LOCAL_DIR} drive:${GDRIVE_DIR}

3. Now submit a backup job to slurm using sbatch -t 4:00:00 -p owners backup_gdrive.sh

bash_tips.md

To unzip large files in place do the following

find . -name '*.zip' -exec sh -c 'unzip -d `dirname {}` {}' ';'

matlab_sherlock.md

If you have followed instructions provided by Research Computing and still failed to successfully call matlab. Make the following changes and try again.

vi ~/.bashrc

Add the following lines to your ~.bashrc file on sherlock 1

module load jre
export MATLAB_JAVA=/share/sw/free/jre/jre1.8.0_91/
module load matlab/R2016b
alias start_matlab='unset _JAVA_OPTIONS ; matlab -nodesktop -nosplash -nodisplay
alias matlab_desktop='unset _JAVA_OPTIONS ; matlab -nosplash'

For sherlock 2

module load java
export MATLAB_JAVA=/share/software/user/open/java/1.8.0_131/jre
module load math
module load matlab/R2017a
alias start_matlab='unset _JAVA_OPTIONS ; matlab -nodesktop -nosplash -nodisplay
alias matlab_desktop='unset _JAVA_OPTIONS ; matlab -nosplash'

Now you are ready to try this

ssh -Y sherlock
# For interactive dev environment
sdev -m 8gb --pty --x11
start_matlab

If you need to obtain larger memory allocation on your node or a longer time-frame then use

srun -N 1 -n 2 --mem-per-cpu=8G -t 4:00:00 --pty --x11 bash 

Once the node has been allocated, you start an commandline matlab session using start_matlab or an interactive desktop session using matlab_desktop.

Notes:

1. Mathworks also has patches for Java exception errors in older versions of MATLAB. There might be similar issues on linux systems. See here

2. To load MATLAB's desktop environment, you need to ensure X11 forwarding works when you ssh from your local machine as well as when you get an allocation on a cluster node.

• Test that you have X11/xquartz working on your local MAC OS X by first running xhost + successfully from your commandline.
• Next ssh sherlock -Y (This ensures trusted X11 Forwarding is enabled)
• On sherlock 1, you can try to run /share/sw/examples/X11/xeyes or /share/sw/examples/X11/xclock. If these programs don't run then you do not have X11 forwarding working. Please google your errors or contact Research Computing for further support.
• If the above programs work then you should be able to run load MATLAB desktop on your local machine.

mount_sherlock.md

Here is an easy to make files/directories easily accessible on a local machine.

How to mount sherlock directories locally All you have to do mount_sherlock to have access to sherlock directories at your fingertips at a local directory $SHERLOCK/

$SHERLOCK/
        |-- .
        |-- home/
        |-- scratch/

If you need to reconnect after getting disconnected, first unmount using unmount_sherlock and then mount again.

Requirements & Setup

• Be sure to have valid sherlock login credentials using kinit.
• Install OSXFuse and sshfs
• Modify your ~/.bashrc with the following environment variables and alias commands

Step 1: Create environment variables that store your sherlock directory paths, and where you want to mount sherlock locally

# Local mount points. Create this directory if it doesn't exist
export SHERLOCK=/path/to/Sherlock

# Make these once and for all 
mkdir $SHERLOCK/home
mkdir $SHERLOCK/scratch

# Directories on Sherlock. Replace <username> and <groupname> appropriately
export SHERLOCK_HOME = /home/<username>
export SHERLOCK_SCRATCH = /scratch/PI/<groupname>/<username>

Step 2: Now make an alias containing mount and unmount commands in your ~/.bashrc

alias mount_sherlock="sshfs sherlock:$SHERLOCK_HOME $SHERLOCK/home; sshfs $SHERLOCK_SCRATCH $SHERLOCK/scratch;"
alias umount_sherlock="umount $SHERLOCK/home; umount $SHERLOCK/scratch;"