Spark
If you have (larger-than-memory) petabytes of JSON/XML/CSV files, a simple workflow, and a thousand node cluster
Dask
If you have (larger-than-memory) 10s-1000s of gigabytes of binary or numeric data (e.g., HDF5, netCDF4, CSV.gz), complex algorithms, and a (single) large multi-core workstation
You need the package manager Homebrew (if not installed), see http://brew.sh/
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Get Spark 'Source Code':
http://spark.apache.org/downloads.html
For basic applications, MPI is as easy to use as any other message-passing system. The sample scripts below contain the complete communications skeleton for a data (or embarrassingly) parallel problem using the mpi4py package.
Within the code is a description of the few functions needed to write typical parallel applications.
mpi-submit.py - Parallel application with simple partitioning: unbalanced load.
mpi-submit2.py - Parallel application with master/slave scheme: dynamically balanced load.
Some (trivial?) commands to work on a remote machine using the SSH, FTP and HTTP protocols.
Generate SSH keys for using a SSH connection without having to enter the password every time you log in.
Generate the key (local machine) [Skip this step if you have already the keys in your machine!]:
Triton, TCC and TSCC, are clusters at the San Diego Supercomputer Center (SDSC): http://www.sdsc.edu/services/hpc/hpc_systems.html#tscc
Account balance:
How to set up a grid of computers using the Mac OS X desktop version.
Any machine can be an agent. Configure the agent using the Sharing Pane of System Preferences. See Managing Xgrid Agents.
Although I'm not a big fan of Fortran, here is a simple example of parsing Unix-like command line arguments to a program following the modern standard of F2003. Note that the following example also works for any F90/F95 code compiled with gfortran or g95.
Example:
$ ./thisprog -h
usage: ./thisprog [-h] [-a ARG_A] [-b ARG_B] [-c]