How to install the required libs for ML4A on OS X with GPU support

HOWTO-ML4A-OSX-GPU.md

How to setup a virtualenv for ml4a on OS X with GPU support

Requirements

This is tested on OS X 10.11.6 running on a MacBook Pro with NVIDIA GeForce GT 750M as of Tuesday, November 22, 2016; if you are note sure about your system, run

sw_vers

it should return something like

ProductName:	Mac OS X
ProductVersion:	10.11.6
BuildVersion:	15G1108

then run

system_profiler -detailLevel mini | grep 'Chipset Model:'

it should return (after a few seconds) something like

Chipset Model: Intel Iris Pro
Chipset Model: NVIDIA GeForce GT 750M

(the relevant line is the second one, that tells that a GPU from NVIDIA is available).

Install Homebrew

First of all, you'll need Homebrew and thus at least the Xcode Command Line Tools from Apple; the commands

xcode-select --install
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

should get you started (they will take quite a while to complete).

## Install CUDA libraries

First install CUDA 8.0 libraries with

brew cask install cuda

At the end of the install process, you should have a

/Developer/NVIDIA/CUDA-8.0/lib

on your filesystem (you can test with ls -d /Developer/NVIDIA/CUDA-8.0/lib).

Then you need cuDNN 5.1 libraries, you can obtain them from the CUDA Developer site, (or asking someone aound); you need the file named

cudnn-8.0-osx-x64-v5.1.tgz

no other version will work!

Now some fiddling with your filesystem is required. Go where you saved the cudnn-8.0-osx-x64-v5.1.tgz file and run the following commands

tar zxvf cudnn-8.0-osx-x64-v5.1.tgz
sudo mv cuda/lib/libcudnn* /usr/local/cuda/lib
sudo mv cuda/include/cudnn.h /usr/local/cuda/include
cd /usr/local/cuda/lib
sudo ln -s libcuda.dylib libcuda.1.dylib
sudo ln -s libcudnn.5.dylib libcudnn5.dylib

It's scary, but works. What you are doing is putting the required libraries in a well known place where the rest of the software will look for them, and providing some alias (using symbolic links).

Install Python and virtualenv

Start by installing Python and virtualenv (we'll use virtualenvwrapper that's a useful addon to virtualenv). Just run

brew install python
pip install --upgrade pip virtualenvwrapper

If all works as expected you should be able to run

mkvirtualenv ml4a-gpu

that will create and acrivate a virtualenv named ml4a-gpu; your prompt should start with (ml4a-gpu2) now. When done you can use deactivate to exit from the virtualenv. On successive runs, just activate the virtualenv with

workon ml4a-gpu

Install all the required Python libraries

To get the list of required libraries in a format that pip understands, just run

curl -sLO https://gist.githubusercontent.com/mapio/5e67c61e8a05c4ed096c43cee644014c/raw/requirements.txt

that should download a requirements.txt file in the current directory. Once the virutalenv is activated (if you have just created it, it is active), you can populate it with all the required libraries once and forever with

pip install -r requirements.txt

it will take a while, but you need to run this just once. A quick check that all is working as expected, run

 python -c 'import tensorflow'

that should return something like

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.dylib locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.dylib locally

if instead it emits warnings, in particular a Signal 11 message… then you had no luck (and I don't know how to help).

The Jupyter wrapper

Now you'll need to run Jupyter but unfortunately some of the libraries you just installed don't live well in a virtualenv. You can fix such hatred downloading a small wrapper that will trick Jupyter and the libraries to behave. Run

curl -sLO https://gist.githubusercontent.com/mapio/5e67c61e8a05c4ed096c43cee644014c/raw/jupyter-notebook.sh
chmod u+x jupyter-notebook.sh

to download the wrapper and make it executable. A good place to download it is inside the folder provided by the teacher where you see notebooks and data.

You are ready to run the notebooks, if you just want a last check before the take-off download the miminalist notebook

curl -sLO https://gist.githubusercontent.com/mapio/5e67c61e8a05c4ed096c43cee644014c/raw/tfguptest.ipynb

run Jupiter with the wrapper

./jupyter-notebook.sh

open the above notebook and execute its single cell; it should not output anything in the browser, but on the terminal you should read something like

Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 750M, pci bus id: 0000:01:00.0)

if you read cpu instead of gpu it means that all your efforts where sadly useless.

jupyter-notebook.sh

#!/bin/bash

export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/cuda/extras/CUPTI/lib

function frameworkpython {
    if [[ ! -z "$VIRTUAL_ENV" ]]; then
        PYTHONHOME=$VIRTUAL_ENV /usr/local/bin/python "$@"
    else
        /usr/local/bin/python "$@"
    fi
}

cat <<EOF | frameworkpython
import sys
from jupyter_core.command import main
sys.argv = ['jupyter.py', 'notebook']
main()
EOF

requirements.txt

h5py
https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow-0.11.0-py2-none-any.whl
jupyter
jupyter
keras
librosa
matplotlib
numpy
Pillow
scikit-learn
scipy
simplejson
jupyterlab

tfguptest.ipynb