Ubuntu Anaconda Deep Learning

README.md

Setting up for Deep Learning on Ubuntu 14.04

Saving this so I can find it later & potentially help others. This sets up the latest as of 10/7/2017

• Python
• Keras
• Tensorflow
• PyTorch

Assumes Anaconda3 is installed. Here is mine:

> conda -V
conda 4.3.21
> python -V
Python 3.6.1 :: Anaconda 4.4.0 (64-bit)

Note that on upgrade to the new conda, I had to clean out old kernels & environments. I hit it with a hammer like so:

rm -rf ~/.ipython ~/.jupyter ~/.conda ~/.local/share/jupyter

Create an environment with the packages we'll want. To do this we'll put our dependencies into a file that is basically the root environment modified to have the new packages for deep learning

NOTE: next time I'll use conda create --name dl --clone root

> conda env export -n root > root_environment.yml
> cp root_environment.yml dl_environment.yml
# edit dl_environement to change the name & add the modules we care about.  See attached.
# adding pytorch torchvision cuda80 from soumith
# adding keras & tensorflow-gpu
# also remove anaconda & conda packages.
> conda env create -f dl_environment.yml

Activate

> source activate dl

Did not need to give jupyter notebook the ability to run this environment...just Python3 kernel was enough.

Here is a handy snippet to validate versions.

# scipy
import scipy
print('scipy: %s' % scipy.__version__)
# numpy
import numpy
print('numpy: %s' % numpy.__version__)
# matplotlib
import matplotlib
print('matplotlib: %s' % matplotlib.__version__)
# pandas
import pandas
print('pandas: %s' % pandas.__version__)
# statsmodels
import statsmodels
print('statsmodels: %s' % statsmodels.__version__)
# scikit-learn
import sklearn
print('sklearn: %s' % sklearn.__version__)
# keras
import keras
print('keras: %s' % keras.__version__)
# tensorflow
import tensorflow
print('tensorflow: %s' % tensorflow.__version__)
# bcolz
import bcolz
print('bcolz: %s' % bcolz.__version__)
# gensim
import gensim
print('gensim: %s' % gensim.__version__)
# nltk
import nltk
print('nltk: %s' % nltk.__version__)
# h5py
import h5py
print('h5py: %s' % h5py.__version__)
# torch
import torch
print('torch: %s' % torch.__version__)

outputs this for me:

scipy: 0.19.0
numpy: 1.12.1
matplotlib: 2.0.2
pandas: 0.20.1
statsmodels: 0.8.0
sklearn: 0.18.1
Using TensorFlow backend.
keras: 2.0.8
tensorflow: 1.3.0
bcolz: 1.0.0
gensim: 2.3.0
nltk: 3.2.3
h5py: 2.7.0
torch: 0.1.12_1

dl_environment.yml

name: dl
channels:
- defaults
- soumith
dependencies:
- _license=1.1=py36_1
- alabaster=0.7.10=py36_0
- asn1crypto=0.22.0=py36_0
- astroid=1.4.9=py36_0
- astropy=1.3.2=np112py36_0
- babel=2.4.0=py36_0
- backports=1.0=py36_0
- beautifulsoup4=4.6.0=py36_0
- bitarray=0.8.1=py36_0
- blaze=0.10.1=py36_0
- bleach=1.5.0=py36_0
- bokeh=0.12.5=py36_1
- boto=2.46.1=py36_0
- bottleneck=1.2.1=np112py36_0
- cairo=1.14.8=0
- cffi=1.10.0=py36_0
- chardet=3.0.3=py36_0
- click=6.7=py36_0
- cloudpickle=0.2.2=py36_0
- clyent=1.2.2=py36_0
- colorama=0.3.9=py36_0
- contextlib2=0.5.5=py36_0
- cryptography=1.8.1=py36_0
- curl=7.52.1=0
- cycler=0.10.0=py36_0
- cython=0.25.2=py36_0
- cytoolz=0.8.2=py36_0
- dask=0.14.3=py36_1
- datashape=0.5.4=py36_0
- dbus=1.10.10=0
- decorator=4.0.11=py36_0
- distributed=1.16.3=py36_0
- docutils=0.13.1=py36_0
- entrypoints=0.2.2=py36_1
- et_xmlfile=1.0.1=py36_0
- expat=2.1.0=0
- fastcache=1.0.2=py36_1
- flask=0.12.2=py36_0
- flask-cors=3.0.2=py36_0
- fontconfig=2.12.1=3
- freetype=2.5.5=2
- get_terminal_size=1.0.0=py36_0
- gevent=1.2.1=py36_0
- glib=2.50.2=1
- greenlet=0.4.12=py36_0
- gst-plugins-base=1.8.0=0
- gstreamer=1.8.0=0
- h5py=2.7.0=np112py36_0
- harfbuzz=0.9.39=2
- hdf5=1.8.17=1
- heapdict=1.0.0=py36_1
- html5lib=0.999=py36_0
- icu=54.1=0
- idna=2.5=py36_0
- imagesize=0.7.1=py36_0
- ipykernel=4.6.1=py36_0
- ipython=5.3.0=py36_0
- ipython_genutils=0.2.0=py36_0
- ipywidgets=6.0.0=py36_0
- isort=4.2.5=py36_0
- itsdangerous=0.24=py36_0
- jbig=2.1=0
- jdcal=1.3=py36_0
- jedi=0.10.2=py36_2
- jinja2=2.9.6=py36_0
- jpeg=9b=0
- jsonschema=2.6.0=py36_0
- jupyter=1.0.0=py36_3
- jupyter_client=5.0.1=py36_0
- jupyter_console=5.1.0=py36_0
- jupyter_core=4.3.0=py36_0
- lazy-object-proxy=1.2.2=py36_0
- libffi=3.2.1=1
- libgcc=4.8.5=2
- libgfortran=3.0.0=1
- libiconv=1.14=0
- libpng=1.6.27=0
- libsodium=1.0.10=0
- libtiff=4.0.6=3
- libtool=2.4.2=0
- libxcb=1.12=1
- libxml2=2.9.4=0
- libxslt=1.1.29=0
- llvmlite=0.18.0=py36_0
- locket=0.2.0=py36_1
- lxml=3.7.3=py36_0
- markupsafe=0.23=py36_2
- matplotlib=2.0.2=np112py36_0
- mistune=0.7.4=py36_0
- mkl=2017.0.1=0
- mkl-service=1.1.2=py36_3
- mpmath=0.19=py36_1
- msgpack-python=0.4.8=py36_0
- multipledispatch=0.4.9=py36_0
- navigator-updater=0.1.0=py36_0
- nbconvert=5.1.1=py36_0
- nbformat=4.3.0=py36_0
- networkx=1.11=py36_0
- nltk=3.2.3=py36_0
- nose=1.3.7=py36_1
- notebook=5.0.0=py36_0
- numba=0.33.0=np112py36_0
- numexpr=2.6.2=np112py36_0
- numpy=1.12.1=py36_0
- numpydoc=0.6.0=py36_0
- odo=0.5.0=py36_1
- olefile=0.44=py36_0
- openpyxl=2.4.7=py36_0
- openssl=1.0.2l=0
- packaging=16.8=py36_0
- pandas=0.20.1=np112py36_0
- pandocfilters=1.4.1=py36_0
- pango=1.40.3=1
- partd=0.3.8=py36_0
- path.py=10.3.1=py36_0
- pathlib2=2.2.1=py36_0
- patsy=0.4.1=py36_0
- pcre=8.39=1
- pep8=1.7.0=py36_0
- pexpect=4.2.1=py36_0
- pickleshare=0.7.4=py36_0
- pillow=4.1.1=py36_0
- pip=9.0.1=py36_1
- pixman=0.34.0=0
- ply=3.10=py36_0
- prompt_toolkit=1.0.14=py36_0
- psutil=5.2.2=py36_0
- ptyprocess=0.5.1=py36_0
- py=1.4.33=py36_0
- pycosat=0.6.2=py36_0
- pycparser=2.17=py36_0
- pycrypto=2.6.1=py36_6
- pycurl=7.43.0=py36_2
- pyflakes=1.5.0=py36_0
- pygments=2.2.0=py36_0
- pylint=1.6.4=py36_1
- pyodbc=4.0.16=py36_0
- pyopenssl=17.0.0=py36_0
- pyparsing=2.1.4=py36_0
- pyqt=5.6.0=py36_2
- pytables=3.3.0=np112py36_0
- pytest=3.0.7=py36_0
- python=3.6.1=2
- python-dateutil=2.6.0=py36_0
- pytz=2017.2=py36_0
- pywavelets=0.5.2=np112py36_0
- pyyaml=3.12=py36_0
- pyzmq=16.0.2=py36_0
- qt=5.6.2=4
- qtawesome=0.4.4=py36_0
- qtconsole=4.3.0=py36_0
- qtpy=1.2.1=py36_0
- readline=6.2=2
- requests=2.14.2=py36_0
- rope=0.9.4=py36_1
- ruamel_yaml=0.11.14=py36_1
- scikit-image=0.13.0=np112py36_0
- scikit-learn=0.18.1=np112py36_1
- scipy=0.19.0=np112py36_0
- seaborn=0.7.1=py36_0
- setuptools=27.2.0=py36_0
- simplegeneric=0.8.1=py36_1
- singledispatch=3.4.0.3=py36_0
- sip=4.18=py36_0
- six=1.10.0=py36_0
- snowballstemmer=1.2.1=py36_0
- sortedcollections=0.5.3=py36_0
- sortedcontainers=1.5.7=py36_0
- sphinx=1.5.6=py36_0
- spyder=3.1.4=py36_0
- sqlalchemy=1.1.9=py36_0
- sqlite=3.13.0=0
- statsmodels=0.8.0=np112py36_0
- sympy=1.0=py36_0
- tblib=1.3.2=py36_0
- terminado=0.6=py36_0
- testpath=0.3=py36_0
- tk=8.5.18=0
- toolz=0.8.2=py36_0
- tornado=4.5.1=py36_0
- traitlets=4.3.2=py36_0
- unicodecsv=0.14.1=py36_0
- unixodbc=2.3.4=0
- wcwidth=0.1.7=py36_0
- werkzeug=0.12.2=py36_0
- wheel=0.29.0=py36_0
- widgetsnbextension=2.0.0=py36_0
- wrapt=1.10.10=py36_0
- xlrd=1.0.0=py36_0
- xlsxwriter=0.9.6=py36_0
- xlwt=1.2.0=py36_0
- xz=5.2.2=1
- yaml=0.1.6=0
- zeromq=4.1.5=0
- zict=0.1.2=py36_0
- zlib=1.2.8=3
- bcolz
- gensim
- py-xgboost
- pytorch
- torchvision
- cuda80
- pip:
  - backports.shutil-get-terminal-size==1.0.0
  - et-xmlfile==1.0.1
  - ipython-genutils==0.2.0
  - jupyter-client==5.0.1
  - jupyter-console==5.1.0
  - jupyter-core==4.3.0
  - prompt-toolkit==1.0.14
  - rope-py3k==0.9.4.post1
  - tables==3.3.0
  - tensorflow-gpu
  - keras
prefix: /home/rallen/anaconda3

oldREADME.md

Setting up for Fast.ai Deep Learning Course 2 on Ubuntu 14.04

Saving this so I can find it later & potentially help others. Given how rapidly some packages evolve, sometimes getting specific versions is necessary. Esp. Keras & Tensorflow. Also, Tensorflow requires Python 3.5. So, this sets up Python 3.5, Keras 1.2.2 and Tensorflow 1.0.1.

Assumes Anaconda3 is installed. Here is mine:

> conda -V
conda 4.3.8
> python -V
Python 3.6.0 :: Anaconda 4.3.0 (64-bit)

Create an environment with the packages we'll want. Updated for a few packages needed for Part 2 of the fast.ai course.

> conda create -n dl3_p35k12tf10tch01 python=3.5 scipy numpy matplotlib scikit-learn pandas pillow statsmodels ipykernel bcolz gensim nltk h5py py-xgboost

Activate

> source activate dl3_p35k12tf10tch01

Add pytorch

> conda install pytorch torchvision cuda80 -c soumith                                                                                                 

Add tensorflow 1.0.1

> pip install tensorflow-gpu==1.0.1                                                                                                                   

Add keras 1.2.2

> pip install keras==1.2.2

Add keras-tqdm to help us with updating nice progress bars.

> pip install keras-tqdm

Give jupyter notebook the ability to run this environment (aka kernel in Jupyter)

> python -m ipykernel install --user --name dl3_p35k12tf10tch01

Use the Kernels menu in jupyter notebook to select the environment you created. Here is a handy snippet to validate versions.

# scipy
import scipy
print('scipy: %s' % scipy.__version__)
# numpy
import numpy
print('numpy: %s' % numpy.__version__)
# matplotlib
import matplotlib
print('matplotlib: %s' % matplotlib.__version__)
# pandas
import pandas
print('pandas: %s' % pandas.__version__)
# statsmodels
import statsmodels
print('statsmodels: %s' % statsmodels.__version__)
# scikit-learn
import sklearn
print('sklearn: %s' % sklearn.__version__)
# keras
import keras
print('keras: %s' % keras.__version__)
# tensorflow
import tensorflow
print('tensorflow: %s' % tensorflow.__version__)
# bcolz
import bcolz
print('bcolz: %s' % bcolz.__version__)
# gensim
import gensim
print('gensim: %s' % gensim.__version__)
# nltk
import nltk
print('nltk: %s' % nltk.__version__)
# h5py
import h5py
print('h5py: %s' % h5py.__version__)
# torch
import torch
print('torch: %s' % torch.__version__)