dusenberrymw

TensorFlow Tips & Tricks

GPU Memory Issues

• nvidia-smi to check for current memory usage.
• watch -n 1 nvidia-smi to monitor memory usage every second.
• Often, extra Python processes can stay running in the background, maintaining a hold on the GPU memory, even if nvidia-smi doesn't show it.
  • Probably due to running Keras in a notebook, and then running the cell that starts the processes again, since this will fork the current process, which has a hold on GPU memory. In the future, restart the kernel first, and stop all process before exiting (even though they are daemons and should stop automatically when the parent process ends).

dusenberrymw

{
 "argv": [
  "/usr/local/opt/python3/bin/python3.5",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "language": "python",
 "display_name": "Python 3 Spark 1.6 SystemML",

dusenberrymw

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>mwd.sleepMac</string>
    <key>ProgramArguments</key>
    <array>
        <string>/path/to/sleepMac.sh</string>
    </array>

dusenberrymw

Quick cheat sheet of helpful tmux commands

1. tmux new - Create and attach to a new session.
2. tmux new -s NAME_HERE - Create and attach to a new session named NAME_HERE.
3. CTRL-b, d - Detach (i.e. exit) from the currently-opened tmux session (alternatively, tmux detach). Note, this means press and hold CTRL, press b, release both, press d.
4. tmux ls - Show list of tmux sessions.
5. tmux a - Attach to the previously-opened tmux session.
6. tmux a -t NAME_HERE - Attach to the tmux session named NAME_HERE.
7. CTRL-d - Delete (i.e. kill) currently-opened tmux session (alternatively tmux kill-session).
8. CTRL-b, [ - Enter copy mode, and enable scrolling in currently-opened tmux session. Press q to exit.
9. CTRL-b, " - Split window horizontally (i.e. split and add a pane below).

dusenberrymw

Jupyter Tips & Tricks

Show available kernels:

• jupyter kernelspec list

Install new custom kernel (such as PySpark):

• Reference: https://ipython.readthedocs.io/en/latest/install/kernel_install.html

1. Install dummy kernel in home directory:

• ipython kernel install --prefix ~/my_custom_jupyter_kernels

2. Edit name of ~/my_custom_jupyter_kernels/share/jupyter/kernels/python3 folder as desired.

dusenberrymw

#!/bin/bash
#
# Copyright (c) 2013-2014 David Ingram
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:

dusenberrymw

Deep Learning Optimization

Currently, the literature generally supports mini-batch SGD (particularly using adaptive learning rate variants like RMSprop or Adam) as achieving the best performance in terms of both optimization metrics and computational time for large deep learning models.

The "Neural Networks Part 3: Learning and Evaluation" lecture of the CS231n course is a good reference -- see the "In practice..." and "Additional References" portions of the "Second-order methods" section. The first paper linked in that latter section, "Large Scale Distributed Deep Networks", is Google's 2012 paper introducing their distributed "DistBelief" system (replaced by TensorFlow after assessing its shortcomings), along with comparisons of a distri

dusenberrymw

Keras Tips & Tricks

fit_generator

• Can using either threading or multiprocessing for concurrent and parallel processing, respectively, of the data generator.
• In the threading approach (model.fit_generator(..., pickle_safe=False)), the generator can be run concurrently (but not parallel) in multiple threads, with each thread pulling the next available batch based on the shared state of the generator and placing it in a shared queue. However, the generator must be threadsafe (i.e. use locks at synchronization points).
• Due to the Python global interpreter lock (GIL), the threading option generally does not benefit from >1 worker (i.e. model.fit_generator(..., nb_worker=1) is best). One possible use case in which >1 threads could be beneficial is the presence of exceptionally long IO times, during which the GIL will be released to enable concurrency. Note also that TensorFlow's `session.run(

dusenberrymw

Cuda + cuDNN Tips & Tricks

Installation

CUDA + cuDNN Installation:

• Install Cuda (2 options):
  • Download from NVIDIA website. Will install to /Developer/NVIDIA/CUDA=#-# (macOS / OS X), with symlinks in /usr/local/cuda/.
  • Alternatively, use brew cask install cuda.
• Download CuDNN from NVIDIA website into local_cudnn_path, then:
  • sudo mkdir /usr/local/cudnn
  • sudo cp -r local_cudnn_path/* /usr/local/cudnn/

dusenberrymw

Grep Tips & Tricks

• grep -aisElr 'search string here' ./*
  • -a -> search files as strings (this finds strings in note annotations)
  • -i -> ignore case
  • -s -> suppress errors
  • -E -> use extended regular expressions
  • -l -> only show the filename(s); remove this to see the line where the search string was found
  • -r -> search recursively
• Use :grep for Grep searching within Vim.