Kent Chiu KentChun33333

The Batch Normalization paper describes a method to address the various issues related to training of Deep Neural Networks. It makes normalization a part of the architecture itself and reports significant improvements in terms of the number of iterations required to train the network.

Issues With Training Deep Neural Networks

Internal Covariate shift

Covariate shift refers to the change in the input distribution to a learning system. In the case of deep networks, the input to each layer is affected by parameters in all the input layers. So even small changes to the network get amplified down the network. This leads to change in the input distribution to internal layers of the deep network and is known as internal covariate shift.

It is well established that networks converge faster if the inputs have been whitened (ie zero mean, unit variances) and are uncorrelated and internal covariate shift leads to just the opposite.

A comparison of Theano with other deep learning frameworks, highlighting a series of low-level design choices in no particular order.

Overview

Differentiation

Differentiation

Symbolic: Theano, CGT; Automatic: Torch, MXNet

Symbolic and automatic differentiation are often confused or used interchangeably, although their implementations are significantly different.

	# Remove anything linked to nvidia
	sudo apt-get remove --purge nvidia*
	sudo apt-get autoremove

	# Search for your driver
	apt search nvidia

	# Select one driver (the last one is a decent choice)
	sudo apt install nvidia-370

	#!/usr/bin/env python
	"""
	Example of using Keras to implement a 1D convolutional neural network (CNN) for timeseries prediction.
	"""

	from __future__ import print_function, division

	import numpy as np
	from keras.layers import Convolution1D, Dense, MaxPooling1D, Flatten
	from keras.models import Sequential

	""" Trains an agent with (stochastic) Policy Gradients on Pong. Uses OpenAI Gym. """
	import numpy as np
	import cPickle as pickle
	import gym

	# hyperparameters
	H = 200 # number of hidden layer neurons
	batch_size = 10 # every how many episodes to do a param update?
	learning_rate = 1e-4
	gamma = 0.99 # discount factor for reward

	//Install Macports.
	//Install aircrack-ng:
	sudo port install aircrack-ng
	//Install the latest Xcode, with the Command Line Tools.
	//Create the following symlink:
	sudo ln -s /System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport /usr/sbin/airport
	//Figure out which channel you need to sniff:
	sudo airport -s

	sudo airport en1 sniff [CHANNEL]

	#!/bin/bash

	# Bash script to install latest version of ffmpeg and its dependencies on Ubuntu 12.04 or 14.04
	# Inspired from https://gist.github.com/faleev/3435377

	# Remove any existing packages:
	sudo apt-get -y remove ffmpeg x264 libav-tools libvpx-dev libx264-dev

	# Get the dependencies (Ubuntu Server or headless users):
	sudo apt-get update

	from scipy.misc import imread, imresize

	from keras.layers import Input, Dense, Convolution2D, MaxPooling2D, AveragePooling2D, ZeroPadding2D, Dropout, Flatten, merge, Reshape, Activation
	from keras.models import Model
	from keras.regularizers import l2
	from keras.optimizers import SGD
	from googlenet_custom_layers import PoolHelper,LRN


	def create_googlenet(weights_path=None):

	"vundle

	set nocompatible

	filetype off

	set rtp+=~/.vim/bundle/Vundle.vim

	call vundle#begin()