Roberto Salas robsalasco

Setting up a SGE cluster on a single Amazon EC2 machine

The gist here provides a script to automate the process of installing Sun Grid Engine (SGE) on a single EC2 machine.

Motivation

SGE is a job scheduler for a computing cluster. This usually involves a cluster of multiple machines. However for many applications we don't need a massive computing cluster and a cluster of 8-30 nodes would be sufficient. In this tutorial we set-up SGE on a single amazon EC2 machine. The reasons for doing so are as follows:

Automation: Setting-up a cluster with SGE is fairly involved as it requires multiple machines communicating with each other and having some shared memory. A single machine with multiple cores is already a simple cluster where the memory is shared across cores.
Moderate Size: Amazon EC2 instances provide a variety of computing options with the number of cores ranging from 1 to 128.
Cost: The On-Demand price structure of AWS makes this a relatively cheap option. Further cost redu

Creating Shakespearean n-grams with just the command-line and regexes

This is a quick example showing how to use regexes to find tri-grams in Shakespeare...well, 570,872 of them, anyway, if we do some basic filtering of non-dialogue.

Though tokenization and n-grams should typically be done using a proper natural language processing framework, it's possible to do in a jiffy from the command-line, using standard Unix tools and ack, the better-than-grep utility.

What are n-grams?

Pushing to Multiple Git Repos

If a project has to have multiple git repos (e.g. Bitbucket and Github) then it's better that they remain in sync.

Usually this would involve pushing each branch to each repo in turn, but actually Git allows pushing to multiple repos in one go.

If in doubt about what git is doing when you run these commands, just

Install caffe with python 3.5 and pyenv

Tested on Ubuntu 14.04.

Setting up a new python environment using pyenv

Install desired version of python 3 (e.g. 3.5.1). Make sure to use the --enable-shared flag to generate python shared libraries, which will later be linked to.

env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.5.1

What this?

So one of the painful points of using docker on OS X is that you need to run a virtualbox VM, which often suffers from performance issues. With xhyve, a OS X virtualization system, and docker-machine-xhyve you can now have docker use the native OS X hypervisor to run containers.

No more dealing with virtualbox shenanigans!

In this script, I've also set up a way to autoconfigure terminal sessions to load docker's environment vars (dependent on docker-machine) so you do not have to run eval $(docker-machine env whatever) every time you open a new terminal window.

	var PaktPub = (function () {
	"use strict";
	var freeLearningURL = 'https://www.packtpub.com/packt/offers/free-learning',
	logoutURL = 'https://www.packtpub.com/logout';
	var notify = function (message, timeout) {
	var useNotificationAPI = false;
	if ("Notification" in window) {
	if (Notification.permission === "granted") {
	useNotificationAPI = true;
	} else if (Notification.permission !== 'denied') {

	#!/usr/bin/Rscript

	###########################################################################################
	# Function: getVoronoiMap.R
	# Author: Manuel Garcia-Herranz
	# Date: 16 Aug 2016
	#
	# Small script to calculate the voronoi tesellation of a list of points, constraining it to the outer perimeter of a shapefile
	#
	# Usage: ./../bin/getVoronoiMap.R [options]

	#!/bin/sh
	# Install brew
	/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
	# Apple hides old versions of stuff at https://developer.apple.com/download/more/
	# Install the latest XCode (8.0).
	# We used to install the XCode Command Line Tools 7.3 here, but that would just upset the most recent versions of brew.
	# So we're going to install all our brew dependencies first, and then downgrade the tools. You can switch back after
	# you have installed caffe.
	# Install CUDA toolkit 8.0 release candidate
	# Register and download from https://developer.nvidia.com/cuda-release-candidate-download

	/*
	The purpose of this script is to daily check Packt's Free Book offer
	and notify via e-mail with the title of the book and link to download page.

	Author: Douglas Matoso - www.dmatoso.com

	Instructions:
	1) Create a new Goggle Script at http://script.google.com
	2) Paste the code below, putting your email in the "email" var.
	3) Execute the checkBook function to test (the script may ask an one time permission)

	# code used for the following map : http://datagistips.blogspot.fr/

	library(raster)
	library(rgdal)
	library(rgeos)

	clc = raster("DATAS/CLC12_RIDF_RGF.tif") # here is the 200m Corine Land Cover GeoTiff
	reg = readOGR("DATAS", "idf_geofla") # ile de france GéoFla Departments

	# RECLASS