The number of instances to use in your job flow is application-dependent and should be based on both the amount of resources required to store and process your data and the acceptable amount of time for your job to complete. As a general guideline, we recommend that you limit 60% of your disk space to storing the data you will be processing, leaving the rest for intermediate output. Hence, given 3x replication on HDFS, if you were looking to process 5 TB on m1.xlarge instances, which have 1,690 GB of disk space, we recommend your cluster contains at least (5 TB * 3) / (1,690 GB * .6) = 15 m1.xlarge core nodes. You may want to increase this number if your job generates a high amount of intermediate data or has significant I/O requirements. You may also want to include additional task nodes to improve processing performance. See Amazon EC2 Instance Types for details on local instan
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Terminal output control (http://www.termsys.demon.co.uk/vtansi.htm) | |
TC='\e[' | |
CLR_LINE_START="${TC}1K" | |
CLR_LINE_END="${TC}K" | |
CLR_LINE="${TC}2K" | |
# Hope no terminal is greater than 1k columns | |
RESET_LINE="${CLR_LINE}${TC}1000D" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
parse_git_branch() { | |
BRANCH=`git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/\1/'` | |
case $BRANCH in | |
'master' ) | |
BRANCH="\e[1m\e[33m${BRANCH}\e[\0m" | |
;; | |
'(no branch)' ) | |
BRANCH="\e[1m\e[31m${BRANCH}\e[\0m" | |
;; | |
*) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package main | |
import ( | |
"fmt" | |
"net/http" | |
) | |
func handler(w http.ResponseWriter, r *http.Request) { | |
fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:]) | |
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#include <errno.h> | |
#include <termios.h> | |
#include <unistd.h> | |
int | |
set_interface_attribs (int fd, int speed, int parity) | |
{ | |
struct termios tty; | |
memset (&tty, 0, sizeof tty); | |
if (tcgetattr (fd, &tty) != 0) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package main | |
import( | |
"fmt" | |
"math" | |
) | |
func main() { | |
fmt.Printf("PI approx\n") | |
piValue := float64(0) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
PS=/bin/ps | |
GREP=/bin/grep | |
PROCESS_NAME=wftpserver | |
START="/etc/init.d/wftpserver start" | |
NOP=/bin/true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sudo easy_install pip # PIP is recommended for installation | |
sudo -s # Creates a new sudo shell | |
export ARCHFLAGS="-arch i386 -arch x86_64" # Need to set some GCC flags | |
pip install fabric # Get the install on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
export EC2_AMITOOL_HOME=~/Amazon/tools/ami/ | |
export PATH=${PATH}:~/Amazon/tools/ami/bin | |
export EC2_HOME=~/Amazon/tools/api/ | |
export PATH=${PATH}:~/Amazon/tools/api/bin | |
export AWS_CLOUDWATCH_HOME=~/Amazon/tools/cloud-watch | |
export PATH=${PATH}:~/Amazon/tools/elastic-load-balancing/bin |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"myapp": { | |
"master": { | |
"master_0": { | |
"host": "localhost", | |
"port": "3306" | |
} | |
}, | |
"slave": { | |
"slave_0": { |