bugcy013

#gitz

git clone https://blah
git checkout -b local_branch origin/remote_branch
git add files
git commit -m"What the hack is this?"
git push origin local_branch #this will create a remote branch with the name "local_branch"

#submit the pull request using the UI

bugcy013

#why dont you go and do something else instead of writing software? thank you!!

while read -r job_id status start_time user job_name priority _ _ used_mem _ need_mem _ ; do
  echo "$job_id $status $user $used_mem $(date -d@$(( $start_time / 1000 )))" ;
done < <(mapred job -list all 2>/dev/null | egrep RUNN) | sort -k 4 -n

#JobId State StartTime UserName Queue Priority UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM info

bugcy013

We encountered some issues using python processes with storm.

1) Storm uses the stdin and stdout of the python processes to communicate with them. This makes them almost impossible to debug with pdb.set_trace().

2) The local topology (and the real storm to a lesser extent) has issues killing the Python processes it creates. Some of our rouge processes pin the CPU. I find myself running something like this on my local machine.
    ps aux | grep python | awk '{ print $2 }' | xargs kill

3) We didn't want to install python packages in the system packages because it adds dependencies on the system storm is running on and we also have different topologies with different bolts that depend of different versions of packages. We install the dependencies (with buildout or virtualenv) in a relative path before creating the uberjar and deploying it, or you can also make the script storm calls a shell script that installs the dependencies locally then runs the python entry point. Either way, it is a little painful, espe

bugcy013

ssh IP -ldeploy

su root

/usr/sbin/visudo

93gg

deploy ALL=(ALL) NOPASSWD: ALL

bugcy013

brew update
# install xcode 4.6.1
brew install storm
brew install leiningen
git clone https://github.com/nathanmarz/storm-starter.git
cd storm-starter
lein deps
lein compile
java -cp $(lein classpath) storm.starter.ExclamationTopology
curl -O http://mirror.cc.columbia.edu/pub/software/apache/incubator/kafka/kafka-0.7.2-incubating/kafka-0.7.2-incubating-src.tgz

bugcy013

#!/usr/bin/env bash
useradd -r -U -s /bin/false -g storm storm

bugcy013

# Install java and make sure JAVA_HOME is properly set.

# Make sure those dependencies are present:
sudo apt-get install g++ uuid-dev libtool autoconf automake

cd ~

# install zeromq
wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
tar -xzf zeromq-2.1.7.tar.gz

bugcy013

#!/bin/bash

# update the local repo listing
sudo apt-get update -y
sudo apt-get upgrade -y

# install gcc, python tools, pip, atlas, and an 
sudo apt-get -y install build-essential \
                        g++ \
                        gcc \

bugcy013

#!/bin/sh

# Credentials for a MySQL user with PROCESS, SUPER permissions
USERNAME=
PASSWORD=

# MySQL Server location
HOST=127.0.0.1
PORT=3306

bugcy013

#!/bin/sh
#
# redis - this script starts and stops the redis-server daemon
#
# chkconfig:   - 85 15 
# description:  Redis is a persistent key-value database
# processname: redis-server
# config:      /etc/redis/redis.conf
# config:      /etc/sysconfig/redis
# pidfile:     /var/run/redis.pid