ee08b397

sudo apt-get remove scala-library scala
sudo wget www.scala-lang.org/files/archive/scala-2.10.4.deb
sudo dpkg -i scala-2.10.4.deb
sudo apt-get update
sudo apt-get install scala
wget http://scalasbt.artifactoryonline.com/scalasbt/sbt-native-packages/org/scala-sbt/sbt/0.12.4/sbt.deb
sudo dpkg -i sbt.deb
sudo apt-get update
sudo apt-get install sbt

ee08b397

#!/usr/bin/env python
from __future__ import with_statement
import os
import re
import shutil
import subprocess
import sys
import tempfile

ee08b397

online install svn
Help -> Install New Software

change tab to spaces
https://alextheedom.wordpress.com/code-and-stuff/how-do-i-change-eclipse-to-use-spaces-instead-of-tabs/

change tab to 2
Window -> Preferences -> Java -> Code Style -> Formatter

Set author

ee08b397

Settings > Search "privacy" > content settings > Manage exceptions

add your site by regex, e.g., learn.com, add [*.]learn.com

ee08b397

#!/usr/bin/env bash

# global parameters
g_tmp_folder="ncdc_tmp";
g_output_folder="ncdc_data";

g_remote_host="ftp3.ncdc.noaa.gov";
g_remote_path="pub/data/noaa";

ee08b397

Setting up Flume NG, listening to syslog over UDP, with an S3 Sink

My goal was to set up Flume on my web instances, and write all events into s3, so I could easily use other tools like Amazon Elastic Map Reduce, and Amazon Red Shift.

I didn't want to have to deal with log rotation myself, so I setup Flume to read from a syslog UDP source. In this case, Flume NG acts as a syslog server, so as long as Flume is running, my web application can simply write to it in syslog format on the specified port. Most languages have plugins for this.

At the time of this writing, I've been able to get Flume NG up and running on 3 ec2 instances, and all writing to the same bucket.

Install Flume NG on instances

ee08b397

How to create a MultiNode - MultiBroker Cluster for Kafka on AWS

PreRequisites :

1. Kafka Binary files : http://kafka.apache.org/downloads.html

2. Atleast 2 AWS machines : AWS EMR or EC2 will be preferable

3. A Kafka Manager Utility to watch up the Cluster : https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem

ee08b397

Best way to create Mutliple files into a single RDD
==================================

val fileRDD = sc.textFile(filename).repartition(1)

Where the filename is the location of your directory only.

ee08b397

LinSSID is a graphical and functional wifi scanner for Ubuntu Linux, which is similar to iwscanner and Inssider. It can be easily installed in Ubuntu and Linux Mint via PPA repository. (Kismet didn't work for my Ubuntu 15)

sudo add-apt-repository -y ppa:wseverin/ppa

sudo apt-get update; sudo apt-get -y install linssid

ee08b397

After reboot, gnome got stuck right after the purple "ubuntu" with several dots below, then only one non-flicking cursor on the top left. Then into recovery mode, try startx it says

X: cannot stat /etc/X11/X (No such file or directory), aborting

Fix

sudo dpkg-reconfigure xserver-xorg-lts-utopic

This restores the symlink /etc/X11/X -> /usr/bin/Xorg. I've tried to modify lightdm to gdm doesn't work. And xorg dpkg reconfigure doesn't work because I don't have it installed. I found I have xserver-xorg-lts-utopic.