bugcy013

This gist includes hive ql scripts to create an external partitioned table for Syslog
generated log files using regex serde; 
Usecase:  Count the number of occurances of processes that got logged, by year, month,
day and process.
 
Includes:
---------
Sample data and structure:           01-SampleDataAndStructure 
Data download:                       02-DataDownload
Data load commands:                  03-DataLoadCommands

bugcy013

This gist includes components of a oozie workflow - scripts/code, sample data 
and commands;  Oozie actions covered: shell action, email action

Action 1: The shell action executes a shell script that does a line count for files in a 
glob provided, and writes the line count to standard output
Action 2: The email action emails the output of action 1
 
 
Pictorial overview of job:
--------------------------

bugcy013

sudo apt-get install oozie oozie-client
sudo apt-get install mysql-server-5.1

mysql -u root -p password
create database oozie;
grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie';
grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie';
exit

sudo vim /etc/oozie/conf/oozie-site.xml

bugcy013

##########
# For verification, you can display the OS release.
##########
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.10
DISTRIB_CODENAME=oneiric
DISTRIB_DESCRIPTION="Ubuntu 11.10"

##########

bugcy013

Chunk data import / Incremental Import in Sqoop

bugcy013

# Add Cloudera RPM-GPG-KEY and repo
rpm --import http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera
rpm -ivh http://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm

# note: if you want to install a specific version,
# modify /etc/yum.repos.d/cloudera-cdh4.repo accordingly.
# For example, if you want to install 4.2.1, use the following baseurl:
# baseurl=http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/4.2.1/

# Install CDH4 httpfs Base

bugcy013

#!/bin/sh
#
# redis - this script starts and stops the redis-server daemon
#
# chkconfig:   - 85 15 
# description:  Redis is a persistent key-value database
# processname: redis-server
# config:      /etc/redis/redis.conf
# config:      /etc/sysconfig/redis
# pidfile:     /var/run/redis.pid

bugcy013

#!/bin/sh

# Credentials for a MySQL user with PROCESS, SUPER permissions
USERNAME=
PASSWORD=

# MySQL Server location
HOST=127.0.0.1
PORT=3306

bugcy013

#!/bin/bash

# update the local repo listing
sudo apt-get update -y
sudo apt-get upgrade -y

# install gcc, python tools, pip, atlas, and an 
sudo apt-get -y install build-essential \
                        g++ \
                        gcc \

bugcy013

# Install java and make sure JAVA_HOME is properly set.

# Make sure those dependencies are present:
sudo apt-get install g++ uuid-dev libtool autoconf automake

cd ~

# install zeromq
wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
tar -xzf zeromq-2.1.7.tar.gz