Rob Cowie robcowie

HDFS Notes

Moving files to and from HDFS

hdfs dfs -copyFromLocal file hdfs://path/to/dir/file
hdfs dfs -copyToLocal hdfs://path/to/dir/file file

Moving files within the cluster

Oozie Notes

Docs

Tools

Arbiter - Workflow generation from yaml

Custom Action Links

Configure S3 filesystem support for Spark on OSX

Homebrew installed Spark 1.6.x

Install hadoop to get the required jars (brew install hadoop)
Create a spark-env.sh (cp /usr/local/Cellar/apache-spark/1.6.1/libexec/conf/spark-env.sh.template /usr/local/Cellar/apache-spark/1.6.1/libexec/conf/spark-env.sh)
Set HADOOP_CONF_DIR in spark-env.sh (export HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/)
Add the required jars to the SPARK_CLASSPATH in spark-env.sh

EMR

Submit an S3DistCP step using awscli

# AMI 3.x
aws emr add-steps\
  --cluster-id j-xxxxxxxxxxx\
  --steps 'Type=CUSTOM_JAR,Name="s3distcpstep",Jar=/home/hadoop/lib/emr-s3distcp-1.0.jar,Args=["--src=s3://","--dest=s3://"]'

	import datetime as dt
	import operator as op


	def date_iterator(from_date, days, reverse=False):
	func = op.sub if reverse else op.add
	return (func(from_date, dt.timedelta(days=d)) for d in range(days))


	def date_range(from_date, to_date, inclusive=True):

	#!/bin/bash
	#
	# This script will check that both NameNodes are alive in HDFS HA
	# configuration and will force failover to the preferred NameNode.
	#
	# Author: Stas Alekseev <me@salekseev.com>
	#

	ACTIVE_NAMENODE=nn1
	STANDBY_NAMENODE=nn2

	# DELETE A TABLE IN THE HIVE METASTORE
	# BE CAREFUL! BACKUP THE DB BEFORE PROCEEDING!

	set @table_name = '';

	SELECT @tbl_id := TBl_ID FROM TBLS WHERE TBL_NAME = @table_name;

	-- Delete partition key vals
	DELETE pvk
	FROM PARTITION_KEY_VALS pvk

	example1 = 1118363648

	def numeric_to_ip(ip):
	parts = []
	while ip:
	parts.append(ip & 255)
	ip = ip >> 8
	return '.'.join([str(p) for p in reversed(parts)])

	import logging
	import time

	logger = logging.getLogger(__name__)


	class NoSuchActivityError(Exception):
	pass

	# export HADOOP_OPTS=-Xmx28G
	export HADOOP_CLIENT_OPTS="-Xmx2048m"

	hadoop distcp -p "hdfs://plat/data/level1/clicks/datehour=2016-12-*" "gs://data-events/data/level1/clicks/"