Info for running hadoop on OSX

hadoop_on_osx.md

brew install hadoop 2.7.1

• http://amodernstory.com/2014/09/23/installing-hadoop-on-mac-osx-yosemite/
• http://zhongyaonan.com/hadoop-tutorial/setting-up-hadoop-2-6-on-mac-osx-yosemite.html
• https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html

test

$ cd /usr/local/Cellar/hadoop/2.7.1/libexec 
$ bin/hdfs dfs -put etc/hadoop input
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'

hive.md

hive stuff

• https://noobergeek.wordpress.com/2013/11/09/simplest-way-to-install-and-configure-hive-for-mac-osx-lion/
• http://blog.joefallon.net/2013/10/install-mysql-on-mac-osx-using-homebrew/

init-hdfs.sh

# hadoop 2.7.1
hdfs dfs -mkdir /tmp;
hdfs dfs -chmod -R 1777 /tmp;
hdfs dfs -mkdir -p /tmp/hadoop-yarn/staging;
hdfs dfs -chmod -R 1777 /tmp/hadoop-yarn/staging;
hdfs dfs -mkdir -p /tmp/hadoopyarn/staging/history/done_intermediate;
hdfs dfs -chmod -R 1777 /tmp/hadoopyarn/staging/history/done_intermediate;
hdfs dfs -chown -R mapred:mapred /tmp/hadoop-yarn/staging;
hdfs dfs -mkdir -p /var/log/hadoop-yarn;
hdfs dfs -chown hadoop:hadoop /var/log/hadoop-yarn;
hdfs dfs -mkdir -p /user/$USER;
hdfs dfs -chown $USER /user/$USER;