Skip to content

Instantly share code, notes, and snippets.

@joshbode
Created August 20, 2012 11:20
Show Gist options
  • Save joshbode/3403318 to your computer and use it in GitHub Desktop.
Save joshbode/3403318 to your computer and use it in GitHub Desktop.
Hack Hadoop (1.0.3) to compile native libraries on OS X (so rmr does not fall back to non-native)
deps:
brew install md5sha1sum automake snappy hadoop
tweak OS X:
# export JAVA_HOME=$(/usr/libexec/java_home)
hack jvm:
# sudo mkdir -p ${JAVA_HOME}/lib/{i386,x86_64}/server
# sudo ln -s ${JAVA_HOME}/{../Libraries,lib/i386/server}/libjvm.dylib
# sudo ln -s ${JAVA_HOME}/{../Libraries,lib/x86_64/server}/libjvm.dylib
# sudo ln -s ${JAVA_HOME}{,/jre}
patch:
src/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c:76
replace:
//if(setnetgrent(cgroup) == 1) {
setnetgrent(cgroup)
build:
# export ac_cv_libname_z='"/usr/lib/libz.dylib"'
# export ac_cv_libname_snappy='"/usr/local/lib/libsnappy.dylib"'
# ant -Dcompile.native=true
install:
# cp build/native/Mac_OS_X-x86_64-64/lib/libhadoop.* $HADOOP_HOME/libexec/lib/native/Mac_OS_X-x86_64-64
R:
# brew install thrift
# git clone https://github.com/RevolutionAnalytics/RHadoop
> install.packages(c('RJSONIO', 'itertools', 'digest', 'rJava'))
> install.packages(paste("~/repos/RHadoop", c('rhdfs', 'rhbase', 'rmr'), 'pkg', sep='/'), repos=NULL, type="source")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment