Skip to content

Instantly share code, notes, and snippets.

@Zejnilovic
Created May 20, 2019 15:32
Show Gist options
  • Save Zejnilovic/97f52d469b78e7426a2cd2bd71cf2ef3 to your computer and use it in GitHub Desktop.
Save Zejnilovic/97f52d469b78e7426a2cd2bd71cf2ef3 to your computer and use it in GitHub Desktop.
A script to build and use Hadoop Native libraries on Mac
# !!CHANGE THIS!!
HVERSION=2.7.5
#!! AND LET IT RUN
brew install gcc autoconf automake libtool cmake snappy gzip bzip2 zlib openssl
cd ~
mkdir -p tmp
cd ~/tmp
wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
tar -xzf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure
make
make check
make install
protoc --version
cd /usr/local/include
ln -s ../opt/openssl/include/openssl .
cd ~/tmp
git clone https://github.com/apache/hadoop.git
cd hadoop
git checkout branch-$HVERSION
mvn package -Pdist,native -DskipTests -Dtar
cp -R hadoop-dist/target/hadoop-$HVERSION/lib $HADOOP_HOME
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native"
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:${HADOOP_HOME}/lib/native
hadoop checknative -a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment