Add native libraries to Apache Hadoop installation

ApacheHadoop_NativeLibs.adoc

Apache Hadoop - add native libraries

If native libraries are not available the following message is displayed with every hadoop command: hadoop checknative

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

• Clone hadoop source code

$ git clone https://github.com/apache/hadoop.git
$ cd hadoop

• Checkout the version 2.7.1 source

$ git checkout branch-2.7.1

• Install required dependencies - OSX: use brew or any other package manager

$ brew install cmake
$ brew install zlib
$ brew install protobuf
$ brew install snappy

• Build project and native dependencies with maven

$ mvn package -Pdist,native -DskipTests -Dtar

• Copy newly created libraries to the hadoop installation

$ mkdir -p $HADOOP_INSTALL/lib/native/osx
$ cp -r hadoop-dist/target/hadoop-2.7.1/lib/native/* $HADOOP_INSTALL/lib/native/osx

• Add shell variables either to the ~/.bash_profile, or $HADOOP_INSTALL/etc/hadoop/hadoop-env.sh

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_INSTALL/lib/native/osx”

Check if native libraries are available:

$ hadoop checknative

Warning

above rules do not cause bzip2 native libs to work in OSX.

The default version of protobuf installed by homebrew is 2.6.1 while 2.5.0 is what is necessary to build the native libraries. If you attempt to build using these instructions exactly you will encounter an error building the 'Apache Hadoop Common' project.

Install protobuf 2.5.0 instead by running
brew install homebrew/versions/protobuf250
From there you can follow the instructions exactly.

export OPENSSL_ROOT_DIR=$(brew --prefix openssl) might be helpful. If you are having trouble with building hadoop-pipes.

If you are getting openssl false in the hadoop checknative command,

copy openssl related files and build again.

ln -sf /usr/local/opt/openssl/lib/libcrypto.dylib /usr/local/lib/libcrypto.dylib
ln -sf /usr/local/opt/openssl/lib/libssl.dylib /usr/local/lib/lib/libssl.dylib # might not exist
ln -sf /usr/local/opt/openssl/include/openssl /usr/local/include/openssl