Apache Hadoop - add native libraries

If native libraries are not available the following message is displayed with every hadoop command: hadoop checknative

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Clone hadoop source code

$ git clone https://github.com/apache/hadoop.git
$ cd hadoop

Checkout the version 2.9.1 source

$ git checkout branch-2.9.1

Install required dependencies - OSX: use brew or any other package manager

$ brew install cmake
$ brew install zlib
$ brew install protobuf
$ brew install snappy
$ brew install homebrew/versions/protobuf250
$ echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.zshrc

export OPENSSL_ROOT_DIR=$(brew --prefix openssl)

Build project and native dependencies with maven

$ mvn package -Pdist,native -DskipTests -Dtar

Copy newly created libraries to the hadoop installation

$ mv $HADOOP_HOME/lib/native/ $HADOOP_HOME/lib/native_backup
$ mkdir -p $HADOOP_HOME/lib/native
$ cp -r hadoop-dist/target/hadoop-2.9.1/lib/native/* $HADOOP_HOME/lib/native

Check if native libraries are available:

$ hadoop checknative -a

Warning

above rules do not cause bzip2 native libs to work in OSX.

WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true hadoop-2.9.1/lib/native/libhadoop.dylib
zlib:    true /usr/lib/libz.1.dylib
snappy:  true /usr/local/lib/libsnappy.1.dylib
zstd  :  false
lz4:     true revision:10301
bzip2:   false
openssl: false build does not support openssl.

lihonosov/ApacheHadoop_NativeLibs.adoc

Apache Hadoop - add native libraries