Add native libraries to Apache Hadoop installation

ApacheHadoop_NativeLibs.adoc

Apache Hadoop - add native libraries

If native libraries are not available the following message is displayed with every hadoop command: hadoop checknative

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

• Clone hadoop source code

$ git clone https://github.com/apache/hadoop.git
$ cd hadoop

• Checkout the version 2.7.1 source

$ git checkout branch-2.7.1

• Install required dependencies - OSX: use brew or any other package manager

$ brew install cmake
$ brew install zlib
$ brew install protobuf
$ brew install snappy

• Build project and native dependencies with maven

$ mvn package -Pdist,native -DskipTests -Dtar

• Copy newly created libraries to the hadoop installation

$ mkdir -p $HADOOP_INSTALL/lib/native/osx
$ cp -r hadoop-dist/target/hadoop-2.7.1/lib/native/* $HADOOP_INSTALL/lib/native/osx

• Add shell variables either to the ~/.bash_profile, or $HADOOP_INSTALL/etc/hadoop/hadoop-env.sh

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_INSTALL/lib/native/osx”

Check if native libraries are available:

$ hadoop checknative

Warning

above rules do not cause bzip2 native libs to work in OSX.

hadoop-pipes related

error

ld: cannot link directly with dylib/framework, your binary is not an allowed client of /usr/lib/libcrypto.dylib for architecture x86_64

situation

• macOS Catalina
• openssl 1.1 already installed by brew

solution

Open and modify hadoop-tools/hadoop-pipes/pom.xml

• <openssl.prefix>/usr/local/opt/openssl</openssl.prefix>

rm -rf hadoop-tools/hadoop-pipes/target

note

Exporting OPENSSL_ROOT_DIR is not enough because of the following profile:

    <profile>
      <id>native</id>
      <activation>
        <activeByDefault>false</activeByDefault>
      </activation>
      <properties>
        <openssl.prefix></openssl.prefix>
      </properties>
      <build>
        <plugins>
          <plugin>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-maven-plugins</artifactId>
            <executions>
              <execution>
                <id>cmake-compile</id>
                <phase>compile</phase>
                <goals><goal>cmake-compile</goal></goals>
                <configuration>
                  <source>${basedir}/src</source>
                  <vars>
                    <JVM_ARCH_DATA_MODEL>${sun.arch.data.model}</JVM_ARCH_DATA_MODEL>
                    <OPENSSL_ROOT_DIR>${openssl.prefix} </OPENSSL_ROOT_DIR>
                  </vars>
                </configuration>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>

$openssl.prefix is going to override OPENSSL_ROOT_DIR