How to install Tensorflow with custom GCC

How to install the latest version of Tensorflow (2.3.0) on a machine where the default gcc is too old or too new and you don't have root access

This solves the following issue when compiling Tensorflow:

ERROR: /home/users/*/tensorflow/tensorflow/core/framework/BUILD:1324:1: ProtoCompile tensorflow/core/framework/op_def.pb.h failed (Exit 1)
bazel-out/host/bin/external/com_google_protobuf/protoc: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by bazel-out/host/bin/external/com_google_protobuf/protoc)
bazel-out/host/bin/external/com_google_protobuf/protoc: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by bazel-out/host/bin/external/com_google_protobuf/protoc)
bazel-out/host/bin/external/com_google_protobuf/protoc: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by bazel-out/host/bin/external/com_google_protobuf/protoc)
bazel-out/host/bin/external/com_google_protobuf/protoc: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by bazel-out/host/bin/external/com_google_protobuf/protoc)

Download, compile and install Python 3.8 as user (if needed)

cd $HOME
wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz
tar zxvf Python-3.8.2.tgz
cd Python-3.8.2
./configure --prefix=$HOME
make
make install

Refer to: https://docs.python.org/3/using/unix.html

Download, compile and install GCC as user

cd $HOME
wget https://ftp.gnu.org/gnu/gcc/gcc-7.3.0/gcc-7.3.0.tar.gz
tar zxvf gcc-7.3.0.tar.gz
cd gcc-7.3.0
./contrib/download_prerequisites
./configure --disable-multilib --prefix $HOME/gcc
make -j 16
make install

Download JAVA SDK (needed by Bazel)

cd $HOME
wget https://download.java.net/java/GA/jdk14.0.1/664493ef4a6946b186ff29eb326336a2/7/GPL/openjdk-14.0.1_linux-x64_bin.tar.gz
tar zxfv openjdk-14.0.1_linux-x64_bin.tar.gz
export JAVA_HOME=~/jdk-14.0.1/

Download, compile and install Bazel as a user

Install binary version of Bazel first.

wget https://github.com/bazelbuild/bazel/releases/download/3.1.0/bazel-3.1.0-dist.zip
unzip bazel-3.1.0-dist.zip -d bazel-3.1.0
cd bazel-3.1.0

Apply the following patch to third_party/grpc/bazel/generate_cc.bzl: https://github.com/bazelbuild/bazel/pull/11860/files

export GCC_HOST_COMPILER_PATH=$HOME/gcc/bin/gcc
export GCC_HOST_COMPILER_PREFIX=$HOME/gcc/bin
export CXX=$HOME/gcc/bin/gcc
export CC=$HOME/gcc/bin/gcc
export LD_LIBRARY_PATH=$HOME/gcc/lib64
export LDFLAGS="-L$HOME/gcc/lib -L$HOME/gcc/lib64"
export CXXFLAGS="-L$HOME/gcc/lib -L$HOME/gcc/lib64"

env BAZEL_LINKOPTS=-static-libstdc++:-static-libgcc BAZEL_LINKLIBS=-l%:libstdc++.a:-lm bash ./compile.sh
mkdir $HOME/bin
cp output/bazel $HOME/bin

Refer to: tensorflow/tensorflow#38718 (comment)

Install CUDA SDK

Refer to: https://stackoverflow.com/questions/39379792/install-cuda-without-root

Install binutils 2.34

cd $HOME	
https://ftp.gnu.org/gnu/binutils/binutils-2.34.tar.gz	
tar zxvf binutils-2.34.tar.gz	
cd binutils-2.34	
./configure --prefix=$HOME/binutils	
make -j 16 MAKEINFO=true	
make install MAKEINFO=true

Refer to: tensorflow/tensorflow#38718

Download Tensorflow source

cd $HOME
git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
git checkout v2.3.1

Configure Tensorflow

cd $HOME/tensorflow
./configure

Compile Tensorflow

export PATH=~/bin:$PATH
export JAVA_HOME=~/jdk-14.0.1/
export GCC_HOST_COMPILER_PATH=$HOME/gcc/bin/gcc
export GCC_HOST_COMPILER_PREFIX=$HOME/gcc/bin
export CXX=$HOME/gcc/bin/gcc
export CC=$HOME/gcc/bin/gcc
export LD_LIBRARY_PATH=$HOME/gcc/lib64
export LDFLAGS="-L$HOME/gcc/lib -L$HOME/gcc/lib64"
export CXXFLAGS="-L$HOME/gcc/lib -L$HOME/gcc/lib64"

env BAZEL_LINKOPTS=-static-libstdc++:-static-libgcc BAZEL_LINKLIBS=-l%:libstdc++.a:-lm BAZEL_CXXOPTS=-std=gnu++0x bazel build //tensorflow/tools/pip_package:build_pip_package --action_env="LD_LIBRARY_PATH=${LD_LIBRARY_PATH}" --verbose_failures

Refer to: bazelbuild/bazel#4137 (comment)

Solving compilation issues.

Following is the list of issues which can occur during Tensorflow compilation and the workarounds solving them:

Solve the issue with protoc linked to incorrect version of stdc++ by modifing third_party/gpus/crosstool/cc_toolchain_config.bzl.tpl.

Change line:

flag_group(flags = ["-lc++" if cpu == "darwin" else "-lstdc++"]),

to:

flag_group(flags = ["-lc++" if cpu == "darwin" else "-l:libstdc++.a"]),

Solve the issue with objcopy by modifing third_party/gpus/crosstool/cc_toolchain_config.bzl.tpl and point it to $HOME/binutils/bin/objcopy
If you encounter with issues while running ar tool you may need to do the following: Create the following ~/gcc/bin/ar file and make it executable:

> cd ~/gcc/bin/
> cat > ar
#!/bin/bash
GCC_AR_PATH=$HOME/gcc/bin
if [ $1 = "-M" ]; then
   exec "$HOME/gcc/bin/gcc-ar" $1
else
   ARGS=$1
   FILENAME=${ARGS:1}
   $HOME/gcc/bin/gcc-ar $(<$FILENAME)
fi
^D
> chomd +x ar

Build the pip package

cd ~/tensorflow
./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

Install Tensorflow and verify GPU is available:

> mkdir test && cd test
> python3 -m venv venv
> source venv/bin/activate
> pip install --upgrade pip

> pip install ~/tmp/tensorflow_pkg/tensorflow-2.3.1-cp38-cp38-linux_x86_64.whl 

> export LD_LIBRARY_PATH=PATH_TO_YOUR_CUDA_LIB64
> python3
import tensorflow as tf
tf.config.list_physical_devices('GPU')

jakublipinski/install_tensorflow_custom_gcc.md