tensorflow_centos6.md

TensorFlow with a non-system GCC

These instructions were written for our system running CentOS 6.6 using the following SHA of Bazel and TensorFlow:

• Bazel : c553e35976582ca861040dac3af12c5ed9f11ede
• TensorFlow : 3972c791b9f4d9a61b9ad6399b481df396f359ff

Requirements

• GCC 4.6.x or greater
• Python 2.7.x or greater
• binutils
• Java 1.7 or greater
• Git

Install GCC

You will need a GCC that supports some of the features of C++11. Here is a list of features I found so far (to be updated):

• nullptr

Compiling GCC is easy with the right instructions. I recommend the ones on the GNU GCC wiki.

The installation directory of GCC from now on will be referred to as $GCC_PREFIX in this document. Replace this placeholder with your own path each time it appears.

Notes:

• Make sure that MPC, GMP and MPFR libraries are either provided by your OS, or built as part of GCC build process using the ./contrib/download_prerequisites script. See Support Libraries section of the GNU GCC wiki. Bazel and TensorFlow build process will call some of GCC binary directly (i.e.: cc1plus) and without any predefined environment variables. If the support libraries can only be found using LD_LIBRARY_PATH, the build process will fail.

Install Python 2.7

You will need Python 2.7.x. I recommend following the instructions from Beyond Linux From Scratch, to which I added some notes.

Notes:

• It does not matter wether you use the GCC that was compiled previously or not.
• If the installation prefix is different than /usr. Add the following at the end of the configure command, where $PYTHON_PREFIX is the Python installation directory.

LDFLAGS='-Wl,-rpath=$PYTHON_PREFIX/lib'

This will guarantee that Python can find its shared library libpython2.7.so.1.0 without defining $LD_LIBRARY_PATH or messing with the system shared library cache.

• Install virtualenv with this Python.

Install binutils

TODO

$BINUTILS_PREFIX

Install Swig

TODO

$SWIG_PREFIX

Install JDK

TODO

$JAVA_HOME

Build Bazel

Version: 0.1.4
SHA: c553e35976582ca861040dac3af12c5ed9f11ede

export CC=$GCC_PREFIX/bin/gcc
export CXX=$GCC_PREFIX/bin/g++
export CPATH=$GCC_PREFIX/include:$CPATH
export LIBRARY_PATH=$GCC_PREFIX/lib64:$GCC_PREFIX/lib:$LIBRARY_PATH
export LDFLAGS="-Wl,-rpath=$GCC_PREFIX/lib64"

# Untar Bazel where you want to install it and rename the folder if you want.
# This folder path will be referred to as $BAZEL_PREFIX.
tar xvf 0.1.4.tar.gz
cd bazel-0.1.4

cp tools/cpp/CROSSTOOL{,.orig}
sed -i "s;/usr/bin/gcc;$GCC_PREFIX/bin/gcc;g" tools/cpp/CROSSTOOL
sed -i "s;/usr/bin/cpp;$GCC_PREFIX/bin/cpp;g" tools/cpp/CROSSTOOL
sed -i "s;/usr/bin/gcov;$GCC_PREFIX/bin/gcov;g" tools/cpp/CROSSTOOL
sed -i "s;/usr/lib/gcc;$GCC_PREFIX/lib/gcc;g" tools/cpp/CROSSTOOL
# Modify the path to ar, compat-ld, and dwp if binutils is not installed in /usr/bin
# append the following line to tools/cpp/CROSSTOOL around line 95
#   cxx_builtin_include_directory: "$GCC_PREFIX/include/c++/$GCC_VERSION"
# append the following near a library_path line
#   linker_flag: \"-Wl,-rpath=$GCC_PREFIX/lib64\"" 
sed -i "324s;$;\ -Wl,-rpath=$GCC_PREFIX/lib64;" scripts/bootstrap/compile.sh
sed -i "329s;$;\ -Wl,-rpath=$GCC_PREFIX/lib64;" scripts/bootstrap/compile.sh
./compile.sh

Build TensorFlow

SHA: 3972c791b9f4d9a61b9ad6399b481df396f359ff

export CC=$GCC_PREFIX/bin/gcc
export CXX=$GCC_PREFIX/bin/g++
export CPATH=$GCC_PREFIX/include:$CPATH
export LIBRARY_PATH=$GCC_PREFIX/lib64:$GCC_PREFIX/lib:$LIBRARY_PATH
export LDFLAGS="-Wl,-rpath=$GCC_PREFIX/lib64"

# git version >= 1.7.4, for older version use --recursive
git clone --recurse-submodules https://github.com/tensorflow/tensorflow
cd tensorflow
git checkout 3972c791b9f4d9a61b9ad6399b481df396f359ff

$PYTHON_PREFIX/bin/virtualenv venv
source venv/bin/activate
pip install numpy wheel argparse

sed -i "s;swig;$SWIG_PREFIX/bin/swig;g" tensorflow/tools/swig/swig.sh

# Required when building with GCC < 5.2 because of a problem involving
# function name mangling with third party library eigen3
sed -i 's;"-DEIGEN_AVOID_STL_ARRAY";"-DEIGEN_AVOID_STL_ARRAY","-fabi-version=6";g' tensorflow/tensorflow.bzl
sed "/name = \"python_op_gen_main\"/a\ \ \ \ copts = ['-fabi-version=6']," tensorflow/python/BUILD

sed -i "s;/usr/bin/cpp;$GCC_PREFIX/bin/cpp;g" third_party/gpus/crosstool/CROSSTOOL
sed -i "s;/usr/bin/gcov;$GCC_PREFIX/bin/gcov;g" third_party/gpus/crosstool/CROSSTOOL
sed -i "s;/usr/lib/gcc;$GCC_PREFIX/lib/gcc;g" third_party/gpus/crosstool/CROSSTOOL
sed -i "57i\ \ cxx_builtin_include_directory: \"$GCC_PREFIX/include\"" third_party/gpus/crosstool/CROSSTOOL
sed -i "116i\ \ linker_flag: \"-Wl,-rpath=$GCC_PREFIX/lib64\"" third_party/gpus/crosstool/CROSSTOOL
sed -i "117i\ \ linker_flag: \"-Wl,-rpath=$CUDA_PREFIX/lib64\"" third_party/gpus/crosstool/CROSSTOOL

sed -i "1 s;^.*$;#!$VIRTUAL_ENV/bin/python;" third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc
sed -i "s;/usr/bin/gcc;$GCC_PREFIX/bin/gcc;g" third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc

./configure
$BAZEL_PREFIX/output/bazel build -c opt --cuda --copt="-I$PYTHON_PREFIX/include/python2.7" --copt="-I$VIRTUAL_ENV/lib/python2.7/site-packages/numpy/core/include" --copt='-fabi-version=6' --linkopt='-lrt'  //tensorflow/tools/pip_package:build_pip_package --verbose_failures --jobs 40
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg