Skip to content

Instantly share code, notes, and snippets.

@sunsided

sunsided/build-tf-all.md

Last active October 9, 2021 02:08
Show Gist options
  • Save sunsided/824394cfaf87c6718697e0fda3bb229c to your computer and use it in GitHub Desktop.
Save sunsided/824394cfaf87c6718697e0fda3bb229c to your computer and use it in GitHub Desktop.
Download ZIP
Build TensorFlow 1.8 with XLA, MKL, CUDA, cuDNN and TensorRT
Raw
build-tf-all.md

Building TensorFlow from Source (non-interactive mode)

These scripts are meant to provide a starting point for easily (and automatically) building TensorFlow on different Ubuntu versions.

The scripts follow the schema:

  • build-tf-cudaX-cudnnY.sh
  • build-tf-nogpu-ARCHITECTURE.sh

To use them, source them, then follow the on-screen instructions, e.g.

source ~/build-tf-cuda9.2-cudnn7.3.1-tensorrt4.sh
./configure
bazel build $BUILD_OPTIONS //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

Note that in some cases it might be required to force bazel to see the installed libraries. For this, you might want to adjust LD_LIBRARY_PATH prior to building, e.g. like so:

export LD_LIBRARY_PATH=/usr/local/cuda-9.2/lib64:/opt/cudnn7.3.1-cuda9.2/lib64:/opt/TensorRT-4.0.1.6-cuda9.2-cudnn7.1/lib:$LD_LIBRARY_PATH
Raw
build-tf-cuda9.0-cudnn7.sh
# Environment variables for building TensorFlow 1.8 on Ubuntu 16.04
#
# CPU type (see https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html):
# - native
#
# This assumes:
# - CUDA 9.0
# - cuDNN 7.0
# - TensorRT 4.0.0.3 (RC)
# - ComputeCPP 0.7.0
# - libibverbs-dev
# - librdmacm-dev
#
# Source this file, then follow the on-screen instructions.
# CPU architecture
export CPU_ARCH=native
# Python
export PYTHON_BIN_PATH=/opt/conda/envs/tensorflow-1.8/bin/python
export PYTHON_LIB_PATH=/opt/conda/envs/tensorflow-1.8/lib/python3.6/site-packages
# GCC
export HOST_C_COMPILER=/usr/bin/gcc-5
export HOST_CXX_COMPILER=/usr/bin/g++-5
export GCC_HOST_COMPILER_PATH=/usr/bin/gcc-5
export CC_OPT_FLAGS=-march=$CPU_ARCH
# Compilation
export TF_DOWNLOAD_CLANG=0
# Memory allocator
export TF_NEED_JEMALLOC=1
# Storage and input
export TF_NEED_S3=1
export TF_NEED_GCP=1
export TF_NEED_HDFS=1
export TF_NEED_KAFKA=1
# OpenCL
export TF_NEED_COMPUTECPP=1
export TF_NEED_OPENCL_SYCL=1
export TF_NEED_OPENCL=1
export COMPUTECPP_TOOLKIT_PATH=/opt/ComputeCpp-CE-0.7.0-Ubuntu-16.04-x86_64/
# CUDA
export TF_NEED_CUDA=1
export TF_CUDA_CLANG=0
export TF_NCCL_VERSION=1.3
export TF_CUDA_VERSION=9.0
export CUDA_TOOLKIT_PATH=/usr/local/cuda-$TF_CUDA_VERSION
# CUDA Compute Capabilities
# https://developer.nvidia.com/cuda-gpus
#
# - 3.7: K80
# - 5.2: Titan X, 980 Ti
# - 5.3: Jetson TX1
# - 6.0: Tesla P100
# - 6.1: GTX 1080 Ti, GTX 1070, Titan XP
# - 7.0: Tesla V100
export TF_CUDA_COMPUTE_CAPABILITIES=5.2,6.1
# cuDNN
export TF_CUDNN_VERSION=7
export CUDNN_INSTALL_PATH=/opt/cudnn$TF_CUDNN_VERSION
# TensorRT
export TF_NEED_TENSORRT=1
export TF_TENSORRT_VERSION=4.0.0.3
export TENSORRT_INSTALL_PATH=/opt/TensorRT-4.0.0.3
# MPI
export TF_NEED_MPI=0
# Android
export TF_SET_ANDROID_WORKSPACE=0
# Memory access GDR/VERBS
# https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/gdr/README.md
export TF_NEED_GDR=1
export TF_NEED_VERBS=1
# XLA JIT: Accelerated Linear Algebra
export TF_ENABLE_XLA=1
# Documentation hints
echo "Targeting $CPU_ARCH architecture."
echo "To build TensorFlow, run:"
echo ""
echo " ./configure"
echo " bazel build --config=opt --config=monolithic --config=mkl --config=cuda //tensorflow/tools/pip_package:build_pip_package"
echo " bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg"
echo ""
Raw
build-tf-cuda9.1-cudnn7.1.sh
# Environment variables for building TensorFlow 1.8 on Ubuntu 18.04
#
# CPU type (see https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html):
# - native
#
# This assumes:
# - CUDA 9.0
# - cuDNN 7.0
# - ComputeCPP 0.7.0
# - TensorRT 4.0.0.3 (RC)
# - libibverbs-dev
# - librdmacm-dev
#
# Source this file, then follow the on-screen instructions.
# CPU architecture
export CPU_ARCH=native
# Python
export PYTHON_BIN_PATH=/home/mmayer/anaconda3/envs/tensorflow-1.8/bin/python
export PYTHON_LIB_PATH=/home/mmayer/anaconda3/envs/tensorflow-1.8/lib/python3.6/site-packages
# GCC
export HOST_C_COMPILER=/usr/bin/gcc-7
export HOST_CXX_COMPILER=/usr/bin/g++-7
export GCC_HOST_COMPILER_PATH=/usr/bin/gcc-4.9
export CC_OPT_FLAGS=-march=$CPU_ARCH
# Compilation
export TF_DOWNLOAD_CLANG=0
# Memory allocator
export TF_NEED_JEMALLOC=1
# Storage and input
export TF_NEED_S3=1
export TF_NEED_GCP=1
export TF_NEED_HDFS=0
export TF_NEED_KAFKA=0
# OpenCL
export TF_NEED_COMPUTECPP=1
export TF_NEED_OPENCL_SYCL=1
export TF_NEED_OPENCL=1
export COMPUTECPP_TOOLKIT_PATH=/opt/ComputeCpp-CE-0.7.0-Ubuntu-16.04-x86_64/
# CUDA
export TF_NEED_CUDA=1
export TF_CUDA_CLANG=0
export TF_NCCL_VERSION=1.3
export TF_CUDA_VERSION=9.1
export CUDA_TOOLKIT_PATH=/usr/local/cuda-$TF_CUDA_VERSION
# CUDA Compute Capabilities
# https://developer.nvidia.com/cuda-gpus
#
# - 3.7: K80
# - 5.2: Titan X, 980 Ti
# - 5.3: Jetson TX1
# - 6.0: Tesla P100
# - 6.1: GTX 1080 Ti, GTX 1070, Titan XP
# - 7.0: Tesla V100
export TF_CUDA_COMPUTE_CAPABILITIES=5.2,6.1
# cuDNN
export TF_CUDNN_VERSION=7.1
export CUDNN_INSTALL_PATH=/opt/cudnn7.1
# TensorRT
export TF_NEED_TENSORRT=0
#export TF_TENSORRT_VERSION=3.0.4
#export TENSORRT_INSTALL_PATH=/opt/TensorRT-3.0.4
# MPI
export TF_NEED_MPI=0
# Android
export TF_SET_ANDROID_WORKSPACE=0
# Memory access GDR/VERBS
# https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/gdr/README.md
export TF_NEED_GDR=1
export TF_NEED_VERBS=1
# XLA JIT: Accelerated Linear Algebra
export TF_ENABLE_XLA=1
# Documentation hints
echo "Targeting $CPU_ARCH architecture."
echo "To build TensorFlow, run:"
echo ""
echo " ./configure"
echo " bazel build --config=opt --config=mkl --config=cuda //tensorflow/tools/pip_package:build_pip_package"
echo " bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg"
echo ""
echo "For extra tooling, run:"
echo " bazel build --config=opt --config=mkl --config=cuda //tensorflow/compiler/aot:tfcompile"
echo " bazel build --config=opt --config=mkl --config=cuda //tensorflow/tools/graph_transforms:transform_graph"
echo " bazel build --config=opt --config=mkl --config=cuda //tensorflow/python/tools:freeze_graph"
echo " bazel build --config=opt --config=mkl --config=cuda tensorflow/contrib/lite/toco:toco"
echo ""
Raw
build-tf-cuda9.2-cudnn7.1-tensorrt4.sh
# Environment variables for building TensorFlow 1.9RC1 on Ubuntu 18.04
#
# CPU type (see https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html):
# - native
#
# This assumes:
# - CUDA 9.2
# - cuDNN 7.1
# - ComputeCPP 0.9.0
# - TensorRT 4.0.1.6
# - libibverbs-dev
# - librdmacm-dev
#
# Source this file, then follow the on-screen instructions.
# CPU architecture
export CPU_ARCH=native
# Python
export PYTHON_BIN_PATH=/opt/anaconda3/envs/tensorflow-1.9/bin/python
export PYTHON_LIB_PATH=/opt/anaconda3/envs/tensorflow-1.9/lib/python3.6/site-packages
# GCC
export HOST_C_COMPILER=/usr/bin/gcc-7
export HOST_CXX_COMPILER=/usr/bin/g++-7
export GCC_HOST_COMPILER_PATH=/usr/bin/gcc-6
export CC_OPT_FLAGS=-march=$CPU_ARCH
# Compilation
export TF_DOWNLOAD_CLANG=0
# Memory allocator
export TF_NEED_JEMALLOC=1
# Storage and input
export TF_NEED_S3=1
export TF_NEED_GCP=1
export TF_NEED_HDFS=0
export TF_NEED_KAFKA=0
# OpenCL
export TF_NEED_COMPUTECPP=1
export TF_NEED_OPENCL_SYCL=1
export TF_NEED_OPENCL=1
export COMPUTECPP_TOOLKIT_PATH=/opt/ComputeCpp-CE-0.9.0-Ubuntu-16.04-x86_64
# CUDA
export TF_NEED_CUDA=1
export TF_CUDA_CLANG=0
export TF_NCCL_VERSION=1.3
export TF_CUDA_VERSION=9.2
export CUDA_TOOLKIT_PATH=/usr/local/cuda-$TF_CUDA_VERSION
# CUDA Compute Capabilities
# https://developer.nvidia.com/cuda-gpus
#
# - 3.7: K80
# - 5.2: Titan X, 980 Ti
# - 5.3: Jetson TX1
# - 6.0: Tesla P100
# - 6.1: GTX 1080 Ti, GTX 1070, Titan XP
# - 7.0: Tesla V100
export TF_CUDA_COMPUTE_CAPABILITIES=5.2
# cuDNN
export TF_CUDNN_VERSION=7.1
export CUDNN_INSTALL_PATH=/opt/cudnn/cudnn7.1-cuda9.2
# TensorRT
export TF_NEED_TENSORRT=1
export TF_TENSORRT_VERSION=4.0.1.6
export TENSORRT_INSTALL_PATH=/opt/tensorrt/4.0.1.6
# MPI
export TF_NEED_MPI=0
# Android
export TF_SET_ANDROID_WORKSPACE=0
# Memory access GDR/VERBS
# https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/gdr/README.md
export TF_NEED_GDR=1
export TF_NEED_VERBS=1
# XLA JIT: Accelerated Linear Algebra
export TF_ENABLE_XLA=1
# Documentation hints
echo "Targeting $CPU_ARCH architecture."
echo "To build TensorFlow, run:"
echo ""
echo " ./configure"
echo " bazel build --config=opt --config=mkl --config=cuda //tensorflow/tools/pip_package:build_pip_package"
echo " bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg"
echo ""
echo "For extra tooling, run:"
echo " bazel build --config=opt --config=mkl --config=cuda //tensorflow/compiler/aot:tfcompile"
echo " bazel build --config=opt --config=mkl --config=cuda //tensorflow/tools/graph_transforms:transform_graph"
echo " bazel build --config=opt --config=mkl --config=cuda //tensorflow/python/tools:freeze_graph"
echo " bazel build --config=opt --config=mkl --config=cuda tensorflow/contrib/lite/toco:toco"
echo ""
Raw
build-tf-cuda9.2-cudnn7.3.1-tensorrt4.sh
# Environment variables for building TensorFlow 1.12 on Ubuntu 18.04
#
# Notes:
# - For some reason, bazel was unable to find the correct temporary directory
# and the specified libraries (even though the locations are specified via
# configuration.) A workaround is to populate the TMP and LD_LIBRARY_PATH
# environment variables explicitly with the lib64 and lib paths of the libraries.
# - You might need to install additional Keras packages as described in the solutions
# to this issue: https://github.com/tensorflow/tensorflow/issues/21518
#
# CPU type (see https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html):
# - native
#
# This assumes:
# - Python 3.6
# - Bazel 0.15+
# - GCC 8
#
# For CUDA:
# - GCC 6.4 (see https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html)
# - CUDA 9.2
# - cuDNN 7.3.1
# - NCCL 2.3.5
# - TensorRT 4.0.1.6
#
# Optionally instead of CUDA:
# - ComputeCPP 1.0.1
#
# Optionally:
# - libibverbs-dev
# - librdmacm-dev
#
# Source this file, then follow the on-screen instructions.
# Bazel 0.18.0 quirks
export TMP=/tmp
# CPU architecture
export CPU_ARCH=native
# Python
export PYTHON_BIN_PATH=/home/mmayer/anaconda3/envs/tensorflow-1.12/bin/python
export PYTHON_LIB_PATH=/home/mmayer/anaconda3/envs/tensorflow-1.12/lib/python3.6/site-packages
# GCC
export HOST_C_COMPILER=/usr/bin/gcc-7
export HOST_CXX_COMPILER=/usr/bin/g++-7
export GCC_HOST_COMPILER_PATH=/usr/bin/gcc-6
export CC_OPT_FLAGS=-march=$CPU_ARCH
# Compilation
export TF_DOWNLOAD_CLANG=0
# OpenCL
export TF_NEED_COMPUTECPP=0
export TF_NEED_OPENCL_SYCL=0
export TF_NEED_OPENCL=0
export COMPUTECPP_TOOLKIT_PATH=/opt/ComputeCpp-CE-1.0.1-Ubuntu-16.04-x86_64
# Apache Ignite support
export TF_NEED_IGNITE=1
# AMD ROCm
export TF_NEED_ROCM=0
# CUDA
export TF_NEED_CUDA=1
export TF_CUDA_CLANG=0
export TF_CUDA_VERSION=9.2
export CUDA_TOOLKIT_PATH=/usr/local/cuda-$TF_CUDA_VERSION
# NCCL
export TF_NCCL_VERSION=2.3.5
export NCCL_INSTALL_PATH=/opt/nccl_2.3.5-2-cuda9.2
export NCCL_HDR_PATH=$NCCL_INSTALL_PATH/include
# CUDA Compute Capabilities
# https://developer.nvidia.com/cuda-gpus
#
# - 3.7: K80
# - 5.2: Titan X, 980 Ti
# - 5.3: Jetson TX1
# - 6.0: Tesla P100
# - 6.1: GTX 1080 Ti, GTX 1070, Titan XP
# - 7.0: Tesla V100
export TF_CUDA_COMPUTE_CAPABILITIES=5.2,6.1
# cuDNN
export TF_CUDNN_VERSION=7.3.1
export CUDNN_INSTALL_PATH=/opt/cudnn7.3.1-cuda9.2
# TensorRT
export TF_NEED_TENSORRT=1
export TF_TENSORRT_VERSION=4.0.1.6
export TENSORRT_INSTALL_PATH=/opt/TensorRT-4.0.1.6-cuda9.2-cudnn7.1/
# MPI
export TF_NEED_MPI=0
# Android
export TF_SET_ANDROID_WORKSPACE=0
# Memory access GDR/VERBS
# https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/verbs/README.md
# https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/gdr/README.md
export TF_NEED_GDR=0
export TF_NEED_VERBS=0
# XLA JIT: Accelerated Linear Algebra
export TF_ENABLE_XLA=1
# Build options
export BUILD_OPTIONS='--config=opt --config=mkl --config=ngraph'
if [ "$TF_NEED_CUDA" == "1" ]; then
BUILD_OPTIONS="$OPTIONS --config=cuda"
fi
if [ "$TF_NEED_GDR" == "1" ]; then
BUILD_OPTIONS="$OPTIONS --config=gdr"
fi
if [ "$TF_NEED_VERBS" == "1" ]; then
BUILD_OPTIONS="$OPTIONS --config=verbs"
fi
# Documentation hints
echo "Targeting $CPU_ARCH architecture."
echo "To build TensorFlow, run:"
echo ""
echo " ./configure"
echo " bazel build \$BUILD_OPTIONS //tensorflow/tools/pip_package:build_pip_package"
echo " bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg"
echo ""
echo "For extra tooling, run:"
echo " bazel build \$BUILD_OPTIONS //tensorflow/compiler/aot:tfcompile"
echo " bazel build \$BUILD_OPTIONS //tensorflow/tools/graph_transforms:transform_graph"
echo " bazel build \$BUILD_OPTIONS //tensorflow/python/tools:freeze_graph"
echo " bazel build \$BUILD_OPTIONS tensorflow/contrib/lite/toco:toco"
echo ""
echo "Note that all --config=... options are available from the \$BUILD_OPTIONS environment variable."
echo "They expand to the following values:"
echo " $BUILD_OPTIONS"
echo ""
Raw
build-tf-nogpu-ivybridge.sh
# Environment variables for building TensorFlow 1.8
#
# CPU type (see https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html):
# - ivybridge (e.g. Intel(R) Xeon(R) CPU E5-2673 v3)
#
# This assumes:
# - ComputeCPP 0.7.0
# - libibverbs-dev
# - librdmacm-dev
#
# Source this file, then follow the on-screen instructions.
# CPU architecture
export CPU_ARCH=ivybridge
# Python
export PYTHON_BIN_PATH=/opt/conda/envs/tensorflow-1.8/bin/python
export PYTHON_LIB_PATH=/opt/conda/envs/tensorflow-1.8/lib/python3.6/site-packages
# GCC
export HOST_C_COMPILER=/usr/bin/gcc-6
export HOST_CXX_COMPILER=/usr/bin/g++-6
export GCC_HOST_COMPILER_PATH=/usr/bin/gcc-6
export CC_OPT_FLAGS=-march=$CPU_ARCH
# Compilation
export TF_DOWNLOAD_CLANG=0
# Memory allocator
export TF_NEED_JEMALLOC=1
# Storage and input
export TF_NEED_S3=1
export TF_NEED_GCP=1
export TF_NEED_HDFS=1
export TF_NEED_KAFKA=1
# OpenCL
export TF_NEED_COMPUTECPP=1
export TF_NEED_OPENCL_SYCL=1
export TF_NEED_OPENCL=1
export COMPUTECPP_TOOLKIT_PATH=/usr/local/ComputeCpp-CE-0.7.0-Ubuntu-16.04-x86_64/
# CUDA
export TF_NEED_CUDA=0
# TensorRT
export TF_NEED_TENSORRT=0
# MPI
export TF_NEED_MPI=0
# Android
export TF_SET_ANDROID_WORKSPACE=0
# Memory access GDR/VERBS
# https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/gdr/README.md
export TF_NEED_GDR=1
export TF_NEED_VERBS=1
# XLA JIT: Accelerated Linear Algebra
export TF_ENABLE_XLA=1
# Documentation hints
echo "Targeting $CPU_ARCH architecture."
echo "To build TensorFlow, run:"
echo ""
echo " ./configure"
echo " bazel build --config=opt --config=monolithic --config=mkl //tensorflow/tools/pip_package:build_pip_package"
echo " bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg"
echo ""
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment