Skip to content

Instantly share code, notes, and snippets.

@yunqu

yunqu/build-ray.md

Last active October 12, 2022 07:54
Show Gist options
  • Save yunqu/bdbe9c34b883398bf97cf5361db63171 to your computer and use it in GitHub Desktop.
Save yunqu/bdbe9c34b883398bf97cf5361db63171 to your computer and use it in GitHub Desktop.
Download ZIP
Building Ray for aarch64
Raw
build-ray.md

Building Ray for aarch64

There are multiple ways to build ray on aarch64 processors. We will use the PYNQ image v2.4 with Ubuntu 18.04 OS as an example. In this document, we will introduce 2 approaches to build ray; we will always prefer the second approach (building on Amazon A1) since it is faster, cleaner, and easy to reproduce.

Building from Source on the Target

The easiest way is to build ray from source on the board. Since the bootstrapping is going to take a lot of disk space, it is highly recommended to use a 32GB SD card for your PYNQ image v2.4. Again, we are only targetting 64-bit ARM architectures, so please only go through the following steps on Zynq Ultrascale boards.

Step 1. Install dependencies

As instructed in the offical document, we will need to install a few dependent libraries first.

sudo apt-get update
sudo apt-get install -y build-essential curl unzip psmisc

Since we already have a Python 3.6 environment, we will just stick with our Python 3 commands. Now install the following:

pip3 install cython==0.29.0 setproctitle psutil

Step 2. Prepare bazel

You need to first install a Java environment.

sudo apt-get install -y openjdk-11-jdk-headless

After that, we have a pre-compiled bazel binary for aarch64. You can copy it over to /usr/local/bin and change its permissions.

sudo chmod 777 /usr/local/bin/bazel

Verify the command is working:

bazel

It should show a help message.

Step 3. Build

You can clone the ray repository, checkout a version, and build from it. If you do not want to build from that tag, another commit that you can try is 931e6a2.

cd /opt/builds
git clone https://github.com/ray-project/ray.git
cd ray
git checkout -b ray-0.6.3 tags/ray-0.6.3
cd python
pip3 install -e .

This will take a long time. But after this you are done!

Building Wheels on AWS A1 Platform

Now we will build wheels on AWS A1, copy the wheels over to your board, and install them! This takes the shortcut since all the Zynq Ultrascale processors share the aarch64 so we do not have to rebuild the packages for different boards.

Let us assume you are on an A1 platform from now on.

Step 1. Prepare bazel

You need to first install a Java environment.

sudo apt-get update
sudo apt-get install -y openjdk-11-jdk-headless

After that, since we have a pre-compiled bazel binary for aarch64, you can copy it over to /usr/local/bin and change its permissions. For building bazel on Amazon A1, please see my other document.

sudo chmod 777 /usr/local/bin/bazel

Verify the command is working:

bazel

Step 2. Install arrow

We need to prepare this package first since ray depends on it. We will follow the steps for installation. For simplicity, the steps are summarized here.

sudo apt-get install -y libjemalloc-dev libboost-dev \
					libboost-filesystem-dev \
                    libboost-system-dev \
                    libboost-regex-dev \
                    python3-dev \
                    autoconf \
                    flex bison
pip3 install six numpy pandas cython pytest psutil

I used /opt/builds as my working directory to host various repositories. For a few additional packages if you have not installed them yet:

sudo apt-get install -y libssl-dev curl cmake

Now we need to set up a few environment variables.

cd /opt/builds
mkdir arrow-dist
export ARROW_HOME=$(pwd)/arrow-dist
export LD_LIBRARY_PATH=$(pwd)/arrow-dist/lib:$LD_LIBRARY_PATH

Eventually the directory arrow-dist will contain all the *.so files that you need to copy over onto the board. Now clone the repository and build arrow from source.

cd /opt/builds
git clone https://github.com/apache/arrow/
cd arrow
git checkout -b apache-arrow-0.12.0 tags/apache-arrow-0.12.0
cd cpp
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
	  -DCMAKE_INSTALL_LIBDIR=lib \
      -DARROW_PYTHON=ON \
      -DARROW_PLASMA=ON \
      -DARROW_BUILD_TESTS=ON \
	  -DPYTHON_EXECUTABLE=/usr/bin/python3 \
      ..
make
sudo make install
cd /opt/builds/arrow/python
PYARROW_WITH_PLASMA=1 python3 setup.py bdist_wheel

After this you should be able to see file pyarrow-0.12.0-cp36-cp36m-linux_aarch64.whl under /opt/builds/arrow/python/dist. Copy this onto the board and install.

sudo pip3 install pyarrow-0.12.0-cp36-cp36m-linux_aarch64.whl

Also, copy the arrow-dist folder onto the board, and run

cp -rf arrow-dist/* /usr/

This delivers all the *.so files as well as the header files.

Step 3. Build Wheel

We will build the wheel for ray. One thing to notice, is that with bazel, we are able to install new versions (not constrained to ray-0.6.3 anymore).

cd /opt/builds
git clone https://github.com/ray-project/ray.git
cd ray
git checkout -b ray-0.6.5 tags/ray-0.6.5
cd python

Patch the following file:

vi /opt/builds/ray/build.sh

Comment the following section around line 98. This makes sure we don't install arrow from pip since we already have it built locally.

$PYTHON_EXECUTABLE -m pip install \
    --target=$ROOT_DIR/python/ray/pyarrow_files pyarrow==0.12.0.RAY \
    --find-links https://s3-us-west-2.amazonaws.com/arrow-wheels/ca1fa51f0901f5a4298f0e4faea00f24e5dd7bb7/index.html

Try to see if you can pip install it.

python3.6 -m pip install -e . --verbose

Finally build the wheel.

python3 setup.py bdist_wheel

You now should be able to see file ray-0.6.5-cp36-cp36m-linux_aarch64.whl under /opt/builds/ray/python/dist. Copy this onto the board and install.

sudo pip3 install ray-0.6.5-cp36-cp36m-linux_aarch64.whl

If you want to rebuild from a clean repository, run in the git repository:

git clean -f -f -x -d
@RA-DL
Copy link

RA-DL commented Oct 24, 2019

I tried to install pyarrow with ray in jetson tx2 but i found this errors when i executed sudo PYARROW_WITH_PLASMA=1 python3 setup.py bdist_wheel , can you help me please?
-- Compiler id: GNU
Selected compiler gcc 7.4.0
-- Arrow build warning level: PRODUCTION
Using ld linker
Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: RELEASE
-- Build output directory: /opt/builds/arrow/python/build/temp.linux-aarch64-3.6/release
-- Searching for Python libs in /usr/lib64;/usr/lib;/usr/lib/python3.6/config-3.6m-aarch64-linux-gnu
-- Looking for python3.6m
-- Found Python lib /usr/lib/python3.6/config-3.6m-aarch64-linux-gnu/libpython3.6m.so
-- Searching for Python libs in /usr/lib64;/usr/lib;/usr/lib/python3.6/config-3.6m-aarch64-linux-gnu
-- Looking for python3.6m
-- Found Python lib /usr/lib/python3.6/config-3.6m-aarch64-linux-gnu/libpython3.6m.so
-- Checking for module 'arrow'
-- No package 'arrow' found
CMake Error at cmake_modules/FindArrow.cmake:139 (message):
Could not find the Arrow library. Looked for headers in , and for libs in
Call Stack (most recent call first):
CMakeLists.txt:203 (find_package)

-- Configuring incomplete, errors occurred!
See also "/opt/builds/arrow/python/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeOutput.log".
See also "/opt/builds/arrow/python/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeError.log".
error: command 'cmake' failed with exit status 1

/**************/opt/builds/arrow/python/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeError.log
Performing C++ SOURCE FILE Test CXX_SUPPORTS_SSE4_2 failed with the following output:
Change Dir: /opt/builds/arrow/python/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_968bb/fast"
/usr/bin/make -f CMakeFiles/cmTC_968bb.dir/build.make CMakeFiles/cmTC_968bb.dir/build
make[1]: Entering directory '/opt/builds/arrow/python/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_968bb.dir/src.cxx.o
/usr/bin/c++ -DCXX_SUPPORTS_SSE4_2 -msse4.2 -o CMakeFiles/cmTC_968bb.dir/src.cxx.o -c /opt/builds/arrow/python/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeTmp/src.cxx
c++: error: unrecognized command line option '-msse4.2'
CMakeFiles/cmTC_968bb.dir/build.make:65: recipe for target 'CMakeFiles/cmTC_968bb.dir/src.cxx.o' failed
make[1]: *** [CMakeFiles/cmTC_968bb.dir/src.cxx.o] Error 1
make[1]: Leaving directory '/opt/builds/arrow/python/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeTmp'
Makefile:126: recipe for target 'cmTC_968bb/fast' failed
make: *** [cmTC_968bb/fast] Error 2

Source file was:
int main() { return 0; }
Performing C++ SOURCE FILE Test CXX_SUPPORTS_ALTIVEC failed with the following output:
Change Dir: /opt/builds/arrow/python/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_f10bb/fast"
/usr/bin/make -f CMakeFiles/cmTC_f10bb.dir/build.make CMakeFiles/cmTC_f10bb.dir/build
make[1]: Entering directory '/opt/builds/arrow/python/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_f10bb.dir/src.cxx.o
/usr/bin/c++ -DCXX_SUPPORTS_ALTIVEC -maltivec -o CMakeFiles/cmTC_f10bb.dir/src.cxx.o -c /opt/builds/arrow/python/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeTmp/src.cxx
c++: error: unrecognized command line option '-maltivec'; did you mean '-mglibc'?
CMakeFiles/cmTC_f10bb.dir/build.make:65: recipe for target 'CMakeFiles/cmTC_f10bb.dir/src.cxx.o' failed
make[1]: *** [CMakeFiles/cmTC_f10bb.dir/src.cxx.o] Error 1
make[1]: Leaving directory '/opt/builds/arrow/python/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeTmp'
Makefile:126: recipe for target 'cmTC_f10bb/fast' failed
make: *** [cmTC_f10bb/fast] Error 2

Source file was:
int main() { return 0; }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment