Skip to content

Instantly share code, notes, and snippets.

@javierluraschi
Last active December 7, 2018 01:22
Show Gist options
  • Save javierluraschi/88e72b8ff028bbf3f0b35d27239e5fe1 to your computer and use it in GitHub Desktop.
Save javierluraschi/88e72b8ff028bbf3f0b35d27239e5fe1 to your computer and use it in GitHub Desktop.
Building rwinlib/arrow 0.11
  • Create EC2 Windows Machine and connect.
  • Download and install MSys2.
  • Launch c:\msys64\mingw64 and run:
pacman -S base-devel
pacman -S msys2-devel
pacman -S mingw-w64-i686-toolchain
pacman -S mingw-w64-x86_64-toolchain
pacman -S mingw-w64-x86_64-cmake
  • Download boost from boost_1_68_0.zip.
  • Extract into c:\Users\Administrator\RStudio\boost_1_68_0.
  • Build boost from mingw64 shell as follows:
cd /c/Users/Administrator/RStudio/boost_1_68_0/
./bootstrap.sh --with-libraries=regex,system,filesystem
./b2 -j2 variant=release define=_GLIBCXX_USE_CXX11_ABI=0 link=static
cd c:\Users\Administrator\RStudio\
github clone http://github.com/jimhester/rarrow
github clone http://github.com/apache/arrow
git checkout f9e5818ef8bd4f1eea13c1185a59077bb1323374
  • Patch arrow:
# arrow/cpp/src/arrow/CMakeLists.txt:
if (NOT ARROW_BOOST_HEADER_ONLY AND ARROW_HDFS)
  set(ARROW_SRCS ${ARROW_SRCS}
    io/hdfs.cc
    io/hdfs-internal.cc
  )
endif()
  • From the mingw64 shell, build arrow:
cd /c/Users/Administrator/RStudio/arrow/cpp
mkdir release
cd release
export BOOST_ROOT=/c/Users/Administrator/RStudio/boost_1_68_0/
cmake .. -DARROW_BUILD_TESTS=FALSE -DARROW_WITH_SNAPPY=FALSE -DARROW_WITH_ZSTD=FALSE -DARROW_WITH_LZ4=FALSE -DARROW_WITH_LZ4=FALSE -DARROW_JEMALLOC=FALSE -DARROW_BUILD_STATIC=TRUE -DARROW_BOOST_VENDORED=FALSE -DARROW_BOOST_USE_SHARED=FALSE -DARROW_WITH_ZLIB=FALSE -DARROW_WITH_BROTLI=FALSE -DARROW_USE_GLOG=FALSE -DPTHREAD_LIBRARY=FALSE -DARROW_BUILD_UTILITIES=FALSE -DCMAKE_BUILD_TYPE=release -G"MSYS Makefiles" -DCMAKE_CXX_COMPILER_ARCHITECTURE_ID=x64 -DARROW_HDFS=FALSE -DCMAKE_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=0"
cmake .. -DARROW_BUILD_TESTS=FALSE -DARROW_WITH_SNAPPY=FALSE -DARROW_WITH_ZSTD=FALSE -DARROW_WITH_LZ4=FALSE -DARROW_WITH_LZ4=FALSE -DARROW_JEMALLOC=FALSE -DARROW_BUILD_STATIC=TRUE -DARROW_BOOST_VENDORED=FALSE -DARROW_BOOST_USE_SHARED=FALSE -DARROW_WITH_ZLIB=FALSE -DARROW_WITH_BROTLI=FALSE -DARROW_USE_GLOG=FALSE -DPTHREAD_LIBRARY=FALSE -DARROW_BUILD_UTILITIES=FALSE -DCMAKE_BUILD_TYPE=release -G"MSYS Makefiles" -DCMAKE_CXX_COMPILER_ARCHITECTURE_ID=x64 -DARROW_HDFS=FALSE -DCMAKE_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=0"
make
make install
  • Merge rarrow and arrow:
cd  /c/Users/Administrator/RStudio/arrow/r
cp /c/Users/Administrator/RStudio/rarrow/configure.win .
cp -r /c/Users/Administrator/RStudio/rarrow/tools .
cp /c/Users/Administrator/RStudio/rarrow/src/Makevars.win src/
mkdir -p windows/arrow-0.11.0/lib/x64
cp -r /c/Program\ Files\ \(x86\)/arrow/include windows/arrow-0.11.0/
cp /c/Program\ Files\ \(x86\)/arrow/lib/libarrow.a windows/arrow-0.11.0/lib/x64/
  • Patch Makevars.win to use VERSION=0.11.0 instead of VERSION=0.9.0.

  • Install RStudio, R, open arrow/r project and build.

  • Build will fail, install missing packages and RTools from propmpts:

install.packages("Rcpp")
install.packages("rlang")
install.packages("purrr")
install.packages("glue")
install.packages("vctrs")
install.packages("fs")
install.packages("withr")
install.packages("R6")
install.packages("bit64")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment