Install OpenBLAS for use with Spark.

install_OpenBLAS_for_Spark.md

• Install OpenBLAS

  • yum install libgfortran

  • Note: OpenBLAS contains both BLAS and LAPACK routines. So in the end, we can create softlinks for the standard BLAS and LAPACK shared library locations.

  • Download from Github: https://github.com/xianyi/OpenBLAS

  • make

  • make install

  • Create links:

    ln -sf /opt/OpenBLAS/lib/libopenblas.so /opt/OpenBLAS/lib/libblas.so
    ln -sf /opt/OpenBLAS/lib/libopenblas.so /opt/OpenBLAS/lib/libblas.so.3
    ln -sf /opt/OpenBLAS/lib/libopenblas.so /opt/OpenBLAS/lib/liblapack.so
    ln -sf /opt/OpenBLAS/lib/libopenblas.so /opt/OpenBLAS/lib/liblapack.so.3
    ln -sf /opt/OpenBLAS/lib/libopenblas.so /usr/lib64/libblas.so
    ln -sf /opt/OpenBLAS/lib/libopenblas.so /usr/lib64/libblas.so.3
    ln -sf /opt/OpenBLAS/lib/libopenblas.so /usr/lib64/liblapack.so
    ln -sf /opt/OpenBLAS/lib/libopenblas.so /usr/lib64/liblapack.so.3
    

  • Test if setup properly:

    • ./bin/spark-shell

      import com.github.fommil.netlib.BLAS
      println(BLAS.getInstance().getClass().getName())
      

    • Should output com.github.fommil.netlib.NativeSystemBLAS.

    • ps aux | grep spark

    • Grab PID of actual java process for Spark

    • lsof -p PID | grep 'blas'

    • Should be pointing to OpenBLAS.