Configure GPAW on euler cluster

Preparation

Compiler: python/3.1.1 (shipped) gcc-6.3.0, open_mpi/3.0.0, libxc/4.3.0, fftw/3.3.3, openblas/0.2.13_seq, scalapack
tar for numpy scipy matplotlib

Working on virtualenv

python -m venv gpaw
source ~/.virtualenv/gpaw/bin/activate

Alternatively, if you want to use the shipped =numpy= and =scipy= versions, then install the virtualenv with system-package enabled like

python -m venv --system-site-packages gpaw

Now we are working on a clean virtualenv with all packages to be installed

Compile numpy with openblas

/Skip these steps if you choose to use the virtualenv with system-site-packages/

Download the numpy source from git repo or from the PyPI:

pip download --no-binary=:all: numpy

Create site.cfg to cope with compiler and library information

cd 'path/to/numpy'
cp site.cfg.example site.cfg

Uncomment and modify the section with openblas

[openblas]                                                                                                                   
libraries = openblas
library_dirs = /cluster/apps/openblas/0.2.13_seq/x86_64/gcc_5.2.0/lib
include_dirs = /cluster/apps/openblas/0.2.13_seq/x86_64/gcc_5.2.0/include
runtime_library_dirs = /cluster/apps/openblas/0.2.13_seq/x86_64/gcc_5.2.0/lib

install numpy

python setup.py build && python setupy.py install

check numpy configuration

python -c "import numpy; numpy.__config__.show()"

Check if openblas_info and openblas_lapack_info are correct.

Compile scipy

More straightforward than numpy. The OpenBLAS environment can be automatically detected if numpy corrected compiled.

cd 'path/to/numpy'
python setup.py build
python setup.py install

check numpy configuration

python -c "import scipy; scipy.__config__.show()"

Alternative install with pip

Define the flag OPENBLAS before pip install. In some cases one or more packages are not compiled against OpenBLAS so you may with recompile manually.

OPENBLAS=$GOTOBLAS2/lib python -m pip install --upgrade --no-binary numpy
OPENBLAS=$GOTOBLAS2/lib python -m pip install --upgrade --no-binary scipy
OPENBLAS=$GOTOBLAS2/lib python -m pip install --upgrade matplotlib

Other packages

matplotlib ase

(Alternative) compile libxc to a recent version

Follow the instructions on http://www.tddft.org/programs/libxc/installation/ for the installation. Optionally install the python bindings to the libxc (from the root dir of libxc)

python -m pip install .

Install libvdwxc

Follow the instructions on the website and specify the root path. MPI compiler is compulsory in some cases as the build system cannot detect the mpicc automatically

./configure --with-mpi CC="mpicc" FC="mpif90" --prefix="$VIRTUAL_ENV/dependency/libvdwxc"
make && make install

Install ELPA

Download the ELPA package (such as http://elpa.mpcdf.mpg.de/html/Releases/2019.05.001/elpa-2019.05.001.tar.gz) and check the =INSTALL.md= inside. For successfull installation of ELPA, use both GNU version of =CC= and =FC=, and disable =AVX2= support:

configure FC=mpif90 CC=mpicc --disable-avx2 --prefix="$VIRTUAL_ENV/dependency/elpa"
make && make install

Test installations

Requires nose and pytest

python -c "import numpy; numpy.test()"
python -c "import scipy; scipy.test()"
ase test

Install GPAW

Download the newest version of GPAW (currently 19.8.1) Modify the customize.py, enable the nessary parts (=openblas=, =libxc=, =libvdwxc=, =elpa=)

Alternatively you may need to add the lib64 with libpython3.xm.so into LIBRARY_PATH (not LD_LIBRARY_PATH)

A sample configuration file looks like:

import os, os.path
from os.path import expandvars
libraries = ["openblas"]
library_dirs += [expandvars("$MODULE_openblas_ROOT_DIR/lib")]
#library_dirs = [""]

# flake8: noqa

compiler = 'gcc'
mpicompiler = 'mpicc'  # use None if you don't want to build a gpaw-python
mpilinker = 'mpicc'
platform_id = 'euler-centos'

# FFTW3:
fftw = True
if fftw:
    libraries += ['fftw3']
    library_dirs += [expandvars("$FFTW3_ROOT_DIR/lib64")]

# ScaLAPACK (version 2.0.1+ required):
scalapack = True
if scalapack:
    libraries += ['scalapack']
    library_dirs += ["$MODULE_scalapack_ROOT_DIR/lib"]
    define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
    define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]

# Build MPI-interface into _gpaw.so:
parallel_python = True
if not parallel_python:
    compiler = 'mpicc'
    define_macros += [('PARALLEL', '1')]
    mpicompiler = None

# Use Elpa (requires ScaLAPACK and Elpa API 20171201):
if 1:
    elpa = True
    elpadir = expandvars("$VIRTUAL_ENV/dependency/elpa")
    libraries += ['elpa']
    library_dirs += ['{}/lib'.format(elpadir)]
    extra_link_args += ['-Wl,-rpath={}/lib'.format(elpadir)]
    include_dirs += ['{}/include/elpa-2019.05.001'.format(elpadir)]

# LibXC:
# In order to link libxc installed in a non-standard location
# (e.g.: configure --prefix=/home/user/libxc-2.0.1-1), use:

# - dynamic linking (requires rpath or setting LD_LIBRARY_PATH at runtime):
if 1:
    xc = expandvars('$MODULE_libxc_ROOT_DIR')
    include_dirs += [xc + 'include']
    library_dirs += [xc + 'lib']
    # You can use rpath to avoid changing LD_LIBRARY_PATH:
    extra_link_args += ['-Wl,-rpath={xc}/lib'.format(xc=xc)]
    if 'xc' not in libraries:
        libraries.append('xc')


# libvdwxc:
if 1:
    libvdwxc = True
    path = expandvars('$VIRTUAL_ENV/dependency/libvdwxc-build')
    extra_link_args += ['-Wl,-rpath=%s/lib' % path]
    library_dirs += ['%s/lib' % path]
    include_dirs += ['%s/include' % path]
    libraries += ['vdwxc']

Then install with normal install procedure

python setup build && python setup install

If installation succeeded but some errors occur, use the method below to delete the files generated (pip uninstall incapable) https://gist.github.com/lovaulonze/4e31d83091a9d9543c0cfd297a1ded07

Test GPAW

Run the tests like on the website

gpaw info #test basic settings, only scalapack flag is no
gpaw test -j 8 #depends on the system
gpaw-python -m gpaw info #scalapack flag should also be yes
mpiexec -n 24 gpaw-python -m gpaw test #may need to submit to the job queue

Make the module file

mkdir -p ~/modules/gpaw/
emacs -nw ~/modules/gpaw/gcc_5.2.0

Then edit the contents in the "gcc_5.2.0" modulefile:

#%Module1.0#####################################################################
#
#STATUS
#stat:moddep:gcc/5.2.0 open_mpi/1.10.0 openblas/0.2.13_seq scalapack/2.0.2 fftw/3.3.4 libxc/3.0.0
#stat:modauto:
#END STATUS

proc ModulesHelp { } {
    global helpmsg
    puts stderr "\t$helpmsg\n"
}

set version gcc_5.2.0
set curmod  [module-info name]

set topdir  /cluster/home/ttian/.virtualenvs/gpaw/bin/

# check if requirements are met
set envguess  [exec /cluster/apps/scripts/guess_compiler.pl --path=compiler=__CC__\ open_mpi=__MPI__]
set need_cb   "gcc"
set need_cc   "5.2.0"
set need_ompi "1.10.0"


#   if { [string compare $envguess "compiler=${need_cb}_$need_cc open_mpi=$need_ompi"] != 0 } {
#     puts stderr "$curmod error: this module requires $need_cb/$need_cc and open_mpi/$need_ompi to be loaded."
#      puts stderr "hint: run 'module purge ; module load $need_cb/$need_cc open_mpi/$need_ompi $curmod' to resolve this problem."
#      break
#   }
if { [string compare $envguess "compiler=${need_cb}_$need_cc open_mpi=$need_ompi"] != 0 } {
        module load ${need_cb}/$need_cc
        module load open_mpi/$need_ompi
	module load openblas/0.2.13_seq
	module load fftw/3.3.4
	module load scalapack/2.0.2
	module load libxc/3.0.0
	module load python/3.6.0
}

# virtualenv
if { [module-info mode load] || [module-info mode switch2] } {
    puts stdout "source /cluster/home/ttian/.virtualenvs/gpaw/bin/activate;"
} elseif { [module-info mode remove] && ![module-info mode switch3] } {
    puts stdout "deactivate;"
}


# not reached on error

Finally add the following line in .bashrc or .bash_profile to enable local modules:

module use $HOME/modules

alchem0x2A/gpaw_config_euler.md