Vertex-CFD FEM solver

vertex_cfd_fem.md

Trilinos installation

mkdir trilinos
cd trilinos/
git clone -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git

Then source the spack

source spack/share/spack/setup-env.sh

Continue installating Trilinos with spack by:

spack env create vertex spack-trilinos16.yaml
spack env activate vertex
spack spec

For Ubuntu 22, the spack-trilinos16.yaml is modified as:

spack:
  concretizer:
    unify: when_possible

  config:
    build_stage:
    - $spack/var/spack/spack-stage/build-$arch-$date/
    misc_cache: $spack/.cache
    build_jobs: 32
    install_tree:
      root: $spack/opt/spack/
      projections:
        all: install-{os}-{target}-{compiler.name}-{compiler.version}/{name}-{version}
    keep_stage: true
    verify_ssl: true
    checksum: true
    dirty: false
    build_language: C
    ccache: false
    db_lock_timeout: 120
    package_lock_timeout: null
    shared_linking:
      type: rpath
      bind: false
    allow_sgid: true
    locks: true
    suppress_gpg_warnings: false
    connect_timeout: 10

  compilers:
  - compiler: # EDIT: change below with your compiler of choice
      spec: [email protected]
      paths: # EDIT: Change those based on the location of the compiler location you prefer
        cc: /home/dineshadepu/anaconda3/envs/gcc_env/bin/x86_64-conda-linux-gnu-gcc
        cxx: /home/dineshadepu/anaconda3/envs/gcc_env/bin/x86_64-conda-linux-gnu-g++
        f77: /home/dineshadepu/anaconda3/envs/gcc_env/bin/x86_64-conda-linux-gnu-gfortran
        fc: /home/dineshadepu/anaconda3/envs/gcc_env/bin/x86_64-conda-linux-gnu-gfortran
      flags: {}
      operating_system: ubuntu22.04 # EDIT: update according to `spack arch`: just the OS version
      target: x86_64
      modules: # EDIT: update with a list of modules you want to load by default
      - gcc/14.2.0

  packages:
    all:
      target: [skylake] # EDIT: Change this according to `spack arch`
      compiler: [[email protected]] # EDIT: Change according to your compiler of choice
      providers: # EDIT: Change versions below according to the default in your machine
        mpi: [openmpi]
        blas: [openblas]
        lapack: [openblas]
        pkgconfig: [pkg-config]

  specs:
  - python
  - ninja
  - boost
  - intel-oneapi-mkl
  - googletest
  - [email protected] +chaco+exodus+hdf5+intrepid2+kokkos+mpi+nox+openmp+panzer+phalanx+shards+shared+stk+tempus+zoltan2

Further, I used conda to install latest gcc and g++ and gave their paths for configuration file:

TODO: put conda installation instruction

Now install all the packages by:

spack install

This downloaded almost 72 packages, and took almost an hour.

Setup the modules

The modules can be set by

spack module tcl refresh

But what actually worked is

spack install lmod
. $(spack location -i lmod)/lmod/lmod/init/bash

Again source spack

. spack/share/spack/setup-env.sh

These above commands will give us access to module. Remember that we are all running this from trilinos directory.

Load the Trilinos module

module use spack/share/spack/modules/linux-ubuntu22.04-skylake/
module load trilinos/16.0.0-gcc-11.4.0-aviqc4r

Your system might generate a different trilinos module file.

VERTEX-CFD Installation

mkdir VERTEX-CFD
cd VERTEX-CFD
git clone https://github.com/ORNL/VERTEX-CFD.git

mkdir build
cd build

Now using vertex-env-cpu.sh file, we will load all the required modules:

#!/bin/bash
module use spack/share/spack/modules/linux-ubuntu22.04-skylake/ # EDIT: Update this with the module location, as an example: module use spack/share/spack/modules/linux-centos7-broadwell/
install=spack/share/spack/modules/linux-ubuntu22.04-skylake/
echo $install

module load gcc/11.4.0

module use $install
module load gcc-runtime
module load openmpi
module load cmake
module load ninja
module load boost
module load netcdf-c
module load parmetis
module load intel-oneapi-mkl
module load cgns
module load googletest
module load hdf5
module load zlib-ng
module load trilinos/16.0.0-gcc-11.4.0-aviqc4r
HDF5_DIR=spack/opt/spack/install-ubuntu22.04-skylake-gcc-11.4.0/hdf5-1.14.6/
ZLIB_DIR=spack/opt/spack/install-ubuntu22.04-skylake-gcc-11.4.0/zlib-ng-2.2.3/

GCC_ROOT=/usr/lib/gcc/
unset LIBRARY_PATH

source vertex-env-cpu.sh

Now generate the files to compile by using the following file:

SOURCE=/home/dineshadepu/life/softwares/VERTEX-CFD/
INSTALL=/home/dineshadepu/life/softwares/VERTEX-CFD/build/install
BUILD="RelWithDebInfo"

rm -rf CMake*
rm -rf .ninja*
rm DartConfiguration.tcl
rm CTestTestfile.cmake
rm build.ninja
rm VertexCFDConfig.cmake
rm -rf Testing

# EDIT: Uncomment this line for GPU and update this based on your NVCC WRAPPER location
#NVCC_WRAPPER=<PATH TO NVCC WRAPPER DIRECTORY>

# Unset variable set by spack modules.
# If this is present, any directories present will be treated
# as "implicit" library paths by CMake and it will strip them
# out of the executable RPATHS. Then you have to set
# LD_LIBRARY_PATH appropriately to run jobs.
unset LIBRARY_PATH

# EDIT: This command is for CPU. Comment it for GPU.
cmake \
    -D CMAKE_BUILD_TYPE=${BUILD} \
    -D CMAKE_INSTALL_PREFIX=${INSTALL} \
    -D VertexCFD_ENABLE_COVERAGE_BUILD=OFF \
    -D CMAKE_CXX_FLAGS="-Wall -Wextra -Wpedantic -fdiagnostics-color" \
    -D VertexCFD_ENABLE_TESTING=OFF \
    -D Trilinos_ROOT=/home/dineshadepu/life/softwares/trilinos/spack/opt/spack/install-ubuntu22.04-skylake-gcc-11.4.0/trilinos-16.0.0 \
    \
    ${SOURCE}

# EDIT: Uncomment this command for GPU.
#cmake \
#    -D CMAKE_BUILD_TYPE=${BUILD} \
#    -D CMAKE_INSTALL_PREFIX=${INSTALL} \
#    -D CMAKE_CXX_COMPILER=${NVCC_WRAPPER} \
#    -D VertexCFD_ENABLE_COVERAGE_BUILD=OFF \
#    -D CMAKE_CXX_FLAGS="-Wall -Wextra -Wpedantic -fdiagnostics-color" \
#    -D VertexCFD_ENABLE_TESTING=ON \
#    -D Trilinos_ROOT=<PATH TO TRILINOS/16.0.0> \
#    \
#    ${SOURCE}

source vertex-config.sh