Building CUDA-aware OpenMPI 5.0 + parallel HDF5 2.0.0 on Freya (MPCDF) GPU nodes

MPI+HDF5_build_on_freya.md

Building CUDA-aware OpenMPI 5.0 + parallel HDF5 2.0.0 on Freya (MPCDF) GPU nodes

Reproducible recipe for a hand-built, CUDA-aware OpenMPI 5.0.10 and a parallel HDF5 2.0.0 linked against it, using the gcc/15 + cuda/13.0 toolchain. This mirrors the site module openmpi_gpu/cuda_11.6-11.6.2-gcc_11-11.2.0/4.1.6 and the older hdf5-2.0.0-openmpi_gpu-4.1 stack, modernised for OpenMPI 5.x.

Built / verified: 2026-07-14 on Freya (SLE 15, SLURM 24.11).

What lands where

Component	Version	Install prefix
OpenMPI (CUDA-aware)	5.0.10	/freya/ptmp/mpa/adutt/openmpi_gpu-5.0
Parallel HDF5	2.0.0	/freya/ptmp/mpa/adutt/hdf5-2.0.0-openmpi_gpu-5.0/parallel
Toolchain	gcc 15.1.0, cuda 13.0.1	modules gcc/15 cuda/13.0

Source trees (kept for rebuilds):

• OpenMPI: /freya/ptmp/mpa/adutt/openmpi-5.0.10 (tarball openmpi-5.0.10.tar.bz2)
• HDF5: /freya/ptmp/mpa/adutt/hdf5-2.0.0 (CMake source tree)

Why this configuration (vs. the OpenMPI 4.1 reference)

The reference 4.1 build used --with-cuda --with-slurm --with-pmi --with-pmix --enable-mpi1-compatibility --with-io-romio-flags=--with-file-system=gpfs and relied on the built-in openib + smcuda BTLs for CUDA-aware InfiniBand. In OpenMPI 5.0:

• openib BTL is removed. There is no UCX on Freya (no module, no system libs), so the InfiniBand path is libfabric / OFI — /usr/lib64/libfabric.so + headers are present. → use --with-ofi=/usr --without-ucx.
• --with-pmi was DELETED in 5.0. Launch is via PMIx; use the bundled PMIx/PRRTE (--with-pmix=internal --with-prrte=internal) and launch with srun --mpi=pmix.
• --with-cuda still enables CUDA-awareness (accelerator/cuda, smcuda, coll/cuda). Point --with-cuda-libdir at the driver stub ($CUDA_HOME/lib64/stubs) so the link test passes on the (GPU-less) login node.
• --enable-mpi1-compatibility and the GPFS ROMIO flag are kept as in the reference.

OpenMPI and HDF5 are pure C (no nvcc kernels), so the gcc-15 ↔ cuda-13 host-compiler pairing is not stressed — CUDA is used only via the driver API for GPU-buffer awareness.

---

Part A — CUDA-aware OpenMPI 5.0.10

# 0. Fetch source (skip if /freya/ptmp/mpa/adutt/openmpi-5.0.10 already exists)
cd /freya/ptmp/mpa/adutt
curl -sSLO https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.10.tar.bz2
tar xjf openmpi-5.0.10.tar.bz2
cd openmpi-5.0.10

# 1. Toolchain
module purge
module load gcc/15 cuda/13.0
export CC=gcc CXX=g++ FC=gfortran

# 2. Configure  (CUDA-aware, OFI/libfabric for IB, PMIx+PRRTE internal, GPFS ROMIO)
./configure \
  --prefix=/freya/ptmp/mpa/adutt/openmpi_gpu-5.0 \
  --with-cuda="$CUDA_HOME" \
  --with-cuda-libdir="$CUDA_HOME/lib64/stubs" \
  --with-ofi=/usr \
  --without-ucx \
  --with-slurm \
  --with-pmix=internal \
  --with-prrte=internal \
  --with-io-romio-flags=--with-file-system=gpfs \
  --enable-mpi1-compatibility \
  2>&1 | tee configure-gpu.log

# 3. Build + install
make -j"$(nproc)"
make install

Confirm the configure summary shows: CUDA support: yes · OpenFabrics OFI Libfabric: yes · IBM Spectrum Scale/GPFS: yes · pmix: internal · PRRTE: internal.

Verify OpenMPI

OMPI=/freya/ptmp/mpa/adutt/openmpi_gpu-5.0
export PATH="$OMPI/bin:$PATH"
export LD_LIBRARY_PATH="$OMPI/lib:$LD_LIBRARY_PATH"

ompi_info --version | head -1                 # Open MPI v5.0.10
mpicc --version | head -1                      # gcc (GCC) 15.1.0
ompi_info --parsable --all | grep opal_built_with_cuda_support   # ... :value:true
ompi_info | grep -iE "MCA accelerator|btl: (ofi|smcuda)|coll: cuda"

Expected: opal_built_with_cuda_support ... true, and components accelerator: cuda, btl: ofi, btl: smcuda, coll: cuda.

---

Part B — Parallel HDF5 2.0.0 against the new MPI

The HDF5 2.0.0 source is a CMake tree at /freya/ptmp/mpa/adutt/hdf5-2.0.0. Options mirror the reference hdf5-2.0.0-openmpi_gpu-4.1 build (parallel, zlib, deprecated symbols, HL + tools; no Fortran/C++/Java), with mpicc from the new MPI as the C compiler. Built out-of-source so the 4.1 build is untouched.

module purge
module load gcc/15 cuda/13.0 cmake/4.0

OMPI=/freya/ptmp/mpa/adutt/openmpi_gpu-5.0
export PATH="$OMPI/bin:$PATH"
export LD_LIBRARY_PATH="$OMPI/lib:$LD_LIBRARY_PATH"

SRC=/freya/ptmp/mpa/adutt/hdf5-2.0.0
BUILD="$SRC/build-openmpi_gpu-5.0"
PREFIX=/freya/ptmp/mpa/adutt/hdf5-2.0.0-openmpi_gpu-5.0/parallel

rm -rf "$BUILD"
cmake -S "$SRC" -B "$BUILD" -G "Unix Makefiles" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$PREFIX" \
  -DCMAKE_C_COMPILER="$(command -v mpicc)" \
  -DHDF5_ENABLE_PARALLEL=ON \
  -DHDF5_ENABLE_ZLIB_SUPPORT=ON \
  -DHDF5_ENABLE_DEPRECATED_SYMBOLS=ON \
  -DHDF5_BUILD_HL_LIB=ON \
  -DHDF5_BUILD_TOOLS=ON \
  -DHDF5_BUILD_UTILS=ON \
  -DHDF5_BUILD_EXAMPLES=ON \
  -DHDF5_BUILD_FORTRAN=OFF \
  -DHDF5_BUILD_CPP_LIB=OFF \
  -DHDF5_BUILD_JAVA=OFF \
  -DHDF5_ENABLE_THREADSAFE=OFF \
  -DBUILD_SHARED_LIBS=ON \
  -DBUILD_STATIC_LIBS=ON \
  -DBUILD_TESTING=OFF

cmake --build "$BUILD" -j"$(nproc)"
cmake --install "$BUILD"

Deviation from the 4.1 reference: BUILD_TESTING=OFF (reference had it ON). This only skips compiling the (non-installed) test binaries and saves build time; the installed library is identical. Set it ON for exact parity.

Verify HDF5

PREFIX=/freya/ptmp/mpa/adutt/hdf5-2.0.0-openmpi_gpu-5.0/parallel
grep -E "HDF5 Version|Parallel HDF5|C Compiler" "$PREFIX/lib/libhdf5.settings"
"$PREFIX/bin/h5pcc" -showconfig 2>/dev/null | head -20 || "$PREFIX/bin/h5cc" -show

Expect Parallel HDF5: ON and the C Compiler pointing at /freya/ptmp/mpa/adutt/openmpi_gpu-5.0/bin/mpicc.

---

Part C — Using the stack

module purge
module load gcc/15 cuda/13.0 cmake/4.0
OMPI=/freya/ptmp/mpa/adutt/openmpi_gpu-5.0
HDF5=/freya/ptmp/mpa/adutt/hdf5-2.0.0-openmpi_gpu-5.0/parallel
export PATH="$OMPI/bin:$HDF5/bin:$PATH"
export LD_LIBRARY_PATH="$OMPI/lib:$HDF5/lib:$LD_LIBRARY_PATH"
export CPATH="$HDF5/include:$CPATH"

• Launch under SLURM with PMIx: srun --mpi=pmix ./your_app. On a GPU compute node SLURM's PMIx supplies the transport key the smcuda BTL needs (see Troubleshooting).
• For idefix (GPU + MPI + HDF5): OMPI_CXX=$IDEFIX_DIR/src/kokkos/bin/nvcc_wrapper, -DHDF5_ROOT=$HDF5. With cuda/13.0's nvcc, C++20 is supported, so the Kokkos-5.1 develop branch (which sets CMAKE_CXX_STANDARD 20) builds without forcing C++17.

Verified

2026-07-14: h5pcc-compiled 2-rank program did a collective H5Fcreate with H5Pset_fapl_mpio → RESULT: OK ranks=2 parallel-HDF5 collective create succeeded.

Troubleshooting

• Error obtaining unique transport key from PMIX (OMPI_MCA_orte_precondition_transports not present) → PML add procs failed at MPI_Init. The smcuda BTL (built because CUDA is enabled) needs a PMIx transport key. A bare mpirun on a login node doesn't provide it. Fixes:
  • Real runs: launch on a compute node with srun --mpi=pmix (SLURM's PMIx provides it).
  • Quick login-node test without GPU IPC: add -mca pml ob1 -mca btl self,sm (i.e. exclude smcuda).

Rebuild helper scripts (already on disk)

• /freya/ptmp/mpa/adutt/openmpi-5.0.10/build-install-gpu.sh — make + install OpenMPI (after ./configure from Part A step 2).
• /freya/ptmp/mpa/adutt/hdf5-2.0.0/build-hdf5-openmpi_gpu-5.0.sh — full HDF5 configure + build + install (Part B in one shot).

System facts this recipe depends on (Freya, 2026-07)

• No UCX (module or system) → InfiniBand via libfabric/OFI (/usr/lib64/libfabric.so).
• SLURM 24.11 with system PMIx (/usr/lib64/libpmix.so.2); launch with srun --mpi=pmix.
• CUDA driver stub for link tests: $CUDA_HOME/lib64/stubs/libcuda.so.
• Modules gcc/15 (15.1.0) and cuda/13.0 (13.0.1); cmake/4.0 for HDF5.

build-install-gpu.sh

#!/bin/bash -l
# Build + install CUDA-aware OpenMPI 5.0.10 (configure already run).
set -eo pipefail
module purge
module load gcc/15 cuda/13.0
cd /freya/ptmp/mpa/adutt/openmpi-5.0.10
make -j"$(nproc)" 2>&1 | tee make-gpu.log
make install 2>&1 | tee install-gpu.log
echo "BUILD_INSTALL_DONE_OK"

build-hdf5-openmpi_gpu-5.0.sh

#!/bin/bash -l
# Build parallel HDF5 2.0.0 against the CUDA-aware OpenMPI 5.0.10 built above.
# Mirrors the reference hdf5-2.0.0-openmpi_gpu-4.1 CMake options, swapping in the
# new mpicc and install prefix. Out-of-source build so the 4.1 build is untouched.
set -eo pipefail
module purge
module load gcc/15 cuda/13.0 cmake/4.0

OMPI=/freya/ptmp/mpa/adutt/openmpi_gpu-5.0
export PATH="$OMPI/bin:$PATH"
export LD_LIBRARY_PATH="$OMPI/lib:${LD_LIBRARY_PATH:-}"

SRC=/freya/ptmp/mpa/adutt/hdf5-2.0.0
BUILD="$SRC/build-openmpi_gpu-5.0"
PREFIX=/freya/ptmp/mpa/adutt/hdf5-2.0.0-openmpi_gpu-5.0/parallel

echo "=== using mpicc: $(command -v mpicc) ==="
mpicc -show

rm -rf "$BUILD"
cmake -S "$SRC" -B "$BUILD" -G "Unix Makefiles" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$PREFIX" \
  -DCMAKE_C_COMPILER="$(command -v mpicc)" \
  -DHDF5_ENABLE_PARALLEL=ON \
  -DHDF5_ENABLE_ZLIB_SUPPORT=ON \
  -DHDF5_ENABLE_DEPRECATED_SYMBOLS=ON \
  -DHDF5_BUILD_HL_LIB=ON \
  -DHDF5_BUILD_TOOLS=ON \
  -DHDF5_BUILD_UTILS=ON \
  -DHDF5_BUILD_EXAMPLES=ON \
  -DHDF5_BUILD_FORTRAN=OFF \
  -DHDF5_BUILD_CPP_LIB=OFF \
  -DHDF5_BUILD_JAVA=OFF \
  -DHDF5_ENABLE_THREADSAFE=OFF \
  -DBUILD_SHARED_LIBS=ON \
  -DBUILD_STATIC_LIBS=ON \
  -DBUILD_TESTING=OFF \
  2>&1 | tee "$SRC/cmake-openmpi_gpu-5.0.log"

cmake --build "$BUILD" -j"$(nproc)" 2>&1 | tee "$SRC/build-openmpi_gpu-5.0.log"
cmake --install "$BUILD" 2>&1 | tee "$SRC/install-openmpi_gpu-5.0.log"
echo "HDF5_DONE_OK"

srun --mpi=pmix ./app

Finally compile Idefix from problem directory with after setting IDEFIX_DIR to the main root directory of Idefix

module purge
module load gcc/15 cuda/13.0 cmake/4.0
OMPI=/freya/ptmp/mpa/adutt/openmpi_gpu-5.0
HDF5=/freya/ptmp/mpa/adutt/hdf5-2.0.0-openmpi_gpu-5.0/parallel
export PATH=$OMPI/bin:$HDF5/bin:$PATH
export LD_LIBRARY_PATH=$OMPI/lib:$HDF5/lib:${LD_LIBRARY_PATH:-}

CC=mpicc CXX=mpic++ cmake  \
-DIdefix_MPI=ON -DIdefix_DEBUG=OFF \
-DIdefix_HDF5=ON   -DKokkos_ENABLE_CUDA=ON \
-DKokkos_ARCH_AMPERE80=ON \
-DHDF5_ROOT=/freya/ptmp/mpa/adutt/hdf5-2.0.0-openmpi_gpu-5.0/parallel \
-DIdefix_PROBLEM_DIR=$PWD \
-GNinja -B ./build -S $IDEFIX_DIR  
cmake --build ./build/