Skip to content

Instantly share code, notes, and snippets.

@kwk

kwk/Containerfile

Last active July 21, 2023 12:57
Show Gist options
  • Save kwk/ce8f6aac2eef8a8983d7b9aa954d35af to your computer and use it in GitHub Desktop.
Save kwk/ce8f6aac2eef8a8983d7b9aa954d35af to your computer and use it in GitHub Desktop.
Download ZIP
Runnning the llvm-test-suite with PGO optimized and plain LLVM in Fedora Rawhide
Raw
README.adoc

README

$ podman build -t test-pgo . # (1)
$ podman run -it test-pgo # (2)

  1. Build the container image to compile tests in and execute them.

  2. Run the container image and enter a bash console after tests have executed.

You should see similar content of results.txt below.

Raw
Containerfile
FROM fedora:rawhide
LABEL description="Test compilers with llvm-test-suite"
USER root
WORKDIR /root
# Install deps to run test-suite
RUN dnf install -y \
cmake \
fedora-packager \
git \
python3-pip \
python3-lit \
python3-pandas \
python3-scipy \
ninja-build \
which \
coreutils \
tcl \
tcl-devel \
tcl-tclreadline \
tcl-tclxml-devel \
tcl-tclxml \
tcl-zlib \
tcl-thread-devel
# Clone test suite (in correct version for installed clang version)
# See https://llvm.org/docs/TestSuiteGuide.html
# RUN export VERSION=`clang --version | grep -ioP 'clang version\s\K[0-9\.]+'` \
# && git clone --depth=1 --branch llvmorg-${VERSION} https://github.com/llvm/llvm-test-suite.git test-suite
RUN git clone --depth=1 https://github.com/llvm/llvm-test-suite.git test-suite
RUN dnf install -y 'dnf-command(copr)' perf
COPY entrypoint.sh /root/entrypoint.sh
USER root
ENTRYPOINT [ "/root/entrypoint.sh" ]
Raw
entrypoint.sh
#!/usr/bin/bash
set -x
function configure_build_run {
# Configure the test suite
cmake \
-DCMAKE_GENERATOR=Ninja \
-DCMAKE_C_COMPILER=/usr/bin/clang \
-DCMAKE_CXX_COMPILER=/usr/bin/clang++ \
-C~/test-suite/cmake/caches/O3.cmake \
~/test-suite
# Build the test-suite
ninja -j30
# Run the tests with lit:
lit -j1 -v -o results.json . || true
}
# Build with regular clang
dnf install -y clang clang-libs clang-resource-filesystem llvm llvm-libs
mkdir -pv ~/rawhide
cd ~/rawhide
configure_build_run
# Install and enable the repository that provides the PGO LLVM Toolchain
# See https://llvm.org/docs/HowToBuildWithPGO.html#building-clang-with-pgo
dnf copr enable -y @fedora-llvm-team/llvm-pgo-optimized fedora-rawhide-x86_64
dnf remove -y clang clang-libs clang-resource-filesystem llvm llvm-libs
dnf install -y clang clang-libs clang-resource-filesystem llvm llvm-libs
mkdir -pv ~/pgo
cd ~/pgo
configure_build_run
# Compare the results
/root/test-suite/utils/compare.py \
--metric exec_time \
--metric compile_time \
--metric link_time \
--lhs-name 16.0.6 \
--rhs-name 16.0.6-pgo \
~/rawhide/results.json vs ~/pgo/results.json > ~/results-1.txt || true
bash
Raw
results.txt
Warning: 'test-suite :: SingleSource/UnitTests/X86/x86-dyn_stack_alloc_realign.test' has no metrics, skipping!
Warning: 'test-suite :: SingleSource/UnitTests/X86/x86-dyn_stack_alloc_realign2.test' has no metrics, skipping!
Warning: 'test-suite :: SingleSource/UnitTests/X86/x86-dyn_stack_alloc_realign.test' has no metrics, skipping!
Warning: 'test-suite :: SingleSource/UnitTests/X86/x86-dyn_stack_alloc_realign2.test' has no metrics, skipping!
Tests: 3067
Metric: exec_time,compile_time,link_time
Program exec_time compile_time link_time
16.0.6 16.0.6-pgo diff 16.0.6 16.0.6-pgo diff 16.0.6 16.0.6-pgo diff
UnitTests/vla 0.00 0.00 inf% 0.04 0.06 37.2% 0.03 0.03 -22.6%
Regression...e/execute/GCC-C-execute-loop-9 0.00 0.00 inf% 0.00 0.00 0.04 0.02 -36.7%
Regression...execute/GCC-C-execute-930518-1 0.00 0.00 inf% 0.00 0.00 0.02 0.03 15.0%
Regression...execute/GCC-C-execute-930614-2 0.00 0.00 inf% 0.00 0.00 0.04 0.03 -31.0%
Regression...execute/GCC-C-execute-930621-1 0.00 0.00 inf% 0.00 0.00 0.04 0.03 -41.1%
Regression...execute/GCC-C-execute-930622-1 0.00 0.00 inf% 0.00 0.00 0.04 0.03 -23.1%
Regression...execute/GCC-C-execute-930718-1 0.00 0.00 inf% 0.00 0.00 0.03 0.03 -1.5%
Regression...xecute/GCC-C-execute-931004-10 0.00 0.00 inf% 0.00 0.00 0.03 0.03 17.4%
Regression...xecute/GCC-C-execute-931004-12 0.00 0.00 inf% 0.00 0.00 0.04 0.02 -42.0%
Regression...xecute/GCC-C-execute-931004-13 0.00 0.00 inf% 0.00 0.00 0.03 0.03 19.4%
Regression...execute/GCC-C-execute-931004-2 0.00 0.00 inf% 0.00 0.00 0.02 0.02 17.5%
Regression...execute/GCC-C-execute-931004-4 0.00 0.00 inf% 0.00 0.00 0.02 0.02 -5.7%
Regression...execute/GCC-C-execute-931017-1 0.00 0.00 inf% 0.00 0.00 0.03 0.03 0.6%
Regression...execute/GCC-C-execute-931102-2 0.00 0.00 inf% 0.00 0.00 0.03 0.02 -16.8%
Regression...execute/GCC-C-execute-931228-1 0.00 0.00 inf% 0.00 0.00 0.03 0.02 -13.9%
Geomean difference -100.0% 1.4% 0.4%
exec_time compile_time link_time
l/r 16.0.6 16.0.6-pgo diff 16.0.6 16.0.6-pgo diff 16.0.6 16.0.6-pgo diff
count 3047.000000 3047.000000 2267.000000 2508.000000 2508.000000 442.000000 2508.000000 2508.000000 2508.000000
mean 1071.711268 1078.298369 inf 0.234005 0.238223 0.029977 0.044340 0.045100 0.039693
std 20774.833726 20887.445279 NaN 1.985318 2.020749 0.186075 0.029814 0.031582 0.278042
min 0.000000 0.000000 -1.000000 0.000000 0.000000 -0.466793 0.012700 0.016100 -0.550725
25% 0.000000 0.000000 -0.147691 0.000000 0.000000 -0.080628 0.026700 0.026400 -0.159443
50% 0.000900 0.000900 0.000000 0.000000 0.000000 0.012460 0.031700 0.031500 0.007392
75% 0.156700 0.159335 0.128012 0.000000 0.000000 0.114654 0.041625 0.041225 0.197752
max 815097.902098 814764.661234 inf 67.937600 67.912200 0.861702 0.219300 0.194300 1.300000
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment