Artefact Evalution for "Bounding Data Races in Space and Time"

This document contains instructions for the artefact evaluation. The draft paper is available at kcsrk.info/papers/pldi18-memory.pdf.

Access to benchmarking machines

The main evaluation in the paper is the overhead of compiling our proposed memory model to relaxed architectures such as ARM and POWER (Fig 7 on page 10). To this end, we have set up an ARM (aarch64) and POWER (ppc64) machine and have provided the ssh keys (pldi-aec-rsa) to access the machine.

Copy the key file to your machine and change its permission to

chmod 600 pldi-aec-rsa

Then you can log into the aarch64 machine as:

ssh -i pldi-aec-rsa [email protected]

and the ppc64 machine as:

ssh -i pldi-aec-rsa [email protected]

Content and Organization

This section describes the content and organisation of the benchmarks. You may skip this section and move to the next. In the rest of this section, I will assume that you are logged into the aarch64 machine. This instructions for benchmarking on ppc64, unless explicitly noted otherwise.

In the home directory, you should have 2 directories:

$ ls
logs  ocamlbench-scripts

We will use ocamlbench-scripts/run-bench.sh to run the benchmarks. The benchmark results are stored in the logs directory.

The compiler variants we will test can be listed with the opam switch command as follows:

$ opam switch
# switch                                      compiler                                               description
4.07.0+trunk+bench                            ocaml-variants.4.07.0+trunk                            4.07.0+trunk+bench
4.07.0+trunk+branch-after-load-aarch64+bench  ocaml-variants.4.07.0+trunk+branch-after-load-aarch64  4.07.0+trunk+branch-after-load-aarch64+bench
4.07.0+trunk+dmb-before-store-aarch64+bench   ocaml-variants.4.07.0+trunk+dmb-before-store-aarch64   4.07.0+trunk+dmb-before-store-aarch64+bench
4.07.0+trunk+profile-reads-and-writes+bench   ocaml-variants.4.07.0+trunk+profile-reads-and-writes   4.07.0+trunk+profile-reads-and-writes+bench
4.07.0+trunk+sra-aarch64+bench                ocaml-variants.4.07.0+trunk+sra                        4.07.0+trunk+sra-aarch64+bench
default                                       ocaml-system.4.02.3                                    default
operf                                         ocaml-system.4.02.3                                    operf

Of these, the compiler switch 4.07.0+trunk+profile-reads-and-writes+bench is used to generate the benchmark characteristics graph in Fig 7(a). The other switches starting with 4.07.0+ are the variants evaluated in the graph in Fig 7(b).

Where are the compiler and benchmark sources?

All the information about the compiler variants and benchmarks are found from a special opam remote named benches:

$ opam repo -v
[NOTE] Use '--all' to see all configured repositories independently of what is selected in the current switch

=-=- Repository configuration for switch operf =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 1 benches git+https://github.com/kayceesrk/ocamlbench-repo#aarch64
 2 default https://opam.ocaml.org/2.0

You can browse the benchmarks in the directory packages in ocamlbench-repo. The various compiler variants used are listed in packages/ocaml-variants. The opam files have information about the location of the compilers. The compiler variants are:

Running the benchmarks

After you have logged into aarch64 or ppc64 machine, make sure you are in the operf switch.

$ opam switch operf

Start a tmux session as the benchmarks take around 4 hours to run.

$ tmux

If the ssh session gets terminated, you can come back and attach to the same session with a tmux attach. Initiate the benchmarking script,

$ cd ocamlbench-scripts
$ bash ./run-bench.sh --nowait

Generating the graphs

Once you have finished running the experiments, you can generate the graphs. There are 3 python scripts make_fig_7a.py, make_fig_7b.py and make_fig_7c.py in the ocamlbench-scripts directory in order to generate the sub-figures in Fig 7. make_fig_7a.py and make_fig_7b.py need to be run on aarch64 machine and make_fig_7c.py on ppc64 machine.

$ python make_fig_7a.py
Writing 'fig_7a.pdf'

and similarly for other figures. scp the pdfs to your local machine to view them.

kayceesrk/pldi18_aec_ldrf.md