List of media for learning more about measurement bias in system benchmarks

Measurement Bias Media List

Talk: "We Have It Easy, But Do We Have It Right?"
-------------------------------------------------------

  [https://www.youtube.com/watch?v=DKVRkfXrBpg]

  Amer Diwan's talk at Google covering the topics in this paper and
  other ideas.


"Why You Should Care About Quantile Regression"
-----------------------------------------------------

  [http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/41873.pdf]


"Our Troubles With Linux And Why You Should Care"
-------------------------------------------------------

  [https://cs.uwaterloo.ca/~brecht/papers/getpaper.php?file=apsys-2011.pdf]

  Details issues in the Linux kernel that cause performance problems
  which may potentially invalidate benchmark results.


"Benchmark Precision and Random Initial State"
----------------------------------------------------

  [http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=2DB8682CC43449285A11A4C2335ABC0C?doi=10.1.1.63.2108&rep=rep1&type=pdf]


"R3: Repeatability, Reproducibility, and Rigor"
-----------------------------------------------------

  [http://janvitek.github.io/pubs/r3.pdf]

  Outlines several deadly sins in systems research.


"Stabilizer: Statistically Sound Performance Evaluation"
--------------------------------------------------------------

  [http://people.cs.umass.edu/~emery/pubs/stabilizer-asplos13.pdf]

  Tool which automatically, and periodically, randomizes code and data
  layout.  This improves on the suggestions from the parent paper about
  randomizing layout: the randomization is more comprehensive because it
  can do, e.g., function-level randomization, while changing link orders
  only does module-level randomization.


"Statistically Rigorous Java Performance Evaluation"
----------------------------------------------------------

  [https://buytaert.net/files/oopsla07-georges.pdf]