Racket performance improvement over 17 years

206-vs-80.md

Arseny Kapoulkine (h/t John Regehr) wrote a nice exploration of how LLVM from 2010 compares to the performance we get today. I decided to try out an old version of Racket, and compare performance to a modern version.

The versions I compared were:

• MzScheme 206p1, released January 2004
• Racket 8.0.0.4, compiled February 2021 (using the Chez Scheme backend)

I measured exactly one benchmark, a mandelbrot set computation from the Computer Language Benchmarks Game, using the implementation in the Racket repository. Of course, that code won't run on v206p1, so I modified it to avoid new features of Racket. The modified version is below.

Of course, many of the features that have been added make the program faster, so it's important to compare with the performance of the modern code. There's even a parallel version that we can test.

Here are the results, measured with hyperfine on a 6-core i7-6700 Linux machine:

Benchmark	MzS 206p1	Racket 8.0.0.4
mandelbrot-old.ss	18.570 s	1.184 s
mandelbrot.rkt		0.255 s
mandelbrot-futures.rkt		0.125 s

On this benchmark, 17 years of compiler improvement produced 15.7x speedup. That's a doubling time of about 4.3 years, rather than the 18 of Proebsting's Law. Further language improvements get us to 148.6x speedup (or 72x if we don't count parallelism), for a doubling time of 2.4 years (or 2.7). That's not quite Moore's law, but it's getting close.

This emphasizes John's point that compiler work doesn't mostly go into making code from 20 years ago run faster, but enabling new code to be fast, and often faster than the old code could be.

The other lesson is that big performance improvements require big changes. Racket today is implemented totally differently from Racket of 2004, and the speedups seen here would not be possible without those changes.

mandelbrot-old.ss

(module mandelbrot-old mzscheme

(define (write-byte b o) (write-char (integer->char b) o))
(define (->fl x) (if (inexact? x) (fx->fl x) x))  
(define O (current-output-port))
(define fx->fl exact->inexact)
(define LIMIT-SQR 4.0)
(define ITERATIONS 50)
(define N (string->number (vector-ref (current-command-line-arguments) 0)))
(define N.0 (fx->fl N))
(define 2/N (/ 2.0 N.0))
(define Crs
  (let ([v (make-vector N)])
    (let loop ([x 0])
      (when (< x N)        
        (vector-set! v x (- (/ (fx->fl (* 2 x)) N.0) 1.5))
        (loop (+ x 1))))
    v))

(define-syntax mandelbrot
  (syntax-rules  ()
    [(mandelbrot Cr Ci)
     (let loop ([i 0] [Zr 0.0] [Zi 0.0])
       (cond [(> (+ (* Zr Zr) (* Zi Zi)) LIMIT-SQR) 0]
             [(= i ITERATIONS) 1]
             [else (let ([Zr (+ (- (* Zr Zr) (* Zi Zi)) Cr)]
                         [Zi (+ (* 2.0 (* Zr Zi)) Ci)])
                     (loop (+ i 1) Zr Zi))]))]))

(fprintf O "P4\n~a ~a\n" N N)
(let loop-y ([y N])
  (let ([Ci (- (* 2/N (->fl y)) 1.0)])
    (let loop-x ([x 0] [bitnum 0] [byteacc 0])
      (if (< x N)
        (let* ([Cr (vector-ref Crs x)]
               [bitnum (+ bitnum 1)]
               [byteacc (+ (arithmetic-shift byteacc 1) (mandelbrot Cr Ci))])
          (cond [(= bitnum 8)
                 (write-byte byteacc O)
                 (loop-x (+ x 1) 0 0)]
                [else (loop-x (+ x 1) bitnum byteacc)]))
        (begin (when (> bitnum 0)
                 (write-byte (arithmetic-shift byteacc (- 8 (and N #x7))) O))
               (when (> y 1) (loop-y (- y 1))))))))
)

Concise, well-written, and informative. Excellent work.