Skip to content

Instantly share code, notes, and snippets.

@nathanmarz
Last active April 4, 2024 23:46
Show Gist options
  • Save nathanmarz/b7c612b417647db80b9eaab618ff8d83 to your computer and use it in GitHub Desktop.
Save nathanmarz/b7c612b417647db80b9eaab618ff8d83 to your computer and use it in GitHub Desktop.
Specter 0.13.0 benchmark
Benchmark code at: https://github.com/nathanmarz/specter/blob/master/scripts/benchmarks.clj
Run against Clojure 1.7.0 and Java 1.7.0 on Mac OSX 10.11.6
Benchmark: get value in nested map (2500000 iterations)
Avg(ms) vs best Code
53.528 1.00 (-> data (get :a) (get :b) (get :c))
54.708 1.02 (-> data :a :b :c)
103.03 1.92 (compiled-select-any p data)
109.92 2.05 (select-any [:a :b :c] data)
111.81 2.09 (select-any [(keypath :a) (keypath :b) (keypath :c)] data)
158.11 2.95 (specter-dynamic-nested-get data :a :b :c)
170.28 3.18 (get-in data [:a :b :c])
********************************
Benchmark: update value in nested map (500000 iterations)
Avg(ms) vs best Code
83.648 1.00 (manual-transform data inc)
86.441 1.03 (transform [:a :b :c] inc data)
529.97 6.34 (update-in data [:a :b :c] inc)
********************************
Benchmark: transform values of a small map (500000 iterations)
Avg(ms) vs best Code
45.241 1.00 (transform MAP-VALS inc data)
74.505 1.65 (reduce-kv (fn [m k v] (assoc m k (inc v))) {} data)
93.615 2.07 (reduce-kv (fn [m k v] (assoc m k (inc v))) (empty data) data)
102.49 2.27 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient {}) data))
170.22 3.76 (transform [ALL LAST] inc data)
424.46 9.38 (into {} (for [[k v] data] [k (inc v)]))
476.91 10.5 (zipmap (keys data) (map inc (vals data)))
491.83 10.9 (into {} (map (fn [e] [(key e) (inc (val e))]) data))
********************************
Benchmark: transform values of large map (600 iterations)
Avg(ms) vs best Code
89.321 1.00 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient clojure.lang.PersistentHashMap/EMPTY) data))
94.117 1.05 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient {}) data))
94.839 1.06 (transform MAP-VALS inc data)
115.98 1.30 (reduce-kv (fn [m k v] (assoc m k (inc v))) {} data)
120.07 1.34 (reduce-kv (fn [m k v] (assoc m k (inc v))) (empty data) data)
132.00 1.48 (transform [ALL LAST] inc data)
188.62 2.11 (into {} (for [[k v] data] [k (inc v)]))
196.81 2.20 (into {} (map (fn [e] [(key e) (inc (val e))]) data))
258.00 2.89 (zipmap (keys data) (map inc (vals data)))
********************************
Benchmark: map a function over a vector (1000000 iterations)
Avg(ms) vs best Code
165.45 1.00 (into [] (map inc) data)
169.01 1.02 (mapv inc data)
188.44 1.14 (transform ALL inc data)
442.41 2.67 (vec (map inc data))
********************************
Benchmark: filter a sequence (500000 iterations)
Avg(ms) vs best Code
110.79 1.00 (filterv even? data)
129.44 1.17 (select [ALL even?] data)
167.40 1.51 (into [] (filter even?) data)
204.44 1.85 (doall (filter even? data))
246.84 2.23 (select-any (filterer even?) data)
********************************
Benchmark: even :a values from sequence of maps (500000 iterations)
Avg(ms) vs best Code
83.723 1.00 (into [] xf data)
128.32 1.53 (select [ALL :a even?] data)
148.40 1.77 (into [] (comp (map :a) (filter even?)) data)
312.08 3.73 (->> data (mapv :a) (filter even?) doall)
********************************
Benchmark: update every value in a tree (represented with vectors) (50000 iterations)
Avg(ms) vs best Code
81.533 1.00 (tree-value-transform (fn [e] (if (even? e) (inc e) e)) data)
88.786 1.09 (transform [TreeValuesProt even?] inc data)
94.852 1.16 (transform [TreeValues even?] inc data)
324.52 3.98 (walk/postwalk (fn [e] (if (and (number? e) (even? e)) (inc e) e)) data)
339.70 4.17 (transform [(walker number?) even?] inc data)
********************************
Benchmark: Traverse into a set (5000 iterations)
Avg(ms) vs best Code
815.69 1.00 (into #{} (traverse ALL data))
824.53 1.01 (set data)
883.21 1.08 (persistent! (reduce conj! (transient #{}) (traverse ALL data)))
997.47 1.22 (set (select ALL data))
1258.4 1.54 (reduce conj #{} (traverse ALL data))
Benchmark code at: https://github.com/nathanmarz/specter/blob/master/scripts/benchmarks.clj
Run against Clojure 1.8.0 and Java 1.7.0 on Mac OSX 10.11.6
Benchmark: get value in nested map (2500000 iterations)
Avg(ms) vs best Code
56.953 1.00 (-> data :a :b :c)
62.537 1.10 (-> data (get :a) (get :b) (get :c))
106.82 1.88 (compiled-select-any p data)
113.30 1.99 (select-any [(keypath :a) (keypath :b) (keypath :c)] data)
118.95 2.09 (select-any [:a :b :c] data)
136.59 2.40 (select-one [:a :b :c] data)
139.85 2.46 (select-first [:a :b :c] data)
147.64 2.59 (select-one! [:a :b :c] data)
165.78 2.91 (specter-dynamic-nested-get data :a :b :c)
175.39 3.08 (get-in data [:a :b :c])
********************************
Benchmark: update value in nested map (500000 iterations)
Avg(ms) vs best Code
88.003 1.00 (transform [:a :b :c] inc data)
100.64 1.14 (manual-transform data inc)
509.91 5.79 (update-in data [:a :b :c] inc)
********************************
Benchmark: transform values of a small map (500000 iterations)
Avg(ms) vs best Code
52.867 1.00 (transform MAP-VALS inc data)
78.891 1.49 (map-vals-map-iterable data inc)
124.30 2.35 (map-vals-map-iterable-transient data inc)
153.92 2.91 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient {}) data))
154.18 2.92 (reduce-kv (fn [m k v] (assoc m k (inc v))) (empty data) data)
158.01 2.99 (reduce-kv (fn [m k v] (assoc m k (inc v))) {} data)
177.76 3.36 (transform [ALL LAST] inc data)
414.94 7.85 (zipmap (keys data) (map inc (vals data)))
471.38 8.92 (into {} (for [[k v] data] [k (inc v)]))
471.92 8.93 (into {} (map (fn [e] [(key e) (inc (val e))]) data))
********************************
Benchmark: transform values of large map (600 iterations)
Avg(ms) vs best Code
115.11 1.00 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient {}) data))
119.70 1.04 (transform MAP-VALS inc data)
120.55 1.05 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient clojure.lang.PersistentHashMap/EMPTY) data))
135.85 1.18 (reduce-kv (fn [m k v] (assoc m k (inc v))) (empty data) data)
138.02 1.20 (map-vals-map-iterable-transient data inc)
142.85 1.24 (reduce-kv (fn [m k v] (assoc m k (inc v))) {} data)
160.36 1.39 (transform [ALL LAST] inc data)
162.97 1.42 (map-vals-map-iterable data inc)
208.67 1.81 (into {} (for [[k v] data] [k (inc v)]))
211.33 1.84 (into {} (map (fn [e] [(key e) (inc (val e))]) data))
236.55 2.06 (zipmap (keys data) (map inc (vals data)))
********************************
Benchmark: map a function over a vector (1000000 iterations)
Avg(ms) vs best Code
164.71 1.00 (into [] (map inc) data)
166.34 1.01 (mapv inc data)
198.93 1.21 (transform ALL inc data)
438.70 2.66 (vec (map inc data))
********************************
Benchmark: filter a sequence (500000 iterations)
Avg(ms) vs best Code
94.955 1.00 (filterv even? data)
121.26 1.28 (select [ALL even?] data)
160.50 1.69 (doall (filter even? data))
163.26 1.72 (into [] (filter even?) data)
229.03 2.41 (select-any (filterer even?) data)
********************************
Benchmark: even :a values from sequence of maps (500000 iterations)
Avg(ms) vs best Code
80.767 1.00 (into [] xf data)
125.40 1.55 (select [ALL :a even?] data)
148.34 1.84 (into [] (comp (map :a) (filter even?)) data)
257.12 3.18 (->> data (mapv :a) (filter even?) doall)
********************************
Benchmark: update every value in a tree (represented with vectors) (50000 iterations)
Avg(ms) vs best Code
72.504 1.00 (tree-value-transform (fn [e] (if (even? e) (inc e) e)) data)
76.809 1.06 (transform [TreeValuesProt even?] inc data)
89.608 1.24 (transform [TreeValues even?] inc data)
247.96 3.42 (transform [(walker number?) even?] inc data)
289.12 3.99 (walk/postwalk (fn [e] (if (and (number? e) (even? e)) (inc e) e)) data)
********************************
Benchmark: Traverse into a set (5000 iterations)
Avg(ms) vs best Code
788.05 1.00 (set data)
846.60 1.07 (into #{} (traverse ALL data))
879.60 1.12 (persistent! (reduce conj! (transient #{}) (traverse ALL data)))
962.73 1.22 (set (select ALL data))
1376.1 1.75 (reduce conj #{} (traverse ALL data))
@sova
Copy link

sova commented Sep 4, 2020

This is useful, but is there any reason you didn't use something like Criterium to benchmark? I'm really weary of micro benchmark results. Thanks for sharing though. :)

i think he did? https://github.com/redplanetlabs/specter/blob/master/scripts/benchmarks.clj

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment