Skip to content

Instantly share code, notes, and snippets.

@aria42
Created April 9, 2012 04:29
Show Gist options
  • Select an option

  • Save aria42/2341435 to your computer and use it in GitHub Desktop.

Select an option

Save aria42/2341435 to your computer and use it in GitHub Desktop.
(defn map-reduce [map-fn reduce-fn n xs]
(let [bspec {:type :mem :merge reduce-fn}
bs (repeatedly n #(bucket/new bspec))
work (fn [b x]
(doseq [[k v] (map-fn x)]
(bucket/merge b k v)))
;; each worker gets own bucket
;; so no contention
workers (map #(partial work %) bs)]
;; workers process xs in par, blocking
(do-work workers xs)
;; merge all bucket users
(reduce bucket/merge-all bs)))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment