Skip to content

Instantly share code, notes, and snippets.

@ragnard
Last active December 12, 2015 05:39
Show Gist options
  • Save ragnard/4723371 to your computer and use it in GitHub Desktop.
Save ragnard/4723371 to your computer and use it in GitHub Desktop.
(defn with-query-results-cursor
"Execute a query using a cursor to efficiently handle large data
sets. result-fn will be applied to a result set seq."
[[sql & params :as sql-params] fetch-size result-fn]
(sql/transaction
(with-open [stmt (.prepareStatement (sql/connection) sql)]
(doseq [[index value] (map vector (iterate inc 1) params)]
(.setObject stmt index value))
(.setFetchSize stmt fetch-size)
(with-open [rset (.executeQuery stmt)]
(result-fn (resultset-seq rset))))))
(defn exec-batched
[conn query batch-size batch-fn]
(sql/with-connection conn
(with-query-results-cursor
query
batch-size
(fn [rs]
(doseq [batch (partition batch-size batch-size nil rs)]
(batch-fn batch))))))
(exec-batched conn
["SELECT * FROM ... WHERE x = ?" 42]
100
(fn [rows]
(doseq [row rows]
(println row)))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment