Last active
August 29, 2015 14:15
-
-
Save tie-rack/b7056482699d7e1e2865 to your computer and use it in GitHub Desktop.
Potential function for splitting a sequence of maps into groups to feed to a function that has a limit on the total number of keys it can handle in any given sequence of maps (SQLite statement parameter limit). (Naming suggestions welcome!)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(defn chunk-rows | |
"Given a seq of maps, split them into groups, such that no group | |
have more than n total keys across its maps." | |
[rows n] | |
(lazy-seq | |
(loop [this-chunk [] | |
this-chunk-size 0 | |
to-go rows] | |
(if (empty? to-go) | |
(list this-chunk) | |
(let [next-row (first to-go) | |
size-with-next-row (+ this-chunk-size (count next-row))] | |
(cond | |
(< n (count next-row)) (throw (ex-info "Map too large" {:map next-row})) | |
(< n size-with-next-row) (cons this-chunk (chunk-rows to-go n)) | |
:else (recur (conj this-chunk next-row) | |
size-with-next-row | |
(rest to-go)))))))) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
;; A sequence of maps read from crew.csv, ready to be inserted as rows | |
;; into a SQLite database | |
(def rows | |
[{:name "Picard", :shirt "red", :rank "Captain"} | |
{:name "Riker", :shirt "red", :rank "Commander"} | |
{:name "Data", :shirt "yellow", :rank "Lt. Commander"} | |
{:name "Worf", :shirt "yellow", :rank "Lt."} | |
{:name "Crusher"} | |
{:name "Troi", :shirt "teal", :rank "Lt. Commander"}]) | |
;; If SQLite had a statement parameter limit of less than 16, trying | |
;; to pass all these rows at once would fail. Here, chunk-rows lets us | |
;; break this up into groups it can handle. | |
(chunk-rows rows 15) | |
;; ([{:name "Picard", :shirt "red", :rank "Captain"} | |
;; {:name "Riker", :shirt "red", :rank "Commander"} | |
;; {:name "Data", :shirt "yellow", :rank "Lt. Commander"} | |
;; {:name "Worf", :shirt "yellow", :rank "Lt."} | |
;; {:name "Crusher"}] | |
;; [{:name "Troi", :shirt "teal", :rank "Lt. Commander"}]) | |
;; A sequence of sequences, each with a maximum of 15 total keys in | |
;; each group. The first group has 13 total keys and the second group | |
;; has 3. Each group can successfully be passed to a function to | |
;; insert them into the database. | |
;; Other examples: | |
(chunk-rows rows 7) | |
;; ([{:name "Picard", :shirt "red", :rank "Captain"} | |
;; {:name "Riker", :shirt "red", :rank "Commander"}] | |
;; [{:name "Data", :shirt "yellow", :rank "Lt. Commander"} | |
;; {:name "Worf", :shirt "yellow", :rank "Lt."} | |
;; {:name "Crusher"}] | |
;; [{:name "Troi", :shirt "teal", :rank "Lt. Commander"}]) | |
(chunk-rows rows 5) | |
;; ([{:name "Picard", :shirt "red", :rank "Captain"}] | |
;; [{:name "Riker", :shirt "red", :rank "Commander"}] | |
;; [{:name "Data", :shirt "yellow", :rank "Lt. Commander"}] | |
;; [{:name "Worf", :shirt "yellow", :rank "Lt."} | |
;; {:name "Crusher"}] | |
;; [{:name "Troi", :shirt "teal", :rank "Lt. Commander"}]) | |
;; (chunk-rows rows 2) ;; <- that will blow up, because there's no | |
;; hope for that insert anyway. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment