Skip to content

Instantly share code, notes, and snippets.

@drbobbeaty
Last active October 9, 2015 16:53
Show Gist options
  • Save drbobbeaty/427ec552d370d6538684 to your computer and use it in GitHub Desktop.
Save drbobbeaty/427ec552d370d6538684 to your computer and use it in GitHub Desktop.
Lazy problem of not being lazy
(defn- get-block
"Function to get one block from LenderX based on the URL and start and limit
values provided. Typically we should keep the `limit` to 25 or less, but this
function will be called for each block - as needed, from LenderX for the
provided URL."
[url st lim & [sess]]
(if (and (string? url) (not (neg? st)) (pos? lim))
(let [ep (str url (if (neg? (.indexOf url "?")) "?" "&") "start=" st "&limit=" lim)
]
(-> (do-get* cm ep (mk-headers "GET" ep nil sess) lenderx-opts)
(assoc :start st
:limit lim)))))
(log-execution-time! get-block {:msg-fn (fn [ret u s l & [sess]] (format "%s - %s" s l))})
(defn- get-all-remaining
"Function to use the `lazy-seq` and recursion to build up the sequence of
all the data that would be returned from LenderX for the GET on the URL
provided. This will be called once for each 'batch' of data and then stitch
it all together as needed."
[resp url & [sess]]
(let [{:keys [data success total start limit]} resp]
(concat
data
(lazy-seq
(if (and (pos? success) (= limit (count data)))
(get-all-remaining (get-block url (+ start limit) limit sess) url sess))))))
(defn get-all
"Function to GET the `url` from LenderX using it's paging scheme of 25 elements
at a time. The function will return a lazy sequence of the rows as returned from
the service."
[url & [sess]]
(lazy-seq
(get-all-remaining (get-block url 0 25 sess) url sess)))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment