Skip to content

Instantly share code, notes, and snippets.

@daveliepmann
Created January 27, 2014 01:59
Show Gist options
  • Save daveliepmann/8642188 to your computer and use it in GitHub Desktop.
Save daveliepmann/8642188 to your computer and use it in GitHub Desktop.
A port of Jer Thorp's New York Times API demo (https://github.com/blprnt/dataart/tree/master/1_Data_and_Aesthetic/Code/NYT_ArticleSearch_v2) from Processing to Clojure using Quil
(ns dataartclj.nyt-monkeys
(:use quil.core)
(:use ring.util.codec)
(:require [cheshire.core :refer :all]))
(def api-key "removed")
(def base-url "http://api.nytimes.com/svc/search/v2/articlesearch.json?")
(def w 1280)
(def h 720)
;; NB: dates must be 8 digits, YYYYMMDD
(defn article-search-by-date
[query start-date end-date]
(parse-string (slurp (str base-url
"q=" (ring.util.codec/url-encode query)
"&begin_date=" start-date
"&end_date=" end-date
"&api-key=" api-key))
true))
;; ;; Hits per year are defined as follows. Use pmap to get it faster if
;; ;; you can bypass rate limiting.
;; (mapv #(:hits (:meta (:response
;; (article-search-by-date "monkeys"
;; (read-string (str % "0101"))
;; (read-string (str % "1231"))))))
;; (range 1901 1982))
;; I exceeded the queries-per-second limit of the NYT API, but here
;; are the results of the sexp above:
(def hits-per-year
'(66 79 67 59 57 67 62 64 80 59 80 51 60 46 29 46 18 9 20 34 37 66 58 86 125 89 111 114 83 93 107 90 83 100 126 82 114 81 68 82 57 59 48 49 48 58 56 50 51 51 64 64 77 59 92 52 74 87 119 70 77 68 41 66 71 58 77 61 68 67 73 91 82 85 84 92 86 90 88 113 87))
(defn setup []
(smooth)
(background 255)
(no-loop))
(def i (atom 0))
(defn draw []
(fill 0 150)
(let [years (count hits-per-year)]
(reset! i 0)
(doseq [hits hits-per-year]
(rect (map-range @i 0 years 100 (- w 100))
(- h 50)
(/ (- w 200) years)
(- 0 (map-range hits 0 (apply max hits-per-year) 0 (- h 100))))
(swap! i inc))))
(defsketch monkey-hits-per-year-in-the-nyt
:draw draw
:setup setup
:size [w h])
;; "There is really a lot more we can do with this."
;; "Here, a search for 'bin laden' on the day he was killed"
(def binladen (article-search-by-date "bin laden" 20110502 20110502))
;; "Find out how many articles"
(:hits (:meta (:response binladen)))
;; 88
;; "Get the headline of the fourth one"
(:headline (nth (:docs (:response binladen)) 3))
;; {:main "Bin Laden Dead", :kicker "Paul Krugman"}
;; "And a snippet from the second one"
(:snippet (second (:docs (:response binladen))))
;; "Based on the Intrade odds on whether Osama bin Laden would be caught by September 2011, it appears someone made a lot of money yesterday."
;; "We can get access to any of the fields that are returned (see API docs). For example the URL to the first story:"
(:web_url (first (:docs (:response binladen))))
;; "http://www.nytimes.com/interactive/2011/05/02/world/20110502-osama-timeline.html"
;; "Or, a JSON list of the keywords associated with the story"
(:keywords (first (:docs (:response binladen))))
;; [{:name "subject", :value "TERRORISM"} {:name "organizations", :value "AL QAEDA"} {:name "glocations", :value ""} {:name "persons", :value "BIN LADEN, OSAMA"} {:name "creative_works", :value ""}]
;; or, if you prefer:
(map vals (:keywords (first (:docs (:response binladen)))))
(("subject" "TERRORISM") ("organizations" "AL QAEDA") ("glocations" "") ("persons" "BIN LADEN, OSAMA") ("creative_works" ""))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment