Nicolas Lacasse nlacasse
nlacasse
/ gist:4255072
Created
December 11, 2012 01:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [97] nlacasse@fibonazi:~> mkdir test | |
| [98] nlacasse@fibonazi:~> stat test | |
| File: ‘test’ | |
| Size: 4096 Blocks: 8 IO Block: 4096 directory | |
| Device: 803h/2051d Inode: 12453533 Links: 2 | |
| Access: (0755/drwxr-xr-x) Uid: ( 1000/nlacasse) Gid: ( 100/ users) | |
| Access: 2012-12-10 17:44:42.579933028 -0800 | |
| Modify: 2012-12-10 17:44:42.579933028 -0800 | |
| Change: 2012-12-10 17:44:42.579933028 -0800 | |
| Birth: - |
nlacasse
/ gist:4504814
Created
January 10, 2013 19:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| creeper@ip-10-196-17-120:/mnt$ head -c 1000 dbp.json | |
| {"mode":"EXTENDED","vertices":[{"dbp-resource":{"type":"string","value":"<http://dbpedia.org/resource/Phagophobe>"},"unique-entity-id":{"type":"string","value":"7:11:<http://dbpedia.org/resource/Phagophobe>"},"_id":485247004,"_type":"vertex"},{"dbp-resource":{"type":"string","value":"<http://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Volume_ray_casting.png/200px-Volume_ray_casting.png>"},"unique-entity-id":{"type":"string","value":"7:11:<http://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Volume_ray_casting.png/200px-Volume_ray_casting.png>"},"_id":182663672,"_type":"vertex"},{"dbp-resource":{"type":"string","value":"<http://dbpedia.org/resource/Knesseth_Israel_(disambiguation)>"},"unique-entity-id":{"type":"string","value":"7:11:<http://dbpedia.org/resource/Knesseth_Israel_(disambiguation)>"},"_id":494210340,"_type":"vertex"},{"dbp-resource":{"type":"string","value":"<http://upload.wikimedia.org/wikipedia/commons/5/59/RamphocaenusCinereiventri |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (ns opal.dbpedia | |
| (:use [clojure.tools.logging :only [log]]) | |
| (:require [clojure.java.io :as io]) | |
| (:import [uk.ac.manchester.cs.owl.owlapi.turtle.parser TurtleParser] | |
| [org.neo4j.unsafe.batchinsert BatchInserters | |
| LuceneBatchInserterIndexProvider] | |
| [org.neo4j.graphdb DynamicRelationshipType])) | |
| ;; PARSING METHODS |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (defn insert-resource-node! | |
| [inserter res] | |
| (if-let [id (get @id-map res)] | |
| ; If the resource has aleady been added, just return the id. | |
| id | |
| ; Otherwise, add the node for the node, and remember its id for later. | |
| (let [id (.createNode inserter {"resource" res})] | |
| (swap! id-map #(assoc! % res id)) | |
| id))) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (defn connect-resource-nodes! | |
| [inserter node1 node2 label] | |
| (let [relationship (DynamicRelationshipType/withName label)] | |
| (.createRelationship inserter node1 node2 relationship nil))) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (defn insert-tuple! | |
| [inserter tuple] | |
| ; Get the resource and label names out of the tuple. | |
| (let [[resource-1 label resource-2 & _ ] tuple | |
| ; Upsert the resource nodes. | |
| node-1 (insert-resource-node! inserter resource-1) | |
| node-2 (insert-resource-node! inserter resource-2)] | |
| ; Connect the nodes with an edge. | |
| (connect-resource-nodes! inserter node-1 node-2 label))) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (defn -main [graph-path & files] | |
| (let [inserter (BatchInserters/inserter graph-path)] | |
| (doseq [file files] | |
| (log :debug (str "Loading file: " file)) | |
| (let [c (atom 0)] | |
| (doseq [tuple (parse-file file)] | |
| (if (= (mod @c 10000) 0) | |
| (log :debug (str file ": " @c))) | |
| (swap! c inc) | |
| (insert-tuple! inserter tuple)))) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (ns opal.dbpedia | |
| (:use [clojure.tools.logging :only [log]]) | |
| (:require [clojure.java.io :as io]) | |
| (:import [uk.ac.manchester.cs.owl.owlapi.turtle.parser TurtleParser] | |
| [org.neo4j.unsafe.batchinsert BatchInserters | |
| LuceneBatchInserterIndexProvider] | |
| [org.neo4j.graphdb DynamicRelationshipType])) | |
| ;; PARSING METHODS |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (defn insert-tuple! | |
| [inserter tuple] | |
| ; Get the resource and label names out of the tuple. | |
| (let [[resource-1 label resource-2 & _ ] tuple | |
| ; Upsert the resource nodes. | |
| node-1 (insert-resource-node! inserter resource-1) | |
| node-2 (insert-resource-node! inserter resource-2)] | |
| ; Connect the nodes with an edge. | |
| (connect-resource-nodes! inserter node-1 node-2 label))) |
nlacasse
/ gist:4957289
Created
February 14, 2013 23:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 1360880797.773173 "sismember" "feeds" "newuser" | |
| 1360880797.773992 "MULTI" | |
| 1360880797.774022 "lpush" "feed.ids:newuser" "d851f290-d8ce-4f63-ad3c-f95f401f5bdf" | |
| 1360880797.774038 "incr" "feed.publishes:newuser" | |
| 1360880797.774052 "hset" "feed.items:newuser" "d851f290-d8ce-4f63-ad3c-f95f401f5bdf" "{\"uuid\":\"fb3760fa-a6a5-4e89-ac32-c393bab95b2a\",\"name\":\"Okie Kupid\",\"emails\":[{\"value\":\"[email protected]\",\"valid\":false}],\"services\":[{\"service\":\"facebook\",\"service_id\":\"100004601530540\",\"token\":\"AAAHCBLxw6KgBAFoSnCoqWHc4vlRlqJ7PbKdUhZCZAuq4nkbYqX8OGW7yCjDb1k4OipPYXTDgB5wmjjuBUx99rlUA8GgACvObWcPjjh0QZDZD\",\"secret\":null,\"expires\":\"2013-04-15T22:25:18.591Z\",\"permissions\":{\"email\":true,\"read_stream\":true,\"user_relationships\":true,\"user_location\":true,\"user_likes\":true,\"user_activities\":true,\"user_interests\":true,\"user_events\":true,\"user_photos\":true,\"user_status\":true,\"friends_relationships\":true,\"friends_location\":true,\"friends_likes\":true,\"friends_acti |