transit.org

Transit

• Craig Andera
• Cognitect
• [email protected]
• http://cognitect.com/podcast

What Problem?

• Sending values between applications
• Written in different languages
• Self-describing
• With extensibility
• Good performance

Why not just use…?

• JSON
• XML
• ___
• Fressian/EDN

{:foo 1
 :bar {:quux [2 3 4]
       :baaz #Point [1 2]}
 :beer-inst #inst "2014-08-26T18:45"}

Basic Idea

• EDN/Fressian approach
• Target existing high-performance parsers
  • JSON, msgpack
• Minimal but rich core type set
• Tagged extensions
• Language mapping to representations

Core

• Ground types
  • nil/null, string, boolean, integers, floats
  • vector, map (stringable keys)
• Included taggged
  • keyword, symbols, bigdec, uuid, set, list, instant
  • maps with composite keys

Libraries

• Clojure
• Java
• Javascript
• ClojureScript
• Python
• Ruby
• Non-Cognitect
  • OCaml
  • Erlang

Examples

(require '[cognitect.transit :as transit])
(import [java.io ByteArrayInputStream ByteArrayOutputStream])

(def out (ByteArrayOutputStream. 4096))
(def writer (transit/writer out :json))

(transit/write writer [1 nil "hi" :foo])
(.toString out)

(->  "[1,null,\"hi\",\"~:foo\"]"
     .getBytes
     java.io.ByteArrayInputStream.
     (transit/reader :json)
     transit/read)

Extensibility

• Provide write handler
  • Implement WriteHandler
• Write handler specifies name and encoding
• Encoding specified in terms of base types
• Can be ignored, flows
  • Unrecognized tag yields TaggedValue

(def out (java.io.ByteArrayOutputStream. 4096))
(defrecord Point [x y])
(defrecord Size [w h])
(defrecord Rectangle [origin size])
(defn size-rep [size]
  [(:w size) (:h size)])
(defn point-rep [pt]
  [(:x pt) (:y pt)])
(defn rectangle-rep [rect]
  [(:origin rect) (:size rect)])
(def writer
  (transit/writer out
                  :json
                  {:handlers {user.Point (transit/write-handler "Point"
                                                                point-rep)
                              user.Size (transit/write-handler "Size" size-rep)
                              user.Rectangle (transit/write-handler "Rect"
                                                                    rectangle-rep)}}))
(transit/write writer (Point. 3 17))
(transit/write writer (Rectangle. (Point. 0 1) (Size. 2 3)))
(.toString out)

(let [out (ByteArrayOutputStream. 4096)
      write-handlers {user.Point (transit/write-handler "Point"
                                                        point-rep)
                      user.Rectangle (transit/write-handler "Rect"
                                                            rectangle-rep)}
      read-handlers {"Rect" (transit/read-handler (fn [[origin size]]
                                                    {:o origin :s size}))
                     ;; "Point" (transit/read-handler (fn [[x y]]
                     ;;                         {:x x :y y}))
                     }
      writer (transit/writer out
                             :json
                             {:handlers write-handlers})]
  (transit/write writer (Rectangle. (Point. 0 1) (Point. 2 3)))
  (-> out
      .toByteArray
      java.io.ByteArrayInputStream.
      ;; Note: no Point reader
      (transit/reader :json {:handlers read-handlers})
      transit/read))

Dynamic Caching

• Lots of repeated keys
• Fixed-size circular rotation cache
  • size ???
• Keywords/symbols cached everywhere
• Strings cached when map keys
• Reader must support, writer may cache

(def out (ByteArrayOutputStream. 4096))
(def writer (transit/writer out :json))
;; (def writer (transit/writer out :json-verbose))
(transit/write writer (into [] (repeat 3 {"long" "hi"
                                          "s" "bye"})))
(.toString out)

Examples

(defn transit [val]
  (let [out (ByteArrayOutputStream. 4096)
        writer (transit/writer out :json)]
    (transit/write writer val)
    (format "%s => %s" (pr-str val) (.toString out))))

(transit {"a" 1 "b" 2})

Level 1

Here is some prose. I have emphasis and italics. And I can do code.

Name	Age	Age in Months
Craig	42	504
Jason	35	420
		0

Org mode tricks

A little helper code:

(defn thresholds
  "Given a seq of thresholds in the range 0.0 to 1.0, and a collection
  of data, sorts the collection and returns the values that are at the
  positions indicated by the thresholds. E.g. (thresholds [0.5 0.75]
  (range 10)) returns {0.5 4, 0.75 6}."
  [tholds coll]
  (let [sorted (-> coll sort vec)
        n (count sorted)]
    (into {}
          (for [thold tholds]
            [thold (nth sorted (* thold n))]))))

I want to measure this function:

(defn bar [n]
  (rand-int (inc (* 2 n))))

Here’s our experiment

(->> (repeatedly 100 #(bar n))
     (thresholds [0.5 0.9 0.99])
     (into (sorted-map))
     vals
     (into [(str (java.util.Date.))
            n
            [:times-2 :plus-1]]))

Date	n	variations	50%	90%	99%
Tue Aug 26 20:33:06 EDT 2014			47	89	99
Tue Aug 26 20:34:04 EDT 2014			43	87	99
Tue Aug 26 20:34:05 EDT 2014			54	89	99
Tue Aug 26 20:34:06 EDT 2014			54	91	99
Tue Aug 26 20:35:33 EDT 2014		(:times-2)	100	185	198
Tue Aug 26 20:37:09 EDT 2014		(:times-2 :plus-1)	112	182	200
Tue Aug 26 20:38:12 EDT 2014	100	(:times-2 :plus-1)	97	185	199
Tue Aug 26 20:38:17 EDT 2014	1000	(:times-2 :plus-1)	1072	1855	2000

Variations:

Name	Explanation
times-2	Multiply operand by two