Michael Drogalis MichaelDrogalis

Onyx 0.5.0: The Cluster a Value

Onyx 0.5.0 hit Clojars this morning, and I couldn't be more proud. More than 4 months of hard work has gone into this release, featuring some of my most creative work to date. At the top of the list of things to discuss is a prolific design change: Onyx has become a masterless compute system. Following directly as a result of this new capability are a few distinctive features - including realtime notification of cluster events via core.async, tunable, fine grained cluster allocation algorithms, and a host of new APIs to enhance usability.

In this post, I'm going to gently take you through I was able to achieve a compute system with no centralized coordinating entity. For a very in depth explanation about how everything works, you can check out the Onyx Internal Documentation, which details algorithm specifics, step-by-step examples, and edge cases. Next, I'll describe how to use some

	(ns dire-vec.core
	(:require [dire.core :refer :all]))

	(defn f-a [] (prn "Called A"))

	(defn f-b [] (prn "Called B"))

	(def fs [{:f #'f-a :m "In before A"} {:f #'f-b :m "In before B"}])

	(doseq [f fs]

	(def chan-a (chan 10)) ;; channel with a buffer of 10 elements
	(def chan-b (chan 10))
	(def chan-c (chan 10))

	(>!! chan-a :message-1) ;; >!! = put onto channel
	(>!! chan-b :message-2)
	(>!! chan-c :message-3)

	(loop [chs [chan-a chan-b chan-c]]
	;; alts!! = try to take a value from any of the supplied channels

	<configuration xmlns="urn:hornetq"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

	<paging-directory>data/paging</paging-directory>

	<bindings-directory>data/bindings</bindings-directory>

	<journal-directory>data/journal</journal-directory>

	(defn graph-to-onyx-tree [graph]
	(reduce
	(fn [workflow root]
	(if (set? root)
	(reduce (fn [result input] (assoc result input (get graph root))) workflow root)
	(assoc workflow root (get graph root))))
	{}
	(keys graph)))

	(graph-to-onyx-tree {#{:A :B :C} {:D :E} :F {:G :H}})

	(def xform
	(comp (map inc)
	(filter even?)
	(dedupe)))

	(def data (vec (interleave (range 18) (range 18))))

	;; lazily transform the data
	(sequence xform data)

	(map #(nth (conj (cycle [% % "Fizz" % "Buzz" "Fizz" % % % "Buzz" % "Fizz" % % "FizzBuzz"]) %) %) (range 1 101))

	;; => (1 2 "Fizz" 4 "Buzz" "Fizz" 7 8 9 "Buzz" 11 "Fizz" 13 14 "FizzBuzz" ...)