gerritjvv’s gists

gerritjvv / core.async-lazy-sequence

Created October 30, 2013 11:07

This is for a situation where you have N amount of threads reading from different sources and want to consume all of the data they produce as a single sequence. Can be described as merging N queues from different sources and works well when the data produced is from IO. e.g. My usage is with kafka, I have multiple kafka topics and partitions to …

	(use 'clojure.core.async)


	;this is the function you want to use
	(defn lazy-channels [chs]
	(lazy-seq (cons (let [ [v _] (alts!! chs)] v) (lazy-channels chs))))


	;now for the tesging
	(def chs [ (chan) (chan) (chan) ]) ; the channels can come from anywhere, here we are using three channels for testing

gerritjvv / gist:6508101

Created September 10, 2013 11:29

Connection as a Sequence sleep retry on nil Lets say you have a connection that you want to stream results from. The connection can also return zero data in which case you would want to sleep and retry until data is available. A complicated solution would be to do this with a loop recur but a more elegant pattern can be created using map and fil…


	;for testing purposes lets simulate a connection that will return nil or 1
	(defn connect-and-get [] (rand-nth [nil 1 1 1 nil 1 nil 1 nil]))

	(def select-data (repeatedly connect-and-get))

	(defn sleep-while-nil [ts s]
	(filter (complement nil?) (map (fn [x] (if-not x (Thread/sleep ts)) x) s)))

	(take 10 (sleep-while-nil 100 select-data))

gerritjvv / gist:6239976

Last active December 21, 2015 03:08

Clojure core.async schedule fixed delay actions

	;Print "Hi" every 2 seconds:



	(defn t []

	(go (loop [] (<! (timeout 2000)) (prn "hi") (recur))))


	;or as a macro

gerritjvv / Buffered IO - DB - Stream PATTERN

Last active December 20, 2015 11:59

This pattern helps to convert a connected resource (i.e. a resource accessed via some kind of connection) into a lazy sequence. The idea is to fill a in-memory buffer with data from a connection that is opened and closed. Then consume of the buffer, and when the buffer is empty fill it again with another connection.

	(defn buffered-select [f-select init-pos]
	"Creates a lazy sequence of messages for this datasource"
	(letfn [
	(m-seq [buff pos]
	(if-let [buff2 (if (empty? buff) (f-select pos) buff)]
	(cons (first buff2) (lazy-seq (m-seq (rest buff2) (inc pos) )))
	)
	)
	]
	(m-seq nil init-pos)

gerritjvv / gist:5955996

Created July 9, 2013 09:30

Count Frequencies of a column in a tsv file


	(def column-n (fn [n sep] (fn [record] (let [s (str record) arr (clojure.string/split s sep)] (arr n)))))

	(defn file-freq [file column sep]

	(with-open [ rdr (clojure.java.io/reader file) ]
	(-> (column-n column sep) (map (line-seq rdr)) frequencies)
	)
	)

gerritjvv / gist:5948398

Created July 8, 2013 12:29

Partitioning Lazy Sequences

	Caution: Partitioning lazy sequence code freeze

	(def l [1 2 3 4 5])
	;create a simple lazy sequence function testing only
	;(rdr l) returns a lazy sequence from l
	(def rdr (fn reader[x] (cons (first x) (lazy-seq (reader (rest x))))))

	;the line below will freeze
	(doall (partition-all 2 (rdr l)) )

gerritjvv / gist:5866679

Created June 26, 2013 11:19

Call external process from Java

	int ret = new ProcessBuilder("gzip", "-t", "t.txt.gz").inheritIO().start().waitFor()

gerritjvv / gist:5866665

Last active July 19, 2024 13:29

Calling an external process from Clojure

	(comment
	This is the easiest and most concise way of calling an external process in Java. The inheritIO methods causes the command to output stdout and errout to the same place as your current process (which most of the times is the console), no need to create mechanisms for reading asynchronously from input streams to get at the information.

	http://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html
	)

	(def ret (.waitFor (-> (ProcessBuilder. ["gzip" "-t" "g.txt.gz"]) .inheritIO .start)))

gerritjvv / gist:5807270

Created June 18, 2013 17:06

Clojure functions for dealing with ftp or sftp server using the commons vfs2 library. from https://github.com/gerritjvv/pseidon

	(ns pseidon.core.ds.ftp

	(:require [pseidon.core.conf :refer [get-conf get-conf2] ]

	)
	(:use clojure.tools.logging
	))

	(import
	'(org.apache.commons.vfs2 FileContent FileObject FileSystemManager FileSystemOptions VFS Selectors)

gerritjvv / gist:5755871

Created June 11, 2013 10:24

Clojure LazySequence From Java

	package lazyseq;

	import org.junit.Test;

	import clojure.lang.AFn;
	import clojure.lang.ASeq;
	import clojure.lang.ArraySeq;
	import clojure.lang.Cons;
	import clojure.lang.IPersistentMap;
	import clojure.lang.ISeq;

Gerrit gerritjvv