Bare message queue using the filesystem & JVM.

README.md

Demonstration of a file based concurrent message queue with the JVM (using Nashorn in this case).

We use simple file locking to write to the file. I managed to get 25k messages/s in doing this on Mac OS X which is plenty fast for most projects.

Notice how read.js is calling into a process. This means we can call into an external SSH server very easily to consume events. They can be consumed by ANY language or client we want. Bash, Python, awk, Scala, Java, Perl, JavaScript, ...

write-many.js demonstrates how to continuously append. This is not batched so likely to be much slower.

write.js takes standard input so we can have a very generic producer.

You can also produce into a remote server very very easily. Just put write.js and then cat | ssh target@host write.js! If that's not elegant for you you may write a REST server for it too.

For replication, need to have a think, that might be a bit more difficult to figure out :-) but in most cases you just need a back up and not proper replication.

I wouldn't use this for large scale production stuff, as that's where you might want to use Kafka instead. However this is all pluggable with Kafka - ie you can begin with this dead simple solution and swap out with Kafka later on to add extra reliability and scalability and all that stuff.

read.js

#!/usr/bin/jjs -scripting
var inputProcess = new java.lang.ProcessBuilder("tail", "-f", "-n", "+0", "/Users/me/sample.txt")
var process = inputProcess.start()
var inputStreamReader = new java.io.InputStreamReader(process.getInputStream())
var bufferedReader = new java.io.BufferedReader(inputStreamReader)
var line = bufferedReader.readLine()
while(!java.lang.Thread.interrupted() && !java.lang.System.out.checkError() && line != null) {
  print(line)
  line = bufferedReader.readLine()
}

write-many.js

#!/usr/bin/jjs -scripting
var tf = "/Users/me/sample.txt"
var raf = new java.io.RandomAccessFile(tf, "rw").getChannel()

while ( !java.lang.System.out.checkError() ) {
  var lock = raf.lock()
  try {
    var line = "this is definitely working for me all the time this is definitely working for me all the time ";
    java.nio.file.Files.write(java.nio.file.Paths.get(tf), (line + "\n").getBytes("UTF-8"), java.nio.file.StandardOpenOption.APPEND)
  } finally {
    lock.release()
  }
}

write.js

#!/usr/bin/jjs -scripting
var inputStreamReader = new java.io.InputStreamReader(java.lang.System.in);
var bufferedReader = new java.io.BufferedReader(inputStreamReader);
var line = bufferedReader.readLine();
var tf = "/Users/me/sample.txt"
var raf = new java.io.RandomAccessFile(tf, "rw").getChannel()

while ( !java.lang.System.out.checkError() && line != null ) {
  var lock = raf.lock()
  try {
    java.nio.file.Files.write(java.nio.file.Paths.get(tf), (line + "\n").getBytes("UTF-8"), java.nio.file.StandardOpenOption.APPEND)
  } finally {
    lock.release()
  }
  line = bufferedReader.readLine();
}