Skip to content

Instantly share code, notes, and snippets.

View hoffrocket's full-sized avatar
👋

Jon Hoffman hoffrocket

👋
  • Anomaly
  • New York, NY
View GitHub Profile
@hoffrocket
hoffrocket / tileraemail.md
Created July 20, 2012 16:02
my dad on tilera cards

Use a 64-core PCI-E board (Made by Tilera: http://www.tilera.com/sites/default/files/productbriefs/TILEncore-Gx36_PB028-02.pdf ) with two onboard 10GB NICS in in-line mode, each core can run its own copy of a hard real-time (RTOS) BSD microkernel with a shared RDMA buffers ad/or a group of CPUS can be run in SMP or parallel mode for heavy calculations.

Pin the market data to as single core or multiple cores (e.g. one for tick data, one for completed orders). Offload all TCP (TOE, and a few dozen settings) to the NIC chips and share a revolving buffer using RDMA (this provides a lock-free mechanism with no mutexes, no context switching, and no buffer copying).

Cascade the market data into additional cores in SMP or Parallel mode (depending upon the calculation attributes) for quantitative algorithmic calculations (e.g. in-stream Kalman filters, SMC/UPF, ANNs etc.). Outbound orders on the second NIC fed by a microkernel pinned to a single core.

Configuration is also be used for ultra fast order cross

@hoffrocket
hoffrocket / TimeTest.java
Created August 2, 2012 16:15
higher precision currentTime in java
import static org.junit.Assert.*;
import org.junit.Test;
public class TimeTest {
private final static long nanoTimeOffset = (System.currentTimeMillis() * 1000000) - System.nanoTime();
public static long currentTimeNanos() {
return System.nanoTime() + nanoTimeOffset;
}
@hoffrocket
hoffrocket / WriteBufferTest.scala
Created August 14, 2012 21:50
WriteBufferTest
import java.io.ByteArrayOutputStream
import java.io.OutputStreamWriter
object WriteBufferTest {
def writeData(appendable: java.lang.Appendable) {
var i = 0
while (i < 1000) {
appendable.append("hello there ").append(i.toString())
@hoffrocket
hoffrocket / parhttp.py
Created August 28, 2012 00:29
Python parallel http requests using multiprocessing
#!/usr/bin/env python
from multiprocessing import Process, Pool
import time
import urllib2
def millis():
return int(round(time.time() * 1000))
def http_get(url):
@hoffrocket
hoffrocket / StreamingDump.scala
Created August 29, 2012 20:33
Mongo streaming dump in scala
import java.io.ByteArrayInputStream
import java.io.ByteArrayOutputStream
import java.io.EOFException
import java.io.InputStream
import java.io.OutputStream
import org.bson.BSONCallback
import org.bson.BSONObject
import com.mongodb.DBCallback
db.getCollectionNames().forEach(function(n){
var shardConfig = db.getSisterDB('config').collections.findOne({_id:"foursquare." + n},{key:1});
if (shardConfig){
var shardKeyNameParts = [];
for (var keyPart in shardConfig.key) {
shardKeyNameParts.push(keyPart + "_" + shardConfig.key[keyPart]);
}
var shardKeyName = shardKeyNameParts.join('_');
var shardKeySizeMb = Math.round(db[n].stats().indexSizes[shardKeyName]/1024/1024);
db[n].getIndexes().forEach(function(i){
@hoffrocket
hoffrocket / disable-java.applescript
Last active December 14, 2015 00:18
Applescript to disable java in safari
#!/usr/bin/osascript
activate application "Safari"
tell application "System Events"
tell process "Safari"
click menu item "Preferences…" of menu 1 of menu bar item "Safari" of menu bar 1
click button "Security" of tool bar 1 of window 1
set theCheckbox to checkbox "Enable Java" of group 1 of group 1 of window "Security"
tell theCheckbox
if (its value as boolean) then click theCheckbox
@hoffrocket
hoffrocket / FileTailer.scala
Last active December 14, 2015 02:59
Buffered FileTailer in scala.
import java.io.{File, RandomAccessFile}
import java.nio.{ByteBuffer, CharBuffer}
import java.nio.charset.Charset
/**
* Read lines from the end of a file.
*
* Partial lines that don't terminate with a newline will be lost
* Lines bigger than the buffer size will also be lost
*/
@hoffrocket
hoffrocket / .pylintrc
Created March 1, 2013 16:49
pylintrc
[MESSAGES CONTROL]
# Brain-dead errors regarding standard language features
# W0142 = *args and **kwargs support
# W0403 = Relative imports
# Pointless whinging
# R0201 = Method could be a function
# W0613 = Unused argument
# W0232 = Class has no __init__ method
# R0903 = Too few public methods
@hoffrocket
hoffrocket / QueueSim.scala
Created March 6, 2013 23:05
Distributed Applicaiton Simulator (may be buggy and/or completely incorrect)
object QueueSim {
class QueuedProcessor extends Processor {
var lastClock: Int = 0
var remainingWork: Int = 0
def advanceClock(currentTime: Int) {
remainingWork = math.max(0, remainingWork - (currentTime - lastClock))
lastClock = currentTime
}
def process(request: Request): Response = {