Skip to content

Instantly share code, notes, and snippets.

View hoffrocket's full-sized avatar
👋

Jon Hoffman hoffrocket

👋
  • Anomaly
  • New York, NY
View GitHub Profile
db.getCollectionNames().forEach(function(n){
var shardConfig = db.getSisterDB('config').collections.findOne({_id:"foursquare." + n},{key:1});
if (shardConfig){
var shardKeyNameParts = [];
for (var keyPart in shardConfig.key) {
shardKeyNameParts.push(keyPart + "_" + shardConfig.key[keyPart]);
}
var shardKeyName = shardKeyNameParts.join('_');
var shardKeySizeMb = Math.round(db[n].stats().indexSizes[shardKeyName]/1024/1024);
db[n].getIndexes().forEach(function(i){
@hoffrocket
hoffrocket / StreamingDump.scala
Created August 29, 2012 20:33
Mongo streaming dump in scala
import java.io.ByteArrayInputStream
import java.io.ByteArrayOutputStream
import java.io.EOFException
import java.io.InputStream
import java.io.OutputStream
import org.bson.BSONCallback
import org.bson.BSONObject
import com.mongodb.DBCallback
@hoffrocket
hoffrocket / parhttp.py
Created August 28, 2012 00:29
Python parallel http requests using multiprocessing
#!/usr/bin/env python
from multiprocessing import Process, Pool
import time
import urllib2
def millis():
return int(round(time.time() * 1000))
def http_get(url):
@hoffrocket
hoffrocket / WriteBufferTest.scala
Created August 14, 2012 21:50
WriteBufferTest
import java.io.ByteArrayOutputStream
import java.io.OutputStreamWriter
object WriteBufferTest {
def writeData(appendable: java.lang.Appendable) {
var i = 0
while (i < 1000) {
appendable.append("hello there ").append(i.toString())
@hoffrocket
hoffrocket / TimeTest.java
Created August 2, 2012 16:15
higher precision currentTime in java
import static org.junit.Assert.*;
import org.junit.Test;
public class TimeTest {
private final static long nanoTimeOffset = (System.currentTimeMillis() * 1000000) - System.nanoTime();
public static long currentTimeNanos() {
return System.nanoTime() + nanoTimeOffset;
}
@hoffrocket
hoffrocket / tileraemail.md
Created July 20, 2012 16:02
my dad on tilera cards

Use a 64-core PCI-E board (Made by Tilera: http://www.tilera.com/sites/default/files/productbriefs/TILEncore-Gx36_PB028-02.pdf ) with two onboard 10GB NICS in in-line mode, each core can run its own copy of a hard real-time (RTOS) BSD microkernel with a shared RDMA buffers ad/or a group of CPUS can be run in SMP or parallel mode for heavy calculations.

Pin the market data to as single core or multiple cores (e.g. one for tick data, one for completed orders). Offload all TCP (TOE, and a few dozen settings) to the NIC chips and share a revolving buffer using RDMA (this provides a lock-free mechanism with no mutexes, no context switching, and no buffer copying).

Cascade the market data into additional cores in SMP or Parallel mode (depending upon the calculation attributes) for quantitative algorithmic calculations (e.g. in-stream Kalman filters, SMC/UPF, ANNs etc.). Outbound orders on the second NIC fed by a microkernel pinned to a single core.

Configuration is also be used for ultra fast order cross

@hoffrocket
hoffrocket / WebServer.java
Created June 21, 2012 15:34
tracelytics java api example
package jon.trace;
import java.io.IOException;
import java.io.OutputStream;
import java.io.PrintWriter;
import java.net.InetSocketAddress;
import java.util.Iterator;
import java.util.List;
import java.util.Set;
import java.util.concurrent.Executors;
@hoffrocket
hoffrocket / ebsdiskhealth.py
Created May 23, 2012 05:03
raid0 ebs disk health monitor
#!/usr/bin/env python
import atexit
import datetime
from datetime import timedelta
import logging
import os
import smtplib
import subprocess as sub
import sys
@hoffrocket
hoffrocket / LRUCache.java
Created March 4, 2012 01:17
Simple LRUCache in java (not threadsafe)
import java.util.HashMap;
import java.util.Map;
public class LRUCache<K, V> {
private final Map<K, Pair<Node<K>, V>> map = new HashMap<K, Pair<Node<K>,V>>();
private Node<K> head = null;
private Node<K> tail = null;
private final int maxSize;
@hoffrocket
hoffrocket / gmetric-kestrel.py
Created September 28, 2011 22:25
ganglia kestrel plugin
#!/usr/bin/env python
import memcache
import re
import subprocess
import sys
KESTREL_HOST = 'localhost:22133'
def record(name, value):