| echo "GET http://localhost:3000/api/slow" | vegeta attack -duration=30s -rate=50 | vegeta report --type=text |
Picking the right architecture = Picking the right battles + Managing trade-offs
| Latency Comparison Numbers (~2012) | |
| ---------------------------------- | |
| L1 cache reference 0.5 ns | |
| Branch mispredict 5 ns | |
| L2 cache reference 7 ns 14x L1 cache | |
| Mutex lock/unlock 25 ns | |
| Main memory reference 100 ns 20x L2 cache, 200x L1 cache | |
| Compress 1K bytes with Zippy 3,000 ns 3 us | |
| Send 1K bytes over 1 Gbps network 10,000 ns 10 us | |
| Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD |
| Mutex: exclusive-member access to a resource | |
| Semaphore: n-member access to a resource | |
| That is, a mutex can be used to syncronize access to a counter, file, database, etc. | |
| A sempahore can do the same thing but supports a fixed number of simultaneous callers. For example, I can wrap my database calls in a semaphore(3) so that my multithreaded app will hit the database with at most 3 simultaneous connections. All attempts will block until one of the three slots opens up. They make things like doing naive throttling really, really easy. | |
| function Semaphore(max) { |
| function CustomEmitter() { | |
| this.events = {}; | |
| } | |
| CustomEmitter.prototype.on = function (type, listener) { | |
| // error handling | |
| this.events[type] = []; | |
| this.events[type].push(listener); | |
| }; |
| 'key.converter.key.subject.name.strategy': 'io.confluent.kafka.serializers.subject.TopicRecordNameStrategy', | |
| 'value.converter.value.subject.name.strategy': 'io.confluent.kafka.serializers.subject.TopicRecordNameStrategy', |
| Sharding partitions the data-set into discrete parts. | |
| Replication duplicates the data-set. | |
| These two things can stack since they're different. Using both means you will shard your data-set across multiple groups of replicas. Put another way, you Replicate shards; a data-set with no shards is a single 'shard'. | |
| A Mongo cluster with three shards and 3 replicas would have 9 nodes. | |
| 3 sets of 3-node replicas. | |
| Each replica-set holds a single shard. |
| docker exec e69e0bb63418 sh -c 'mongodump -u root -p admin --authenticationDatabase admin -d pia-entry-generator --archive --quiet' > db.dump | |
| docker exec -i e69e0bb63418 sh -c 'mongorestore -u root -p admin --authenticationDatabase admin --archive' < db.dump |
| // MONGO COMMAND LINE | |
| // start mongod | |
| ulimit -n 2048 && mongod | |
| // ctrl-c ends in same window, but can also use | |
| killall mongod | |
| // start mongod with data directory in another location | |
| mongod -dbpath /path/to/data/dir | |
| // open command line interface (mongo shell) | |
| mongo | |
| // restore a DB dump from a binary (BSON) dump file |
| db.getCollection('test').aggregate([{$match: {name: "item6"}},{$unwind: '$slots'}, {$sort: {"slots.date": -1}}, | |
| {$group: {_id: '$_id',root: { $mergeObjects: '$$ROOT' }, slots: {$push: '$slots'}}}, { "$limit": 1 }, { | |
| $replaceRoot: { | |
| newRoot: { | |
| $mergeObjects: ['$root', '$$ROOT'] | |
| } | |
| } | |
| }, { | |
| $project: { | |
| root: 0 |
