A checklist for designing and developing internet scale services, inspired by James Hamilton's 2007 paper "On Desgining and Deploying Internet-Scale Services."
| package benchmark; | |
| import org.openjdk.jmh.annotations.*; | |
| import java.lang.invoke.MethodHandle; | |
| import java.lang.invoke.MethodHandles; | |
| import java.lang.reflect.Field; | |
| import java.lang.reflect.InvocationTargetException; | |
| import java.util.concurrent.TimeUnit; |
"Consensus is impossible" is a gloss for "there's always an execution that doesn't terminate".
Once you've internalized that you can't distinguish slow nodes from dead nodes, how do you deal with that?
"When you're engaged in a battle of wits with a Sicilian and death is on the line, you certainly need to think about epistemic logic."
A correct distributed program achieves (nontrivial) distributed property X. So we need to ask:
| <!-- | |
| ~ Copyright (c) 2014 Scaling Data. All rights reserved. This gist is licensed under the Apache Software License v2.0. | |
| --> | |
| <assembly | |
| xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2" | |
| xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | |
| xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd"> | |
| <id>dist</id> |
| Producer | |
| Setup | |
| bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1 | |
| bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test --partitions 6 --replication-factor 3 | |
| Single thread, no replication | |
| bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196 |
As Kyle brought up, Consul at the moment has a single known case of a potential inconsistency (Could be unknown cases lurking). Currently Consul works by electing a leader, who "leases" the position for LeaderLeaseTimeout interval. At each interval, it checks that a quorum of nodes still believes it to be the leader. At the same time, if a follower does not hear from the leader within randomInterva(HeartbeatTimeout, 2 * HeartbeatTimeout), it will start a new election.
| # | |
| # tor relay / bridge config | |
| # | |
| # apt-get install tor | |
| # | |
| # yum install tor (after adding EPEL or similar as a repo) | |
| # | |
| # dump this config file into the default tor config file location (/etc/tor/torrc) | |
| # | |
| # see also: https://www.torproject.org/docs/installguide.html.en |
We can look at two sides of the management coin: What do the individuals get out of it? And what benefit does the whole system derive from it?
I will disregard any benefits that accrue to managers just by holding the position of managing. Those are just circular logic. Circular logic abounds in discussions of management and hierarchy. For example, consider status reports. It will be said that status reports are necessary so managers know what their employees are working on. It’s
| # Install `dnsmasq` and configure for *.test domains | |
| $ brew install dnsmasq | |
| $ vim /usr/local/etc/dnsmasq.conf | |
| # Reload configuration and clear cache | |
| $ sudo launchctl unload /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist | |
| $ sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist | |
| $ dscacheutil -flushcache |
| The regex patterns in this gist are intended only to match web URLs -- http, | |
| https, and naked domains like "example.com". For a pattern that attempts to | |
| match all URLs, regardless of protocol, see: https://gist.github.com/gruber/249502 | |
| License: https://opensource.org/license/bsd-3-clause | |
| # Single-line version: | |
| (?i)\b((?:https?:(?:/{1,3}|[a-z0-9%])|[a-z0-9.\-]+[.](?:com|net|org|edu|gov|mil|aero|asia|biz|cat|coop|info|int|jobs|mobi|museum|name|post|pro|tel|travel|xxx|ac|ad|ae|af|ag|ai|al|am|an|ao|aq|ar|as|at|au|aw|ax|az|ba|bb|bd|be|bf|bg|bh|bi|bj|bm|bn|bo|br|bs|bt|bv|bw|by|bz|ca|cc|cd|cf|cg|ch|ci|ck|cl|cm|cn|co|cr|cs|cu|cv|cx|cy|cz|dd|de|dj|dk|dm|do|dz|ec|ee|eg|eh|er|es|et|eu|fi|fj|fk|fm|fo|fr|ga|gb|gd|ge|gf|gg|gh|gi|gl|gm|gn|gp|gq|gr|gs|gt|gu|gw|gy|hk|hm|hn|hr|ht|hu|id|ie|il|im|in|io|iq|ir|is|it|je|jm|jo|jp|ke|kg|kh|ki|km|kn|kp|kr|kw|ky|kz|la|lb|lc|li|lk|lr|ls|lt|lu|lv|ly|ma|mc|md|me|mg|mh|mk|ml|mm|mn|mo|mp|mq|mr|ms|mt|mu|mv|mw|mx|my|mz|na|nc|ne|nf|ng|ni|nl|no|np|nr|nu|nz|om|pa|pe|pf|pg|ph|pk|pl|pm|pn|pr|ps |
