Skip to content

Instantly share code, notes, and snippets.

@ipv1337

ipv1337/backend-architecures.md

Forked from ragingwind/Backend Architectures Keywords and References.md
Created June 24, 2013 08:04
Show Gist options
  • Save ipv1337/5848454 to your computer and use it in GitHub Desktop.
Save ipv1337/5848454 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
backend-architecures.md

Backend Architectures

ARCHITECTURES

Twitter

ror, scala, jetty, erlang, thrift, mongrel, comet server, my-sql, memchached, varnish, kestrel(mq), starling, gizzard, cassandra, hadoop, vertica, munin, nagios, awstats

Facebook

php (with hiphop compiler), thrift, java(tomcat, jetty, minor),epoll, erlang, tornado, nodejs my-sql, memcahced, hadoop, hbase, hive, scribe(-hdfs), bigpipe, varnish, haystack, cassandra

Tumblr (Korean)

centos, sciapache, apache, nginx, (move out of)php, scala(selection), ruby, thrift, my-sql, redis, hbase, memcached, gearman, kafka, kestrel, finagle, varnish, ha-proxy, func, capistrano, puppet, jenkins

Quora

aws(ec2, s3), ubuntu, cloudfront, python, pylons, paste, tornado, thrift, comet server, memcached, haproxy, nginx

Pinterest

python, django, tornado, node.js, rabbitmq, nginx, haproxy, varnish, memcached, membase, redis, my-sql, mrjob, hadoop(elastic map reduce)

Instagram (Korean)

aws(s3, ebs), cloudfront, ubuntu, django(high-cpu extra-large), gunicorn, fabric, gearman, pyapns, twisted, postgre-sql(quadruple extra-large), mdadm(sofeware raid with ebs), repmgr, pgbouncer, redis, memcached, node2dm, munin, pingdom, pagerduty, sentry

Pulse

aws(ec2, s3, elb), tornado, scribe, mrjob, node-readability, haproxy, tornado, gae, mapreduce, django(appengine), google-cloud-storage, memcache, redis

Netflix

aws(ec2, s3, ebs, rds, dynamodb, sdb, sqs, sns, emr, elb, eip, vpc, direct-connect, iam), java(tomcat), mongodb, my-sql, casandra, hadoop, zookeeper, evcache, asgard, groovy, grails, zuul, priam and more netflix opensouces)

Spoqa

linux(2.6), nginx, uwsgi, aws(s3), dotcloud, mysql, redis, celery

StyleShare

ubuntu(12.04), aws(ec2, s3, elb), nginx, werkzeug, flask, postgre-sql, pgpool, memcached, gevent, celery, rabbitmq, fabric, boto, exceptional, flask-exceptional

Gae9(개9)

rabbitmq, celery, phash

TheBorn Coupon(더본쿠폰)

ubuntu(12.04), nhn ncloud, django, apache, mod_wsgi, ms-sql, memcached, fabric, south, wand, rsync, py-bcrypt, python-gcm, apns

Lezhin Comics(레진코믹스)

gae, "천만명 이하 규모는 구글 앱 엔진을 써도 충분하다."

REFERENCES

INFRA, PLATFORMS, FRAMEWORKS

DATABASE, STORAGE

  • PostgreSQL: most advanced open source database.
  • repmgr: open source tools that helps DBAs and System administrators manage a cluster of PostgreSQL databases.
  • pgpool Wiki: middleware that works between PostgreSQL servers and a PostgreSQL database client.
  • PgBouncer: lightweight connection pooler for PostgreSQL.
  • SQLAlchemy: Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.
  • South: intelligent schema and data migrations for ​Django projects.
  • twitter/gizzard · GitHub: flexible sharding framework for creating eventually-consistent distributed datastores
  • cassandra: used for high velocity writes, and lower velocity reads
  • hadoop: process unstructured and large datasets, hundreds of billions of rows.
  • vertica: used for analytics and large aggregations and joins so they don't have to write MapReduce jobs. (twitter)
  • mrjob: Run MapReduce jobs on Hadoop or Amazon Web Services
  • Apache Solr: popular, blazing fast open source enterprise search platform from the Apache LuceneTM project
  • fatcache: Memcache on SSD.
  • google-cloud-storage: Store, access and manage your data on Google’s storage infrastructure. Take advantage of the scale and efficiency we have built over the years.
  • haystack: Facebook photo Infrastructure.
  • Netflix/EVCache: distributed in-memory data store for the cloud.
  • GAE, Google App Engine: Lets you run web applications on Google's infrastructure. App Engine applications are easy to build, easy to maintain, and easy to scale as your traffic and data storage needs grow. With App Engine, there are no servers to maintain

DEPLOY, MONITORING, UTILITIES

  • Fabric: library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
  • boto/boto · GitHub: python interface to aws
  • capistrano/capistrano · GitHub: Remote multi-server automation tool.
  • puppetlabs/puppet · GitHub: Server automation framework and application.
  • Exceptional: Exceptional tracks errors in web apps. It reports them in real-time.and gathers the info you need to fix them fast.
  • jzempel/flask-exceptional · GitHub: Exceptional extension for Flask.
  • rsync: utility that provides fast incremental file transfer.
  • Nagios: The Industry Standard in IT Infrastructure Monitoring
  • Munin: networked resource monitoring tool that can help analyze resource trends
  • AWStats: powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically
  • pingdom: Website monitoring. Monitor your server and network uptime and performance for free.
  • sentry: realtime error logging and aggregation platform
  • pagerduty: SaaS IT on-call schedule management, alerting and incident tracking.
  • scribe: server for aggregating log data that's streamed in realtime from clients. It is designed to be scalable and reliable
  • Sphinx: tool that makes it easy to create intelligent and beautiful
  • pHash.org: perceptual hash library
  • dahlia/wand · GitHub: The ctypes-based simple ImageMagick binding for Python.
  • py-bcrypt: strong password hashing for Python.
  • mdadm: manage MD devices aka Linux Software RAID.
  • twitter/snowflake · GitHub: network service for generating unique ID numbers at high scale with some simple guarantees
  • node-readability: Server side readability with node.js
  • Netflix/asgard: Web interface for application deployments and cloud management in Amazon Web Services (AWS)
  • Netflix/Priam: Co-Process for backup/recovery, Token Management, and Centralized Configuration management for Cassandra.
  • Netflix/zuul: edge service that provides dynamic routing, monitoring, resiliency, security, and more.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment