Skip to content

Instantly share code, notes, and snippets.

@ragingwind

ragingwind/Backend Architectures Keywords and References.md

Last active July 4, 2024 13:00
Show Gist options
  • Save ragingwind/5840075 to your computer and use it in GitHub Desktop.
Save ragingwind/5840075 to your computer and use it in GitHub Desktop.
Download ZIP
Backend Architectures Keywords and References
Raw
Backend Architectures Keywords and References.md

Backend Architectures

Twitter

ror, scala, jetty, erlang, thrift, mongrel, comet server, my-sql, memchached, varnish, kestrel(mq), starling, gizzard, cassandra, hadoop, vertica, munin, nagios, awstats

Facebook

php (with hiphop compiler), thrift, java(tomcat, jetty, minor), epoll, erlang, tornado, nodejs my-sql, memcahced, hadoop, hbase, hive, scribe(-hdfs), bigpipe, varnish, haystack, cassandra

Tumblr (Korean)

centos, sciapache, apache, nginx, (move out of)php, scala(selection), ruby, thrift, my-sql, redis, hbase, memcached, gearman, kafka, kestrel, finagle, varnish, ha-proxy, func, capistrano, puppet, jenkins

Quora

aws(ec2, s3), ubuntu, cloudfront, python, pylons, paste, tornado, thrift, comet server, memcached, haproxy, nginx

Pinterest

python, django, tornado, node.js, rabbitmq, nginx, haproxy, varnish, memcached, membase, redis, my-sql, mrjob, hadoop(elastic map reduce)

Instagram (Korean)

aws(s3, ebs), cloudfront, ubuntu, django(high-cpu extra-large), gunicorn, fabric, gearman, pyapns, twisted, postgre-sql(quadruple extra-large), mdadm(sofeware raid with ebs), repmgr, pgbouncer, redis, memcached, node2dm, munin, pingdom, pagerduty, sentry

Pulse

aws(ec2, s3, elb), tornado, scribe, mrjob, node-readability, haproxy, tornado, gae, mapreduce, django(appengine), google-cloud-storage, memcache, redis

Netflix

aws(ec2, s3, ebs, rds, dynamodb, sdb, sqs, sns, emr, elb, eip, vpc, direct-connect, iam), java(tomcat), mongodb, my-sql, casandra, hadoop, zookeeper, evcache, asgard, groovy, grails, zuul, priam and more netflix opensouces)

Spoqa

linux(2.6), nginx, uwsgi, aws(s3), dotcloud, mysql, redis, celery

StyleShare

ubuntu(12.04), aws(ec2, s3, elb), nginx, werkzeug, flask, postgre-sql, pgpool, memcached, gevent, celery, rabbitmq, fabric, boto, exceptional, flask-exceptional

Gae9(개9)

rabbitmq, celery, phash

TheBorn Coupon(더본쿠폰)

ubuntu(12.04), nhn ncloud, django, apache, mod_wsgi, ms-sql, memcached, fabric, south, wand, rsync, py-bcrypt, python-gcm, apns

Lezhin Comics(레진코믹스)

gae, "천만명 이하 규모는 구글 앱 엔진을 써도 충분하다.(Translate)" - @xguru

Cartoon Service(만화서비스) - Smartstudy

ubuntu(11.04) nbp (nhn business platform), my-sql(innodb), mongodb, django, fabric, uwsgi, nginx, memcached, cacti, npk, rsync, lftp

99designs

aws(ec2, s3, elb, rds, cloudwatch), varnish, php, ergo, my-sql, memcached, mongodb, redis, new-relic, statsd

TripAdvisor

linux, apache, java, tomcat, postgres-sql, lucene, velocity, memcached, jgroups, hadoop, cacti, nagios, custom[?]

  • supoorting infra: java, python, ruby, php, perl

Trello

nodejs, haproxy, redis, mongodb

GOV.UK

ubuntu(10.04), skyscape, akamai, puppet, puppetdb, nginx, unicorn, gunicorn, upstart, haproxy, varnish,

  • redirection: perl(manage and test), php(some), nodejs(side-by-side browser)
  • applications: ror, sinatra, scala, play(2.0), django, mapit
  • databases and other storage: mongodb, my-sql, postgres-sql, elasticsearch, solr, rabbitmq
  • monitoring, managing and alerting: statsd, logstash, ganglia, graphie, nagios
  • supporting tools: jenkins, new-relic, google-analytics, dyn, ses, font-forge, font-tools, zendesk, jekyll, heroku, clojure

StackExchange

windows server, sql-server, redis

YouPorn

php-frm, haproxy, activemq, varnish, redis, nginx, my-sql, syslog-ng, symfony2

LinkedIn

SoundCloud

Tagged

REFERENCES

INFRA, PLATFORMS, FRAMEWORKS

DATABASE, STORAGE, DATA-MINING

  • PostgreSQL: most advanced open source database.
  • repmgr: open source tools that helps DBAs and System administrators manage a cluster of PostgreSQL databases.
  • pgpool Wiki: middleware that works between PostgreSQL servers and a PostgreSQL database client.
  • PgBouncer: lightweight connection pooler for PostgreSQL.
  • SQLAlchemy: Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.
  • South: intelligent schema and data migrations for ​Django projects.
  • twitter/gizzard · GitHub: flexible sharding framework for creating eventually-consistent distributed datastores
  • cassandra: used for high velocity writes, and lower velocity reads
  • hadoop: process unstructured and large datasets, hundreds of billions of rows.
  • vertica: used for analytics and large aggregations and joins so they don't have to write MapReduce jobs. (twitter)
  • mrjob: Run MapReduce jobs on Hadoop or Amazon Web Services
  • Apache Lucene - Welcome to Apache Lucene
  • Apache Solr: popular, blazing fast open source enterprise search platform from the Apache LuceneTM project
  • fatcache: Memcache on SSD.
  • google-cloud-storage: Store, access and manage your data on Google’s storage infrastructure. Take advantage of the scale and efficiency we have built over the years.
  • haystack: Facebook photo Infrastructure.
  • Netflix/EVCache: distributed in-memory data store for the cloud.
  • Elasticsearch: Open Source Distributed Real Time Search & Analytics
  • SQL-Server: Microsoft SQL Server.
  • Doctrine: PHP libraries primarily focused on providing persistence services

DEPLOY, MONITORING, UTILITIES

  • Fabric: library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
  • boto/boto · GitHub: python interface to aws
  • capistrano/capistrano · GitHub: Remote multi-server automation tool.
  • puppetlabs/puppet · GitHub: Server automation framework and application.
  • Exceptional: Exceptional tracks errors in web apps. It reports them in real-time.and gathers the info you need to fix them fast.
  • jzempel/flask-exceptional · GitHub: Exceptional extension for Flask.
  • rsync: utility that provides fast incremental file transfer.
  • Nagios: The Industry Standard in IT Infrastructure Monitoring
  • Munin: networked resource monitoring tool that can help analyze resource trends
  • AWStats: powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically
  • pingdom: Website monitoring. Monitor your server and network uptime and performance for free.
  • sentry: realtime error logging and aggregation platform
  • pagerduty: SaaS IT on-call schedule management, alerting and incident tracking.
  • scribe: server for aggregating log data that's streamed in realtime from clients. It is designed to be scalable and reliable
  • Sphinx: tool that makes it easy to create intelligent and beautiful
  • pHash.org: perceptual hash library
  • dahlia/wand · GitHub: The ctypes-based simple ImageMagick binding for Python.
  • py-bcrypt: strong password hashing for Python.
  • mdadm: manage MD devices aka Linux Software RAID.
  • twitter/snowflake · GitHub: network service for generating unique ID numbers at high scale with some simple guarantees
  • node-readability: Server side readability with node.js
  • Netflix/asgard: Web interface for application deployments and cloud management in Amazon Web Services (AWS)
  • Netflix/Priam: Co-Process for backup/recovery, Token Management, and Centralized Configuration management for Cassandra.
  • Netflix/zuul: edge service that provides dynamic routing, monitoring, resiliency, security, and more.
  • Cacti®: The Complete RRDTool-based Graphing Solution.
  • LFTP: sophisticated file transfer program.
  • lqez/npk: neat package system.
  • etsy/statsd: Simple daemon for easy stats aggregation.
  • new-relic: Application Performance Management & Monitoring.
  • Upstart - Wikipedia, the free encyclopedia
  • MapIt: map postcodes and geographical points to administrative areas
  • logstash: open source log management.
  • Ganglia: scalable distributed monitoring system for high-performance computing systems such as clusters and Grid.
  • Jenkins: extendable open source continuous integration server.
  • Graphite: Scalable Realtime Graphing.
  • Google-Analytics: Web Analytics & Reporting.
  • dyn: Managed DNS | Email Delivery | SMTP | Domain Registration
  • FontForge: outline font editor.
  • DTL FontTools: Dutch Type Library.
  • Zendesk.com: Customer Service Software.
@endalamaw1
Copy link

You are managing a web
app, built on django framework. Nginx is used as a reverse proxy and hosted on Ubuntu
14.04 server. It uses celery as a task processor, for sending email and downloading
files. The app uses redis as a caching server and PostgreSQL as the main database.
For reporting purposes, the app fetches aggregate data from PostgreSQL and stores in
Elasticsearch. Kafka is used for real-time messaging.
a. You are getting an INTERNAL SERVER ERROR, while accessing the main page
of the app. (a) What might be the possible source of the issue? (b) What
procedure will you follow to fix the issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment