Skip to content

Instantly share code, notes, and snippets.

View ntulip's full-sized avatar

Nick Tulip ntulip

View GitHub Profile

Scaling your API with rate limiters

The following are examples of the four types rate limiters discussed in the accompanying blog post. In the examples below I've used pseudocode-like Ruby, so if you're unfamiliar with Ruby you should be able to easily translate this approach to other languages. Complete examples in Ruby are also provided later in this gist.

In most cases you'll want all these examples to be classes, but I've used simple functions here to keep the code samples brief.

Request rate limiter

This uses a basic token bucket algorithm and relies on the fact that Redis scripts execute atomically. No other operations can run between fetching the count and writing the new count.

@ntulip
ntulip / message_queue_pipeline.py
Created June 29, 2016 19:10
Scrapy pipeline to enque scraped items to message queue using carrot
from scrapy.xlib.pydispatch import dispatcher
from scrapy import signals
from scrapy.exceptions import DropItem
from scrapy.utils.serialize import ScrapyJSONEncoder
from carrot.connection import BrokerConnection
from carrot.messaging import Publisher
from twisted.internet.threads import deferToThread
@ntulip
ntulip / certchain.sh
Created January 18, 2016 18:18 — forked from joshenders/certchain.sh
Simple Bash function to visualize PKI chain-of-trust
function certchain() {
# Usage: certchain
# Display PKI chain-of-trust for a given domain
# GistID: https://gist.github.com/joshenders/cda916797665de69ebcd
if [[ "$#" -ne 1 ]]; then
echo "Usage: ${FUNCNAME} <ip|domain[:port]>"
return 1
fi
local host_port="$1"
{
"title": "Apache logs",
"services": {
"query": {
"list": {
"0": {
"query": "*",
"alias": "",
"color": "#7EB26D",
"id": 0,
@ntulip
ntulip / Ubuntu12-04
Created June 4, 2012 15:36 — forked from devilankur18/Ubuntu12-04
Graylog Setup in ubuntu 12.04
Grab the deb packages for Elasticsearch from http://www.elasticsearch.org/download/ and both the Graylog2 server and web interface from http://graylog2.org/download.
#install the elastic search from debian
dpkg -i elasticsearch-0.19.3.deb
References
http://imcol.in/category/monitoring/
@ntulip
ntulip / Graphite Install Ubuntu 11.04 Natty
Created May 2, 2012 01:31
Graphite Install Ubuntu 11.04 Natty
####################################
# BASIC REQUIREMENTS
# http://graphite.wikidot.com/installation
# http://geek.michaelgrace.org/2011/09/how-to-install-graphite-on-ubuntu/
####################################
sudo apt-get update
wget http://launchpad.net/graphite/1.0/0.9.8/+download/graphite-web-0.9.8.tar.gz
wget http://launchpad.net/graphite/1.0/0.9.8/+download/carbon-0.9.8.tar.gz
wget http://launchpad.net/graphite/1.0/0.9.8/+download/whisper-0.9.8.tar.gz
www:
requirements:
- faye
- jade
@ntulip
ntulip / webapp.rb
Created November 16, 2010 16:42 — forked from igrigorik/webapp.rb
require 'rubygems'
require 'rack'
class Object
def webapp
class << self
define_method :call do |env|
func, *attrs = env['PATH_INFO'].split('/').reject(&:empty?)
[200, {}, send(func, *attrs)]
end
require 'fileutils'
start_time = Time.now
SOURCE_DB = {
:name => 'db_name',
:user => 'db_user',
:password => 'db_pass',
:host => 'localhost'
@ntulip
ntulip / test.cs
Created October 22, 2010 16:32 — forked from gorsuch/test.cs
using System;
using System.IO;
using System.Text;
using Amazon;
using Amazon.S3;
using Amazon.S3.Model;
namespace AWSUploadTest
{