Luís Barbosa bymyslf

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

System Design Cheatsheet

Picking the right architecture = Picking the right battles + Managing trade-offs

Basic Steps

Clarify and agree on the scope of the system

User cases (description of sequences of events that, taken together, lead to a system doing something useful)
- Who is going to use it?
- How are they going to use it?

Scaling your API with rate limiters

The following are examples of the four types rate limiters discussed in the accompanying blog post. In the examples below I've used pseudocode-like Ruby, so if you're unfamiliar with Ruby you should be able to easily translate this approach to other languages. Complete examples in Ruby are also provided later in this gist.

In most cases you'll want all these examples to be classes, but I've used simple functions here to keep the code samples brief.

Request rate limiter

This uses a basic token bucket algorithm and relies on the fact that Redis scripts execute atomically. No other operations can run between fetching the count and writing the new count.

First of all, please note that token expiration and revoking are two different things.

Expiration only happens for web apps, not for native mobile apps, because native apps never expire.
Revoking only happens when (1) uses click the logout button on the website or native Apps;(2) users reset their passwords; (3) users revoke their tokens explicitly in the administration panel.

1. How to hadle JWT expiration

A JWT token that never expires is dangerous if the token is stolen then someone can always access the user's data.

Quoted from JWT RFC:

Falsehoods programmers believe about time

This is a compiled list of falsehoods programmers tend to believe about working with time.

Don't re-invent a date time library yourself. If you think you understand everything about time, you're probably doing it wrong.

Falsehoods

There are always 24 hours in a day.
February is always 28 days long.
Any 24-hour period will always begin and end in the same day (or week, or month).

Advanced Distributed System Design Course - Udi Dahan

Notes by Craig Phillips

Fallacies of Distributed Computing

There are 11 fallacies of Distributed Computing:
1. The network is reliable
2. Latency isn’t a problem
3. Bandwidth isn’t a problem
4. The network is secure

The topology won’t change

Pretende usar `UUID` como PK em vez de `Int/BigInt` no seu banco de dados? Pense novamente...

TL;TD

Não use UUID como PK nas tabelas do seu banco de dados.

	namespace BloomFilter
	{
	using System;
	using System.Collections;

	/// <summary>
	/// Bloom filter.
	/// </summary>
	/// <typeparam name="T">Item type </typeparam>
	public class Filter<T>

	using System;
	using System.Collections.Concurrent;
	using System.Collections.Generic;
	using System.ComponentModel;
	using System.Timers;

	namespace DebounceUtils
	{
	/// <summary>
	/// Event debouncer helps to prevent calling the same event handler too often (like mark Dirty or Invalidate)


	type Set struct {
	list map[int]struct{} //empty structs occupy 0 memory
	}

	func (s *Set) Has(v int) bool {
	_, ok := s.list[v]
	return ok
	}