chanks

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

sloria

The Best of the Best Practices (BOBP) Guide for Python

A "Best of the Best Practices" (BOBP) guide to developing in Python.

In General

Values

• "Build tools for others that you want to be built for you." - Kenneth Reitz
• "Simplicity is alway better than functionality." - Pieter Hintjens

bamanzi

# although StackOverflow has this answer http://stackoverflow.com/a/1128583
# but that code is unreadable, thus I have this (also based on a SO anower: http://stackoverflow.com/a/1128721 )

# get the name of the branch we are on
_git_repo_name() { 
    gittopdir=$(git rev-parse --git-dir 2> /dev/null)
    if [[ "foo$gittopdir" == "foo.git" ]]; then
        echo `basename $(pwd)`
    elif [[ "foo$gittopdir" != "foo" ]]; then
        echo `dirname $gittopdir | xargs basename`

miguelgrinberg

#!flask/bin/python
from flask import Flask, jsonify, abort, request, make_response, url_for
from flask_httpauth import HTTPBasicAuth

app = Flask(__name__, static_url_path = "")
auth = HTTPBasicAuth()

@auth.get_password
def get_password(username):
    if username == 'miguel':

yanofsky

This is free and unencumbered software released into the public domain.

Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.

In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit

doobeh

<form method="post">
    {{ form.hidden_tag() }}
    {{ form.example }}
    <input type="submit">
</form>

sent-hil

One of my favorite past times is to look at the notebooks of famous scientists. Da Vinci's notebook is well known, but there plenty others. Worshipping Da Vinci like no other, I bought a Think/Create/Record journal, used it mostly to keep jot down random thoughts and take notes. This was great in the beginning, but the conformity of lines drove me nuts. Only moleskines made blank notebooks, so I had to buy one.

At the same time I started a freelance project. The project itself is irrelevant, but suffice to say it was very complex and spanned several months. It seemed like a perfect opportunity to use the moleskine. Looking back, all my entries fell under few categories:

• Todo
• Question
• Thought
• Bug
• Feature