Jaimy Emmanuel jemmanuel

Easy parallel python with concurrent.futures

As of version 3.3, python includes the very promising concurrent.futures module, with elegant context managers for running tasks concurrently. Thanks to the simple and consistent interface you can use both threads and processes with minimal effort.

For most CPU bound tasks - anything that is heavy number crunching - you want your program to use all the CPUs in your PC. The simplest way to get a CPU bound task to run in parallel is to use the ProcessPoolExecutor, which will create enough sub-processes to keep all your CPUs busy.

We use the context manager thusly:

with concurrent.futures.ProcessPoolExecutor() as executor:

"Improving numerical routines in Scala Breeze" GSoC 2014 proposal.

Abstract

Breeze is a great numerical processing library. However, it lacks some high-level functions that you can find in other libraries like SciPy. The second issue is that Breeze lacks documentation. This makes the entry barrier higher for new contributors.

My proposal is to revamp documentation and to introduce interpolation and

For most of the tools on here, if you follow the GitHub repos back to the user that owns the repo, they usually have other cool Clojure tools they build. Just a general observation about the Clojure community.

Learnings

Clojure home - There's a ton of good stuff here, like the rationale behind the design and explanations of the best features of Clojure.
Clojure.github.io - The core Clojure documentation. Useful for looking up functions and such.
Clojuredoc - Community written Clojure documentation and articles
Clojure Koans - Learn Clojure with Koans
- When you get stuck, use these YouTube videos to help you.
4Clojure - Solve simple problems to stretch your Clojure skills

General Background and Overview

Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
Models and Issues in Data Stream Systems
Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
[Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep=rep1&t

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

Backend Architectures

Twitter

ror, scala, jetty, erlang, thrift, mongrel, comet server, my-sql, memchached, varnish, kestrel(mq), starling, gizzard, cassandra, hadoop, vertica, munin, nagios, awstats

Introduce Gizzard of Twitter - 트위터의 새로운 분산 관리 라이브러리 Gizzard를 소개합니다.
The Architecture Twitter Uses to Deal with 150M Active Users, 300K QPS, a 22 MB/S Firehose, and Send Tweets in Under 5 Seconds
Scaling Twitter: Making Twitter 10000 Percent Faster

Each of these commands will run an ad hoc http static server in your current (or specified) directory, available at http://localhost:8000. Use this power wisely.

Discussion on reddit.

Python 2.x

$ python -m SimpleHTTPServer 8000

	// Just before switching jobs:
	// Add one of these.
	// Preferably into the same commit where you do a large merge.
	//
	// This started as a tweet with a joke of "C++ pro-tip: #define private public",
	// and then it quickly escalated into more and more evil suggestions.
	// I've tried to capture interesting suggestions here.
	//
	// Contributors: @r2d2rigo, @joeldevahl, @msinilo, @_Humus_,
	// @YuriyODonnell, @rygorous, @cmuratori, @mike_acton, @grumpygiant,

	import java.util.ArrayList;
	import java.util.Iterator;
	import java.util.List;
	import java.util.concurrent.Callable;
	import java.util.concurrent.ExecutorService;
	import java.util.concurrent.Future;
	import java.util.concurrent.LinkedBlockingQueue;
	import java.util.concurrent.ThreadPoolExecutor;
	import java.util.concurrent.TimeUnit;

	import java.util.concurrent.Callable;
	import java.util.concurrent.ExecutorService;
	import java.util.concurrent.Future;
	import java.util.concurrent.LinkedBlockingQueue;
	import java.util.concurrent.ThreadPoolExecutor;
	import java.util.concurrent.TimeUnit;

	public class FuturesA {

	public static void run() throws Exception {