Igor Brigadir igorbrigadir

#Non-mathematical Introductions

#Videos

General Background and Overview

Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
Models and Issues in Data Stream Systems
Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
[Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep=rep1&t

Concurrent Package overview

The package is a ton of utilities for developing concurrent programs in Java: concurrent maps, synchronization strategies, blocking queues, thread management, and many more. The latter is the one we are concerned about: thread management. Thread management in this package all starts with the Executors helper class and the ExecutorService interface. The Executors helper class provides easy methods to create an ExecutorService as either a cached thread pool that grows on demand as new threads are needed or a fixed thread pool that ensures tasks are queued once the threads are exhausted. As a developer all you need to do is simply submit new tasks and the task will be executed in the background once a thread is available. The result is a Future . With the future in hand, you can poll for the completion of the event or simply wait for it. With only a few lines of code, we have created a re-usable and thread managed system for querying APIs.

import java.util.con

Tuning Storm+Trident

Tuning a dataflow system is easy:

The First Rule of Dataflow Tuning:
* Ensure each stage is always ready to accept records, and
* Deliver each processed record promptly to its destination

	def consumer(func):
	'''
	Decorator taking care of initial next() call to "sending" generators

	From PEP-342
	http://www.python.org/dev/peps/pep-0342/
	'''
	def wrapper(args,*kw):
	gen = func(args, *kw)
	next(gen)

	#!/usr/bin/env python

	import os
	import xml.parsers.expat
	from xml.sax.saxutils import escape
	from optparse import OptionParser
	from math import log10


	# How much data we process at a time

	import marisa_trie
	from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer


	# hack to store vocabulary in MARISA Trie
	class _MarisaVocabularyMixin(object):

	def fit_transform(self, raw_documents, y=None):
	super(_MarisaVocabularyMixin, self).fit_transform(raw_documents)
	self._freeze_vocabulary()

	# (C) Mathieu Blondel, November 2013
	# License: BSD 3 clause

	import numpy as np
	from scipy.linalg import svd


	def frequent_directions(A, ell, verbose=False):
	"""
	Return the sketch of matrix A.

	The Eternal Flame (God Wrote in Lisp)
	Bob Kanefsky / Julia Ecklar


	F G C
	I was taught assembler in my second year of school.
	F G C
	It's kinda like construction work, with a toothpick for a tool.
	F G C Em Am
	So when I made my senior year, I threw my code away,

	# this forces dpkg not to call sync() after package extraction and speeds up install
	RUN echo "force-unsafe-io" > /etc/dpkg/dpkg.cfg.d/02apt-speedup
	# we don't need and apt cache in a container
	RUN echo "Acquire::http {No-Cache=True;};" > /etc/apt/apt.conf.d/no-cache