astockwell

import re
import string
import time

pattern_words_only = re.compile('[^A-Za-z0-9 ]+', re.UNICODE)

dictionary_words = [line.rstrip('\n') for line in open('/usr/share/dict/words')]
# print(len(dictionary_words))

two_cities = 'It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way - in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. I

astockwell

Golang Worker (Toy) Example

Demonstrates an example of a work queue fanning-out work items to multiple concurrent workers (goroutines).

The work queue source is lines from a log file in this example, but can be anything. The number of workers is variable and can be adjusted from 1 to 100,000's. As of Go 1.5, the runtime uses all visible CPUs (whatever your operating system considers to be a CPU) to schedule concurrent goroutine execution.

In this example, "work" done on each log file line is simulated by the worker sleeping for a randomized # of seconds from 0-5. When running, you can observe that the workers pull "work" items in an unpredictable order (e.g. you never know which worker will get the next item) and they operate in parallel.

This example also includes 2 variables (dateDict and ipsToInvestigate) that multiple goroutines share and write to (potentially simultaneously), and thus must be protected by a mutex/semaphore (mu).