jorendorff

%% loop.erl - Test speed of message delivery across processes.
%%
%% Run this with:
%%     timer:tc(loop, loop_test, [100, 10000]).

-module(loop).

-export([loop_test/2]).

jorendorff

$ git log --all --graph --decorate
* commit ae97f6d12a8ce341f91f204a6ee96719249ae46f (HEAD -> rayon, origin/rayon)
| Author: Jason Orendorff <[email protected]>
| Date:   Mon Aug 8 17:06:35 2016 -0500
| 
|     Use Rayon to render each row of pixels as a separate task, processed by Rayon's built-in thread pool.
|    
| * commit 5a5c074a6809a90edb6fbf65759aae23ea5d3878 (origin/lockfree, lockfree)
|/  Author: Jim Blandy <[email protected]>
|   Date:   Tue Oct 27 10:45:05 2015 +0100

jorendorff

#![feature(test)]

extern crate test;

mod offthread {
    use std::sync::mpsc;
    use std::thread;

    pub trait OffThreadIterator: Iterator + Send + Sized + 'static {
        fn off_thread(self) -> mpsc::IntoIter<Self::Item>

jorendorff

# To run this code, you'll need the sample data (133 MB download, unzips to 492 MB):
#     http://bit.ly/2avfASU
#     tar xjf sample.tar.bz2

defmodule Elindex.Searcher do
  def search(word) do
    File.ls!("sample")
    |> Stream.map(fn(filename) ->
         fn ->
            Path.join("sample", filename)

jorendorff

#!/bin/env python3

import os
DIR = "./sample"

for filename in os.listdir(DIR):
    with open(os.path.join(DIR, filename)) as f:
        tokens = f.read().lower().split()

        ??? store the tokens in an index... somehow ???

jorendorff

$ python3 search.py
> ice cream
Saturday Night Live
San Miguel Beermen
Android (operating system)
Durham, North Carolina
Pharrell Williams
New England
Guanajuato
Lafayette, Louisiana

jorendorff

 555038 United States
 195472 Animal
 186797 France
 166573 Germany
 166180 Canada
 165461 India
 157166 England
 154855 World War II
 146055 United States Census Bureau
 145477 United Kingdom

jorendorff

History of the Center of the Universe
John Burnet (classicist)
w:Charles Lyell
Principles of Geology
Palus Maeotis
wikiquote:Charles Lyell
Empedocles
Earth (classical element)
Water (classical element)
Air (classical element)

jorendorff

This is a sketch of how module loading might work in HTML.

1. A "load task" is the end-to-end job of loading a module and all its dependencies, linking everything together, and running the code. It's a high-level concept.

   Load tasks come in from <script type=module> elements, plus wherever else (maybe a DOM API). The loader can have multiple load tasks in flight at once.

2. The loader contains a collection of modules shared by all load tasks. Each module is in one of these states:

   • loading - waiting for the network;

• waiting for dependencies - loaded and parsed, not linked;

jorendorff

// "Term" means word.
// Note: The hits in each Vec<Hit> are sorted by document_id.
type TermIndex = HashMap<String, SortedVec<Hit>>;

struct Hit {
    /// Index into a single giant list of all the documents in the corpus
    document_id: u32,

    /// List of all the places within this document where the term appears.
    offsets: SortedVec<u32>