soaxelbrooke

Text analytics engine!

Hey guys! I'm @soaxelbrooke, and I am here to show you ladies and guys how to create a basic text analytics engine with Elasticsearch.

Getting the data

Let's get the data first! These are product reviews from Amazon, which can be found here.

$ curl http://times.cs.uiuc.edu/~wang296/Data/LARA/Amazon/AmazonReviews.zip -o reviews.zip

soaxelbrooke

from typing import Optional, Hashable, TypeVar

class CallableDict(dict):
    V = TypeVar('V')
    
    """ A callable dictionary useful for functional programming """
    def __call__(self, key: Hashable, default: Optional[V]=None) -> Optional[V]:
        return self.get(key, default)

soaxelbrooke

#!/bin/bash

DYNAMODB_USER=stuart

cd /home/${DYNAMODB_USER}/bin
mkdir -p dynamodb
cd dynamodb

wget http://dynamodb-local.s3-website-us-west-2.amazonaws.com/dynamodb_local_latest.tar.gz
tar -xvzf dynamodb_local_latest.tar.gz

soaxelbrooke

// A prototype-based test wrapper based on generators.
// See original Haskell quickcheck paper: http://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf

// --------------------------- //
// Scroll to bottom for usage! //
// --------------------------- //

import R from 'ramda';

const RUNS_PER_TEST = 50;

soaxelbrooke

logging.basicConfig(format='%(levelname)s:%(asctime)s.%(msecs)03d [%(threadName)s] - %(message)s',
                    datefmt='%Y-%m-%d,%H:%M:%S',
                    level=getattr(logging, os.getenv('LOG_LEVEL', 'INFO')))

soaxelbrooke

WITH tokens AS (
  -- Just edit MY_TABLE, MY_TEXT_COL, and MY_PKEY_COL, and watch it go!
  SELECT MY_PKEY_COL AS pkey, (unnest(to_tsvector(MY_TEXT_COL))).* FROM MY_TABLE
), token_stream AS (
  SELECT pkey, unnest(positions) AS token_idx, lexeme
  FROM tokens ORDER BY pkey, token_idx
), token_counts AS (
  SELECT lexeme, sum(count) AS count
  FROM (
    SELECT lexeme, array_length(positions, 1) AS count FROM tokens

soaxelbrooke

#!/usr/bin/env python3


import sqlite3
import csv
import sys

quantize = '--quantize' in sys.argv

soaxelbrooke

Postgrest doesn't like you executing arbitrary queries, but you can get around it by defining a function that executes the query for you:

$ psql mydb
mydb=# create function custom_query(query text) returns setof json as $f$
    begin 
    return query execute format('with tmp as (%s) select row_to_json(tmp.*) from tmp;', query); 
    end
 $f$ language plpgsql;

soaxelbrooke

""" Class for tracking experiments in a local sqlite database """

class SqliteExperiment:
    def __init__(self, hparams, metrics, experiment_id=None):
        self.experiment_id = experiment_id or str(uuid4())
        self.hparams = hparams
        self.metrics = metrics
        self.metric_names = ['experiment_id', 'measured_at'] + [n for n, t in metrics]
        self.log_every = int(os.environ.get('LOG_EVERY', 10000))
        self.last_log = None

soaxelbrooke

function postTagData(myUrl, documentId, fullText, tag, start, end) {
    let form = new FormData();
    form.set('document_id', documentId);
    form.set('full_text', fullText);
    form.set('region_start', start);
    form.set('region_end', end);
    return window.fetch(myUrl, {
        method: 'POST',
        body: form
    });