emaadmanzoor

Expand The Edinburgh Twitter FSD Corpus

The Python scripts attached here take care of the following tedious work, and should help one quickly get started with some real work on the corpus:

• Respect the Twitter API rate limits and throttle API hits.
• Don't hit the API for already expanded tweet ID's, so you can resume tweet expansion after stopping midway.
• Parse the API response and dump it into the correct column in the sqlite3 database.
• Gracefully handle exceptions while acquiring tweets from the API.
• Wrap version 1.1 of the Twitter API.
• Start from a specified tweet ID, assuming the input file is sorted in increasing order of tweet ID.

emaadmanzoor

import random
import operator

t = int(raw_input())
randint = random.randint

def deterministic(a,b,c,n):
    no = 0
    for p in xrange(n):
        for q in xrange(n):

emaadmanzoor

See the project website for more details.

Please report any issues to [email protected].

Correlation Results

The attention potential (as estimated in section 4), when evaluated on this Twitter dataset:

• Is 73.61% correlated with the retweets obtained.
• Is significantly correlated (p < 0.05).

emaadmanzoor

See the project website for more details.

Please report any issues to [email protected].

Execution

Running this requires having the following files in the same directory as calculate_cluster_statistics.py:

• all_links.p
• all_tweets.p

emaadmanzoor

StreamSpot Bootstrap Clusters

www3.cs.stonybrook.edu/~emanzoor/streamspot/

Below are the bootstrap clusters used for the experiments in the StreamSpot paper for each of following datasets:

• all (01-C50_k10_all.txt): Chunk length of 50, 10 clusters.
• ydc (02-C25_k5_ydc.txt): Chunk length of 25, 5 clusters.
• gfc (03-C50_k5_gfc.txt): Chunk length of 50, 5 clusters.

emaadmanzoor

Contents

1. README
   1. StreamSpot format
   2. CDM format
      1. CDM events
      2. CDM subjects, objects and principals
      3. CDM simple edges
   3. Combining CDM records into a StreamSpot edge
2. [Assigning a graph ID to each StreamSpot edge](https://gist.github.com/emaadmanzoor/b23d46c9d4d064746e1ef667cb8070bc#assi

emaadmanzoor

#!/usr/bin/env python

# Copyright 2016 Emaad Ahmed Manzoor
# License: Apache License, Version 2.0
# http://www.eyeshalfclosed.com/blog/2016/07/22/spark-streaming-statistics/

"""
Get Spark Streaming microbatch statistics:
    - Batch start time
    - Scheduling delay (in seconds) for each microbatch

emaadmanzoor

95865 Recitation - Model Evaluation Demo

• Lecture slides: https://speakerdeck.com/emaadmanzoor/introduction-to-classification
• Course website: http://www.andrew.cmu.edu/user/georgech/95-865/
• Other recitations: http://www.eyeshalfclosed.com/teaching/

Contact:

• Emaad Ahmed Manzoor (emaad[at]cmu.edu)
• George Chen (georgechen[at]cmu.edu)

emaadmanzoor

emaadmanzoor

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=gb760
ExecStart=/bin/sh -c '/home/gb760/kafka/bin/kafka-server-start.sh /home/gb760/kafka/config/server.properties > /home/gb760/kafka/kafka.log 2>&1'
ExecStop=/home/gb760/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal