Extract words from all your tweets and count their occurences

stopwords-en

able
about
above
abst
accordance
according
accordingly
across
act
actually
added
adj
affected
affecting
affects
after
afterwards
again
against
ah
all
almost
alone
along
already
also
although
always
am
among
amongst
an
and
announce
another
any
anybody
anyhow
anymore
anyone
anything
anyway
anyways
anywhere
apparently
approximately
are
aren
arent
arise
around
as
aside
ask
asking
at
auth
available
away
awfully
b
back
be
became
because
become
becomes
becoming
been
before
beforehand
begin
beginning
beginnings
begins
behind
being
believe
below
beside
besides
between
beyond
biol
both
brief
briefly
but
by
c
ca
came
can
cannot
can't
cause
causes
certain
certainly
co
com
come
comes
contain
containing
contains
could
couldnt
d
date
did
didn't
different
do
does
doesn't
doing
done
don't
down
downwards
due
during
e
each
ed
edu
effect
eg
eight
eighty
either
else
elsewhere
end
ending
enough
especially
et
et-al
etc
even
ever
every
everybody
everyone
everything
everywhere
ex
except
f
far
few
ff
fifth
first
five
fix
followed
following
follows
for
former
formerly
forth
found
four
from
further
furthermore
g
gave
get
gets
getting
give
given
gives
giving
go
goes
gone
got
gotten
h
had
happens
hardly
has
hasn't
have
haven't
having
he
hed
hence
her
here
hereafter
hereby
herein
heres
hereupon
hers
herself
hes
hi
hid
him
himself
his
hither
home
how
howbeit
however
hundred
i
id
ie
if
i'll
im
immediate
immediately
importance
important
in
inc
indeed
index
information
instead
into
invention
inward
is
isn't
it
itd
it'll
its
itself
i've
j
just
k
keep
keeps
kept
kg
km
know
known
knows
l
largely
last
lately
later
latter
latterly
least
less
lest
let
lets
like
liked
likely
line
little
'll
look
looking
looks
ltd
m
made
mainly
make
makes
many
may
maybe
me
mean
means
meantime
meanwhile
merely
mg
might
million
miss
ml
more
moreover
most
mostly
mr
mrs
much
mug
must
my
myself
n
na
name
namely
nay
nd
near
nearly
necessarily
necessary
need
needs
neither
never
nevertheless
new
next
nine
ninety
no
nobody
non
none
nonetheless
noone
nor
normally
nos
not
noted
nothing
now
nowhere
o
obtain
obtained
obviously
of
off
often
oh
ok
okay
old
omitted
on
once
one
ones
only
onto
or
ord
other
others
otherwise
ought
our
ours
ourselves
out
outside
over
overall
owing
own
p
page
pages
part
particular
particularly
past
per
perhaps
placed
please
plus
poorly
possible
possibly
potentially
pp
predominantly
present
previously
primarily
probably
promptly
proud
provides
put
q
que
quickly
quite
qv
r
ran
rather
rd
re
readily
really
recent
recently
ref
refs
regarding
regardless
regards
related
relatively
research
respectively
resulted
resulting
results
right
run
s
said
same
saw
say
saying
says
sec
section
see
seeing
seem
seemed
seeming
seems
seen
self
selves
sent
seven
several
shall
she
shed
she'll
shes
should
shouldn't
show
showed
shown
showns
shows
significant
significantly
similar
similarly
since
six
slightly
so
some
somebody
somehow
someone
somethan
something
sometime
sometimes
somewhat
somewhere
soon
sorry
specifically
specified
specify
specifying
still
stop
strongly
sub
substantially
successfully
such
sufficiently
suggest
sup
sure
t
take
taken
taking
tell
tends
th
than
thank
thanks
thanx
that
that'll
thats
that've
the
their
theirs
them
themselves
then
thence
there
thereafter
thereby
thered
therefore
therein
there'll
thereof
therere
theres
thereto
thereupon
there've
these
they
theyd
they'll
theyre
they've
think
this
those
thou
though
thoughh
thousand
throug
through
throughout
thru
thus
til
tip
to
together
too
took
toward
towards
tried
tries
truly
try
trying
ts
twice
two
u
un
under
unfortunately
unless
unlike
unlikely
until
unto
up
upon
ups
us
use
used
useful
usefully
usefulness
uses
using
usually
v
value
various
've
very
via
viz
vol
vols
vs
w
want
wants
was
wasn't
way
we
wed
welcome
we'll
went
were
weren't
we've
what
whatever
what'll
whats
when
whence
whenever
where
whereafter
whereas
whereby
wherein
wheres
whereupon
wherever
whether
which
while
whim
whither
who
whod
whoever
whole
who'll
whom
whomever
whos
whose
why
widely
will
willing
wish
with
within
without
won't
words
world
would
wouldn't
www
x
y
yes
yet
you
youd
you'll
your
youre
yours
yourself
yourselves
you've
z
zero

stopwords-sk

a
aby
aj
ak
ako
ale
alebo
and
ani
áno
ano
asi
až
az
bez
bude
budem
budeš
budes
budeme
budete
budú
budu
by
bol
bola
boli
bolo
byť
byt
cez
čo
co
či
ci
ďalší
ďalšia
ďalšie
dalsi
dalsia
dalsie
dnes
do
ho
ešte
este
for
i
inak
ja
je
jeho
jej
ich
iba
iné
ine
iny
iný
som
si
sme
sú
su
k
kam
každý
každá
každé
každí
kazdy
kazda
kazde
kazdi
kde
keby
keď
ked
kto
ktorá
ktoré
ktorou
ktorý
ktorí
ktora
ktore
ktorou
ktory
ktori
ku
lebo
len
mať
mat
má
ma
mal
mám
mam
máš
mas
máme
mame
máte
mate
majú
maju
medzi
mi
mna
mne
mnou
musieť
musiet
môcť
moct
môj
moj
môže
moze
my
na
nad
nám
nam
náš
naši
nas
nasi
nie
nech
než
nez
nič
nic
niektorý
niektory
nové
nový
nová
nové
noví
nove
novy
nova
nove
novi
o
od
odo
of
on
ona
ono
oni
ony
po
pod
podľa
podla
pokiaľ
pokial
potom
práve
prave
pre
prečo
preco
preto
pretože
pretoze
prvý
prvá
prvé
prví
prvy
prva
prve
prvi
pred
predo
pri
pýta
pyta
s
sa
so
si
ste
sú
su
svoje
svoj
svojich
svojím
svojími
svojim
svojimi
ta
tak
takže
takze
tam
táto
tato
teda
te
tě
ten
tento
tie
tieto
tým
tym
týmto
tymto
tiež
tiez
to
toto
toho
tohoto
tom
tomto
tomuto
toto
tu
tú
tu
túto
tuto
tvoj
ty
tvojími
tvojimi
už
uz
v
vám
vam
váš
vaše
vas
vase
vo
viac
však
vsak
všetok
vsetok
vy
z
za
zo
že
ze

tweets.sh

#!/usr/bin/env bash

#   extract words from all your tweets (except retweets) and count their occurences
#
#   author: mnicky [http://mnicky.github.com]
#
#   You need:
#       - classic unix utilities
#       - 't' twitter client: http://sferik.github.com/t
#       - text files with stopwords (with unix line endings) - you can grab some other at http://code.google.com/p/stop-words
#

# some settings
USER="billgates"   #can be also passed as the first parameter
EN_STOPWORDS_FILE="stopwords-en"
SK_STOPWORDS_FILE="stopwords-sk"

# check for the presence of 't' twitter client
t version >/dev/null 2>&1 || { echo >&2 "Please install the 't' twitter client from http://sferik.github.com/t"; exit 1; }

# the twitter user can be also given as the first parameter to this script
if [ "$1" != "" ]; then USER=$1; fi

echo "downloading the tweets of: @$USER..."
t timeline $USER -n 3200 | grep -vi "@$USER" | grep -v '^$' > $USER-tweets

echo "extracting words..."
cat $USER-tweets | tr ' ' '\n' | grep -v '^$' | grep -v '@.\+' | grep -v 'http'  | tr 'A-Z' 'a-z' | grep -wvf $EN_STOPWORDS_FILE | tr '[:punct:]' ' ' | tr ' ' '\n' | grep -v '^$' | grep -wvf $EN_STOPWORDS_FILE | grep -vwf $SK_STOPWORDS_FILE | grep -v '^.\{1,2\}$' | grep -v '^[0-9]\+$' > $USER-words

echo "counting..."
cat $USER-words | sort | uniq -c | sort -nr > $USER-counts

echo "done."

Usage: ./tweets.sh billgates