Skip to content

Instantly share code, notes, and snippets.

@neilkod
Created October 27, 2011 15:56
Show Gist options
  • Save neilkod/1319966 to your computer and use it in GitHub Desktop.
Save neilkod/1319966 to your computer and use it in GitHub Desktop.
my stopwords code, optimized for social media
given a text file containing stopwords, return a python list of its contents.
this list is optimized for twitter/social media and filters out stuff
like RT, nowplaying, lastfm, 4sq etc.
def get_stopwords(file='stopwords.txt'):
words = open(file,'r')
stopwords = [word.strip() for word in words]
return set(stopwords)
nkodner@hadoop4 mongodb$ cat stopwords.txt
youtube
nowplaying
lastfm
rt
4sq
RT
The
Your
a
about
above
across
after
afterwards
again
against
all
almost
alone
along
already
also
although
always
am
among
amongst
amoungst
amount
an
and
another
any
anyhow
anyone
anything
anyway
anywhere
are
around
as
at
back
be
became
because
become
becomes
becoming
been
before
beforehand
behind
being
below
beside
besides
between
beyond
bill
bit
both
bottom
but
by
call
can
cannot
cant
co
com
con
could
couldnt
cry
de
describe
detail
do
doesn
done
down
due
during
each
eg
eight
either
eleven
else
elsewhere
empty
enough
etc
even
ever
every
everyone
everything
everywhere
except
few
fifteen
fify
fill
find
fire
first
five
for
former
formerly
forty
found
four
from
front
full
further
get
give
go
had
has
hasnt
have
he
hence
her
here
hereafter
hereby
herein
hereupon
hers
herself
him
himself
his
how
however
http
hundred
ie
if
in
inc
indeed
interest
into
is
it
its
itself
just
keep
last
latter
latterly
least
less
ltd
ly
made
many
may
me
meanwhile
might
mill
mine
more
moreover
most
mostly
move
much
must
my
myself
name
namely
neither
never
nevertheless
next
nine
no
nobody
none
noone
nor
not
nothing
now
nowhere
of
off
often
on
once
one
only
onto
or
other
others
otherwise
our
ours
ourselves
out
over
own
part
per
perhaps
please
put
rather
re
same
see
seem
seemed
seeming
seems
serious
several
she
should
show
side
since
sincere
six
sixty
so
some
somehow
someone
something
sometime
sometimes
somewhere
still
such
system
take
ten
than
that
the
their
them
themselves
then
thence
there
thereafter
thereby
therefore
therein
thereupon
these
they
thickv
thin
third
this
those
though
three
through
throughout
thru
thus
to
together
too
top
toward
towards
twelve
twenty
two
un
under
until
up
upon
us
very
via
was
we
well
were
what
whatever
when
whence
whenever
where
whereafter
whereas
whereby
wherein
whereupon
wherever
whether
which
while
whither
who
whoever
whole
whom
whose
why
will
with
within
without
would
yet
you
your
yours
yourself
yourselves
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment