Skip to content

Instantly share code, notes, and snippets.

@emmagrimaldi

emmagrimaldi/clean_tokens.py

Created September 1, 2018 00:23
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save emmagrimaldi/a2df5cb37859affb5b0b9dcf85685d8c to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save emmagrimaldi/a2df5cb37859affb5b0b9dcf85685d8c to your computer and use it in GitHub Desktop.
Download ZIP
clean_tokens
Raw
clean_tokens.py
# importing the library needed in this section
from nltk.corpus import stopwords
# assigning the english stop-words to the sw list
sw = stopwords.words('english')
# assigning the non stop-words contained in the tokens list
# to a new list named clean_tokens through a list comprehension
clean_tokens = [token for token in tokens if token not in sw]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment