Skip to content

Instantly share code, notes, and snippets.

@sai-teja-ponugoti
Created June 9, 2020 19:41
Show Gist options
  • Save sai-teja-ponugoti/e1d3e43e21561a5a7c10b939b06d2b2d to your computer and use it in GitHub Desktop.
Save sai-teja-ponugoti/e1d3e43e21561a5a7c10b939b06d2b2d to your computer and use it in GitHub Desktop.
# importing NLTK libarary stopwords
import nltk
from nltk.corpus import stopwords
nltk.download('stopwords')
nltk.download('punkt')
from nltk.tokenize import word_tokenize
print(stopwords.words('english'))
# random sentecnce with lot of stop words
sample_text = "Oh man, this is pretty cool. We will do more such things."
text_tokens = word_tokenize(sample_text)
tokens_without_sw = [word for word in text_tokens if not word in stopwords.words('english')]
print(text_tokens)
print(tokens_without_sw)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment