Skip to content

Instantly share code, notes, and snippets.

@MLWhiz

MLWhiz/stem_text.py

Created February 9, 2019 08:00
Show Gist options
  • Save MLWhiz/a961920ea4af8d8ea2ca682946cc3796 to your computer and use it in GitHub Desktop.
Save MLWhiz/a961920ea4af8d8ea2ca682946cc3796 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
stem_text.py
from nltk.stem import SnowballStemmer
from nltk.tokenize.toktok import ToktokTokenizer
def stem_text(text):
tokenizer = ToktokTokenizer()
stemmer = SnowballStemmer('english')
tokens = tokenizer.tokenize(text)
tokens = [token.strip() for token in tokens]
tokens = [stemmer.stem(token) for token in tokens]
return ' '.join(tokens)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment