Skip to content

Instantly share code, notes, and snippets.

@emmagrimaldi

emmagrimaldi/tokens.py

Created September 1, 2018 00:03
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save emmagrimaldi/7199a754ae6ac7366b66faa19f6ab494 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save emmagrimaldi/7199a754ae6ac7366b66faa19f6ab494 to your computer and use it in GitHub Desktop.
Download ZIP
tokens
Raw
tokens.py
# importing the libraries needed in this section
import nltk
from nltk.tokenize import RegexpTokenizer
# defining the text
text = 'Rome was founded in 753BC by its first king, Romulus.'
# instantiating the tokenizer object. By passing r'\w+' to the RegexpTokenizer
# I am selecting groups of single words, discarding the punctuation
tokenizer = RegexpTokenizer(r'\w+')
# getting the tokens
tokens = tokenizer.tokenize(text)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment