Skip to content

Instantly share code, notes, and snippets.

@emmagrimaldi
emmagrimaldi / tokens.py
Created September 1, 2018 00:03
tokens
# importing the libraries needed in this section
import nltk
from nltk.tokenize import RegexpTokenizer
# defining the text
text = 'Rome was founded in 753BC by its first king, Romulus.'
# instantiating the tokenizer object. By passing r'\w+' to the RegexpTokenizer
# I am selecting groups of single words, discarding the punctuation
tokenizer = RegexpTokenizer(r'\w+')