Skip to content

Instantly share code, notes, and snippets.

@mbednarski

mbednarski/tokenize.py

Last active February 26, 2018 20:17
Show Gist options
  • Save mbednarski/a6522978b59cda771e088eacf26a6c93 to your computer and use it in GitHub Desktop.
Save mbednarski/a6522978b59cda771e088eacf26a6c93 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenize.py
def tokenize_corpus(corpus):
tokens = [x.split() for x in corpus]
return tokens
tokenized_corpus = tokenize_corpus(corpus)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment