Skip to content

Instantly share code, notes, and snippets.

@pranjalAI

pranjalAI/tokenize_dict.py

Created November 28, 2020 18:15
Show Gist options
  • Save pranjalAI/c67192e5aaf0687bb2a3cda321618cfd to your computer and use it in GitHub Desktop.
Save pranjalAI/c67192e5aaf0687bb2a3cda321618cfd to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenize_dict.py
gen_docs=[[w.lower() for w in word_tokenize(text)] for text in data_list]
dictionary=gensim.corpora.Dictionary(gen_docs)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment