Skip to content

Instantly share code, notes, and snippets.

@simonespa

simonespa/tokenizer.py

Created September 29, 2023 10:55
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save simonespa/398d9d61de4900fd49865f90ee376c1f to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save simonespa/398d9d61de4900fd49865f90ee376c1f to your computer and use it in GitHub Desktop.
Download ZIP
Tokenizer
Raw
tokenizer.py
import re
vocabulary = set(re.split('\W', text))
vocabulary.update(set(re.findall('\W', text)))
embeddings = dict.fromkeys(vocabulary, [])
print(f'Vocabulary size: {len(vocabulary)}')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment