Skip to content

Instantly share code, notes, and snippets.

@hanneshapke

hanneshapke/call_berttokenizer.py

Created March 9, 2020 17:45
Show Gist options
  • Save hanneshapke/f555d5ea0a9d99b8ea0f69f73f0e33ed to your computer and use it in GitHub Desktop.
Save hanneshapke/f555d5ea0a9d99b8ea0f69f73f0e33ed to your computer and use it in GitHub Desktop.
Download ZIP
Raw
call_berttokenizer.py
tokens = bert_tokenizer.tokenize(text)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment