Skip to content

Instantly share code, notes, and snippets.

@piegu

piegu/Byte-Level-BPE_universal_tokenizer_but_en_vs_pt_tokenizer.ipynb

Last active July 3, 2020 07:47
Show Gist options
  • Save piegu/3bb3786834839e140c057c5697fa6132 to your computer and use it in GitHub Desktop.
Save piegu/3bb3786834839e140c057c5697fa6132 to your computer and use it in GitHub Desktop.
Download ZIP
English vs Portuguese tokenizer on Portuguese Wikipedia of Byte-Level-BPE_universal_tokenizer_but_en_tokenizer.ipynb
Display the source blob
Display the rendered blob
Raw
Byte-Level-BPE_universal_tokenizer_but_en_vs_pt_tokenizer.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment