Skip to content

Instantly share code, notes, and snippets.

View KENTHD8mg's full-sized avatar

KENT KENTHD8mg

2 followers · 10 following
  • China HangZhou
View GitHub Profile
@xenova
xenova / tiktoken-to-hf.ipynb
Last active February 8, 2026 23:15
Convert tiktoken tokenizers to the Hugging Face tokenizers format
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.