wilmoore/readme.md Secret

Last active June 9, 2025 17:07

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Select an option

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/wilmoore/dd068233e29dc9449bbabdb37d15d59c.js"></script>
Save wilmoore/dd068233e29dc9449bbabdb37d15d59c to your computer and use it in GitHub Desktop.

Download ZIP

AI :: NLP :: Tokenization

Raw

AI :: NLP :: Tokenization

⪼ Made with 💜 by Polyglot.

..

`assistant`

AI Token Explanation (ChatGPT)

Tokenization

Splitting text into model-readable subword units...Used by all modern LLMs including GPT, BERT, Claude, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment