Skip to content

Instantly share code, notes, and snippets.

@gamingflexer

gamingflexer/main.py

Created July 19, 2023 09:02
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save gamingflexer/3364999976db4f8ba8df7829d7dfe384 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save gamingflexer/3364999976db4f8ba8df7829d7dfe384 to your computer and use it in GitHub Desktop.
Download ZIP
Anthropic's tokenizer for Claude
Raw
main.py
from transformers import PreTrainedTokenizerFast
fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="/home/ubuntu/LLM/module/claude-v1-tokenization.json")
text = "Hello, this is a test input."
tokens = fast_tokenizer.tokenize(text)
tokens
@danikhan632
Copy link
Copy Markdown

danikhan632 commented Nov 8, 2023

wget https://public-json-tokenization-0d8763e8-0d7e-441b-a1e2-1c73b8e79dc3.storage.googleapis.com/claude-v1-tokenization.json

this is for downloading the tokenizer

@ayansengupta17
Copy link
Copy Markdown

ayansengupta17 commented Mar 11, 2024

Could you provide the link for Claude 3?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment