Skip to content

Instantly share code, notes, and snippets.

@freedomtowin

freedomtowin/intro-zero_hugging-face-course-tokenizers.txt

Created November 17, 2024 23:16
Show Gist options
  • Save freedomtowin/2e71dcd6a7e1f232a34b431e8576bcaf to your computer and use it in GitHub Desktop.
Save freedomtowin/2e71dcd6a7e1f232a34b431e8576bcaf to your computer and use it in GitHub Desktop.
Download ZIP
Example of creating a Gist using Python
Raw
intro-zero_hugging-face-course-tokenizers.txt
{'input_ids': tensor([[ 101, 1045, 2293, 2023, 3185, 999, 102],
[ 101, 2009, 2001, 3100, 1012, 102, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 0]])}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment