Skip to content

Instantly share code, notes, and snippets.

@freedomtowin

freedomtowin/intro-zero_hugging-face-course-tokenizers.txt

Created November 17, 2024 23:16
Show Gist options
  • Save freedomtowin/1b1167f1c2afedc1f82c3f9f06414352 to your computer and use it in GitHub Desktop.
Save freedomtowin/1b1167f1c2afedc1f82c3f9f06414352 to your computer and use it in GitHub Desktop.
Download ZIP
Example of creating a Gist using Python
Raw
intro-zero_hugging-face-course-tokenizers.txt
Tokens: ['i', "'", 've', 'been', 'waiting', 'for', 'a', 'hugging', '##face', 'course', 'my', 'whole', 'life', '.']
Input IDs: tensor([[ 1045, 1005, 2310, 2042, 3403, 2005, 1037, 17662, 12172, 2607,
2026, 2878, 2166, 1012]])
Final Inputs: {'input_ids': [101, 1045, 1005, 2310, 2042, 3403, 2005, 1037, 17662, 12172, 2607, 2026, 2878, 2166, 1012, 102], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
Decoded Inputs: [CLS] i've been waiting for a huggingface course my whole life. [SEP]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment