Skip to content

Instantly share code, notes, and snippets.

@freedomtowin

freedomtowin/intro-zero_hugging-face-course-tokenizers.py

Created November 17, 2024 23:16
Show Gist options
  • Save freedomtowin/500159ba690541039664a9800c78b71c to your computer and use it in GitHub Desktop.
Save freedomtowin/500159ba690541039664a9800c78b71c to your computer and use it in GitHub Desktop.
Download ZIP
Example of creating a Gist using Python
Raw
intro-zero_hugging-face-course-tokenizers.py
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
sentences = ["I love this movie!", "It was okay."]
# Tokenize and pad
encoded_input = tokenizer(sentences, padding=True, return_tensors="pt")
print(encoded_input)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment