Skip to content

Instantly share code, notes, and snippets.

@lopespm

lopespm/max_content_length_of_embeddings_model.py

Created June 9, 2024 08:36
Show Gist options
  • Save lopespm/342bd297f0cd6a2819f69a1ea10e7777 to your computer and use it in GitHub Desktop.
Save lopespm/342bd297f0cd6a2819f69a1ea10e7777 to your computer and use it in GitHub Desktop.
Download ZIP
[Python] Check which is the maximum content length for a given embeddings model
Raw
max_content_length_of_embeddings_model.py
# In this example, we get the maximum content length for the intfloat/multilingual-e5-large model (https://huggingface.co/intfloat/multilingual-e5-large)
from transformers import AutoConfig
checkpoint = "intfloat/multilingual-e5-large"
config = AutoConfig.from_pretrained(checkpoint)
print(f"Maximum context length for this model: {config.max_position_embeddings}")
### Output "Maximum context length for this model: 514"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment