Yes — while not literal basis vectors like in linear algebra, there are useful analogues in neural embedding spaces that serve similar purposes.
- A basis is a minimal set of vectors that span a space.
- Any vector can be written as a linear combination of basis vectors.
- In 3D space:
v = a·x̂ + b·ŷ + c·ẑ
In LLMs, words and concepts live in high-dimensional vector spaces (e.g., 768+ dimensions). These spaces encode semantic relationships geometrically.
So what’s the equivalent of basis vectors here?
You can find interpretable directions such as:
vec("man") - vec("woman")→ gender axisvec("king") - vec("queen")→ royalty axis
These act like semantic axes you can project onto:
They’re not formal basis vectors, but they behave similarly.
By applying PCA or SVD to embedding spaces, you get dominant axes:
- First few components often align with interpretable properties (e.g. frequency, sentiment).
- These can serve as an orthogonal basis for dimensionality reduction and interpretation.
- Linear classifiers (a.k.a. probes) can separate grammatical, syntactic, or semantic properties.
- These learned directions form subspaces useful for reasoning and analysis.
- No fixed or universal basis vectors
- Directions may not be orthogonal
- Embedding spaces may be nonlinear manifolds, not strict vector spaces
| Conceptual Role | Analogue in Embedding Space |
|---|---|
| Basis vectors | Latent directions, semantic analogies |
| Orthogonal components | PCA / SVD axes |
| Projection | Dot product or cosine along latent axes |
| Vector decomposition | Probing via linear classifiers |
You can:
- Use cosine similarity to find projection strength along semantic directions
- Apply PCA to visualize embeddings
- Build linear probes to test for specific properties