Skip to content

Instantly share code, notes, and snippets.

@usametov

usametov/clustering-clj.md

Created April 11, 2025 03:23
Show Gist options
  • Save usametov/15a9f471c052b06ce9fa7617396fb0f5 to your computer and use it in GitHub Desktop.
Save usametov/15a9f471c052b06ce9fa7617396fb0f5 to your computer and use it in GitHub Desktop.
Download ZIP
Clustering embedding vectors, k-means smile
Raw
clustering-clj.md

https://grok.com/share/bGVnYWN5_42d104d8-881b-46ad-874e-b3f85d45a8f7

Clustering embedding vectors is a great way to group similar items, and picking random samples from each cluster can help with tasks like data exploration or sampling for further analysis. Since you’re working in Clojure and mentioned KNN (likely referring to K-Nearest Neighbors, though it’s more a classification algorithm than a clustering one), I’ll assume you meant clustering algorithms in general, like K-Means, and I’ll focus on options available in Clojure or libraries accessible from it.

Clustering Algorithms

Here’s an overview of clustering algorithms you can use for your 1000 embedding vectors, along with their suitability:

  1. K-Means Clustering

    • How it works: Partitions data into k clusters by minimizing the variance within each cluster. It assigns each vector to the nearest centroid, then updates centroids iteratively.
    • Pros:
      • Simple and fast, scales well for 1000 vectors.
      • Works well for spherical clusters and embeddings with clear separations.
    • Cons:
      • Requires you to specify k (number of clusters).
      • Sensitive to outliers and initial centroid placement.
      • Assumes clusters are roughly equal in size and density.
    • Use case: Good if your embeddings form compact, roughly equal-sized groups.

  2. Hierarchical Clustering

    • How it works: Builds a tree (dendrogram) of clusters by either merging smaller clusters (agglomerative) or splitting larger ones (divisive). You can cut the tree at a desired level to get clusters.
    • Pros:
      • No need to specify k upfront; you can choose clusters after seeing the dendrogram.
      • Captures nested structures in data.
    • Cons:
      • Computationally expensive for large datasets (O(n²) or worse), though 1000 vectors is manageable.
      • Requires choosing a linkage criterion (e.g., single, complete, average).
    • Use case: Useful if you suspect hierarchical relationships in your embeddings or want flexibility in cluster granularity.

  3. DBSCAN (Density-Based Spatial Clustering of Applications with Noise)

    • How it works: Groups vectors based on density, forming clusters where points are close together and marking sparse areas as noise.
    • Pros:
      • Doesn’t require specifying k.
      • Handles irregularly shaped clusters and identifies outliers automatically.
    • Cons:
      • Sensitive to parameters (epsilon for distance, minPts for core points).
      • Struggles with clusters of varying densities.
    • Use case: Ideal if your embeddings have uneven densities or you expect noise/outliers.

  4. Gaussian Mixture Models (GMM)

    • How it works: Assumes data points are generated from a mixture of Gaussian distributions and estimates their parameters (mean, covariance).
    • Pros:
      • Probabilistic assignments (soft clustering), so vectors can belong to multiple clusters with probabilities.
      • Handles elliptical clusters better than K-Means.
    • Cons:
      • Requires specifying k.
      • Computationally intensive and sensitive to initialization.
    • Use case: Good for embeddings with overlapping or less distinct clusters.

  5. Spectral Clustering

    • How it works: Uses the eigenvalues of a similarity matrix to reduce dimensionality, then applies clustering (e.g., K-Means) in the lower-dimensional space.
    • Pros:
      • Captures complex, non-linear cluster structures.
      • Works well for high-dimensional embeddings.
    • Cons:
      • Requires specifying k.
      • Computationally expensive for large datasets (due to similarity matrix).
    • Use case: Useful if your embeddings form non-convex clusters.

Clojure Libraries and Tools

Clojure doesn’t have a native, comprehensive machine learning ecosystem like Python’s scikit-learn, but several libraries and interop options can help you cluster your vectors and sample from clusters. Here are your options:

  1. Neanderthal

    • What it is: A high-performance library for matrix and vector operations in Clojure, optimized for CPU/GPU.
    • Clustering support:
      • Doesn’t have built-in clustering algorithms, but you can implement K-Means or others using its fast matrix operations.
      • For example, compute Euclidean distances for K-Means or similarity matrices for Spectral Clustering.
    • Sampling:
      • Use Clojure’s rand-nth or shuffle to pick random vectors from each cluster after grouping.
    • Pros:
      • Very fast for numerical computations, especially for 1000 vectors.
      • Native Clojure, no JVM interop issues.
    • Cons:
      • Requires implementing clustering algorithms yourself.
      • Steeper learning curve for custom implementations.
    • Example (K-Means skeleton): 
      (require '[uncomplicate.neanderthal.core :as nc])
(require '[uncomplicate.neanderthal.native :as nn])
(defn k-means [vectors k max-iter]
  (let [n (count vectors)
        centroids (take k (shuffle vectors))]
    ;; Implement centroid assignment and update using Neanderthal ops
    ))

  2. Clj-ml

    • What it is: A Clojure machine learning library wrapping Weka (Java-based ML toolkit).
    • Clustering support:
      • Supports K-Means, Hierarchical Clustering, and DBSCAN via Weka.
      • Example: Use weka.clusterers.SimpleKMeans or weka.clusterers.HierarchicalClusterer.
    • Sampling:
      • After clustering, group vectors by cluster ID and use rand-nth to sample.
    • Pros:
      • Easy to use for standard algorithms.
      • No need to implement clustering logic yourself.
    • Cons:
      • Weka dependency can be heavy.
      • Limited flexibility for custom algorithms.
    • Example (K-Means with clj-ml): 
      (require '[clj-ml.clusterers :as c])
(defn cluster-and-sample [vectors k]
  (let [dataset (c/make-dataset :vectors (mapv vector vectors))
        clusterer (c/make-clusterer :k-means {:number-clusters k})
        clustered (c/clusterer-build clusterer dataset)
        assignments (c/clusterer-cluster clusterer dataset)]
    ;; Group vectors by assignments, then sample
    (->> assignments
         (map-indexed (fn [i c] [c (nth vectors i)]))
         (group-by first)
         (map (fn [[_ v]] (rand-nth (map second v)))))))

  3. Java Interop with Smile

    • What it is: Smile is a Java-based ML library with a rich set of clustering algorithms.
    • Clustering support:
      • Offers K-Means, DBSCAN, Hierarchical Clustering, GMM, Spectral Clustering, and more.
      • Example: smile.clustering.KMeans, smile.clustering.DBSCAN.
    • Sampling:
      • Retrieve cluster labels and sample using Clojure’s core functions.
    • Pros:
      • Comprehensive and well-documented.
      • Fast and optimized for Java ecosystems.
    • Cons:
      • Requires Java interop, which can feel clunky in Clojure.
      • Dependency management via Maven/Leiningen.
    • Example (K-Means with Smile): 
      (import '[smile.clustering KMeans])
(defn cluster-and-sample [vectors k]
  (let [data (into-array (map double-array vectors))
        kmeans (KMeans/fit data k)
        labels (.y kmeans)
        grouped (group-by (fn [i] (aget labels i)) (range (count vectors)))]
    (map (fn [[_ idxs]] (nth vectors (rand-nth idxs))) grouped)))

  4. Incanter

    • What it is: A Clojure library for statistical computing, inspired by R.
    • Clustering support:
      • Basic support for K-Means and Hierarchical Clustering.
      • Less comprehensive than Smile or Weka but simpler to use.
    • Sampling:
      • Similar to others, group by cluster and use rand-nth.
    • Pros:
      • Lightweight and Clojure-native.
      • Good for quick prototyping.
    • Cons:
      • Limited algorithm selection.
      • Slower for large datasets compared to Neanderthal or Smile.
    • Example: 
      (require '[incanter.core :as i])
(require '[incanter.clustering :as ic])
(defn cluster-and-sample [vectors k]
  (let [matrix (i/matrix vectors)
        clusters (ic/k-means matrix :k k)
        grouped (group-by :cluster (map-indexed (fn [i c] {:idx i :cluster c}) clusters))]
    (map (fn [[_ v]] (nth vectors (:idx (rand-nth v)))) grouped)))

  5. Python Interop with Libpython-clj

    • What it is: A library to call Python code (e.g., scikit-learn) from Clojure.
    • Clustering support:
      • Access scikit-learn’s full suite: K-Means, DBSCAN, GMM, Spectral Clustering, etc.
    • Sampling:
      • Return cluster labels to Clojure and sample as usual.
    • Pros:
      • Access to state-of-the-art implementations.
      • Familiar for those who know Python’s ecosystem.
    • Cons:
      • Overhead of Python interop.
      • Requires setting up a Python environment.
    • Example: 
      (require '[libpython-clj.python :as py])
(py/initialize!)
(def sklearn-cluster (py/import-module "sklearn.cluster"))
(defn cluster-and-sample [vectors k]
  (let [kmeans (py/call-attr sklearn-cluster "KMeans" :n_clusters k)
        labels (py/call-attr kmeans "fit_predict" (into-array (map double-array vectors)))
        grouped (group-by identity labels)]
    (map (fn [[_ idxs]] (nth vectors (rand-nth idxs))) grouped)))

Recommendations

For your case (1000 embedding vectors, clustering, and random sampling in Clojure):

  • If you want simplicity and speed: Use K-Means via Smile (Java interop). It’s fast, robust, and handles 1000 vectors easily. Sample with rand-nth afterward.
  • If you suspect non-spherical clusters or outliers: Try DBSCAN in Smile or clj-ml (Weka). Tune parameters carefully.
  • If you want flexibility in cluster numbers: Use Hierarchical Clustering in Smile or clj-ml and cut the dendrogram at a desired level.
  • If you’re comfortable with Python: Use libpython-clj with scikit-learn for the most options and best-tested implementations.
  • If you want a pure Clojure solution: Implement K-Means with Neanderthal for performance or use Incanter for simplicity, but be prepared for some manual work.

Practical Tips

  • Preprocessing: Ensure your embedding vectors are normalized (e.g., unit length) if using cosine similarity, or standardize them for Euclidean distance-based algorithms like K-Means.
  • Choosing k: For K-Means or GMM, use the elbow method or silhouette score to estimate k. Smile and scikit-learn have utilities for this.
  • Sampling: After clustering, group vectors by cluster label and use (rand-nth vectors) to pick random ones. For reproducibility, set a random seed with (set! *random* (java.util.Random. 42)).
  • Performance: 1000 vectors is small, so even hierarchical clustering or GMM should be fast enough on a modern machine.

Example Workflow (Smile, K-Means)

(ns my-project.core
  (:import [smile.clustering KMeans]))

(defn cluster-and-sample [vectors k n-samples-per-cluster]
  (let [data (into-array (map double-array vectors))
        kmeans (KMeans/fit data k)
        labels (.y kmeans)
        grouped (group-by (fn [i] (aget labels i)) (range (count vectors)))]
    (mapcat (fn [[_ idxs]]
              (take n-samples-per-cluster (shuffle idxs)))
            grouped)))

;; Example usage
(def vectors [[0.1 0.2] [0.15 0.25] [0.8 0.9] ...]) ;; Your 1000 vectors
(def sampled-idxs (cluster-and-sample vectors 5 2)) ;; 5 clusters, 2 samples each
(def sampled-vectors (map #(nth vectors %) sampled-idxs))

If you have more details (e.g., vector dimensionality, expected cluster shapes, or performance constraints), I can refine the recommendation further. Also, if you meant something specific by “knn” (e.g., a custom algorithm), let me know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment