cutecutecat cutecutecat

How to use the LAION-400m benchmark scripts

Preset

Machine G: with nvidia GPU, such like A10 to build external K-means, with minimum hardware requirements:
- at least 120GB memory
Machine D: An instance like i7ie.2xlarge to do the benchmark, with minimum hardware requirements:
- at least 4000G NVME SSD
- at least 64GB memory
- at least 8 vcpus

Guide to creating a 2-layer index

Warning

The VectorChord v0.5.2 is recommended.

Please see step 4 for more information.

Step 1 - Train the bottom layer (Layer 2)

You can still use the original script.

conda create -n data PYTHON=3.11
conda activate data
conda install conda-forge::pgvector-python numpy pytorch::faiss-cpu conda-forge::psycopg tqdm

EXPORT OMP_WAIT_POLICY=PASSIVE

python sample_kmeans_export.py \
  --url "postgresql://postgres:123@localhost:5432/postgres" \
 --table embeddings2 \

Step 1

Download data from http://ann-benchmarks.com/gist-960-euclidean.hdf5

wget http://ann-benchmarks.com/gist-960-euclidean.hdf5
mv gist-960-euclidean.hdf5 gist.hdf5

Step 2

Insert data into data table:

With instance c7i.4xlarge (>= 16 cores)

Prepare environment

conda create -n data PYTHON=3.11
conda install anaconda::h5py anaconda::numpy conda-forge::hnswlib

Download file

	import numpy as np
	from pgvecto_rs.psycopg import register_vector
	import psycopg

	# generate random data
	rows = 100000
	dimensions = 128
	embeddings = np.random.rand(rows, dimensions)
	categories = np.random.randint(100, size=rows).tolist()
	queries = np.random.rand(10, dimensions)

	from os.path import join
	import os
	import time
	import argparse
	from pathlib import Path
	from tqdm import tqdm

	import psycopg
	import h5py
	from pgvecto_rs.psycopg import register_vector

	import seaborn as sns
	import matplotlib.pyplot as plt


	qps = [25.69, 18.91, 17.13, 16.18, 14.45, 7.24, 4.51]
	recall = [0.9079, 0.9461, 0.9497, 0.9535, 0.9608, 0.9650, 0.9672]
	p99_latency = [148.25, 260.68, 269.5, 302.29, 456.09, 1042.41, 1675.42]

	fig, (ax1, ax3) = plt.subplots(1, 2, figsize=(12, 5), gridspec_kw={'width_ratios': [20, 7]})

	import argparse
	import datetime
	import logging
	import math

	import psutil
	import numpy as np
	import psycopg
	from tqdm import tqdm
	from pgvector.psycopg import register_vector

	from matplotlib import ticker
	import numpy as np
	import pandas as pd
	import seaborn as sns
	import matplotlib.pyplot as plt

	QPS = [13.42, 12.32, 11.23, 8.75, 27.65, 25.36, 20.41, 16.66]

	original_data = {
	"acc": [