Skip to content

Instantly share code, notes, and snippets.

@mdouze

mdouze/train_ivf_with_gpu.ipynb

Last active November 11, 2024 06:53
Show Gist options
  • Save mdouze/46d6bbbaabca0b9778fca37ed2bcccf6 to your computer and use it in GitHub Desktop.
Save mdouze/46d6bbbaabca0b9778fca37ed2bcccf6 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
train_ivf_with_gpu.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@fferroni
Copy link

fferroni commented Apr 13, 2023 •
edited
Loading

Hi @mdouze , is it correct that something like:
"PCAR1024,IVF262144_HNSW32,PQ512x8" (with inner product)
cannot be trained on the GPU?
https://gist.github.com/mdouze/46d6bbbaabca0b9778fca37ed2bcccf6?permalink_comment_id=3111847#gistcomment-3111847
You mention that PQ quantisation is not implemented?

@ucas010
Copy link

ucas010 commented Nov 11, 2024 •
edited
Loading

hi,dear
my dataset dim=768 and num=1billion
and what type of ivf PQ is good for the index in CPU ?
thx
@mdouze

@ucas010
Copy link

ucas010 commented Nov 11, 2024

the up code
index2 could be write directly ?
such as faiss.write_index(index2,"index2.idx") ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment