Skip to content

Instantly share code, notes, and snippets.

@a-agmon
Created December 21, 2023 09:30
Show Gist options
  • Save a-agmon/537af634fd6fc290f3792b967e6b94a0 to your computer and use it in GitHub Desktop.
Save a-agmon/537af634fd6fc290f3792b967e6b94a0 to your computer and use it in GitHub Desktop.
save embed
let results: Vec<Result<Tensor, _>> = sentences
.par_chunks(350)
.map(|chunk| bert_model.create_embeddings(chunk.to_vec()))
.collect();
let embeddings = Tensor::cat(
&results
.iter()
.map(|r| r.as_ref().unwrap())
.collect::<Vec<_>>(),
0,).unwrap();
embeddings
.save_safetensors("my_embedding", "embeddings.bin")
.unwrap();
println!("embeddings.bin saved");
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment