This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ab: Abkhaz | |
ar: Arabic | |
as: Assamese | |
br: Breton | |
ca: Catalan | |
cnh: Hakha Chin | |
cs: Czech | |
cv: Chuvash | |
cy: Welsh | |
de: German |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fn main() -> anyhow::Result<()> { | |
let model = SentenceEmbeddingsBuilder::remote(SentenceEmbeddingsModelType::AllMiniLmL12V2).create_model()?; | |
let json = fs::read_to_string("data/books.json")?; | |
let library: Library = serde_json::from_str(&json)?; | |
let mut embeddedbooks = Vec::new(); | |
for book in library.books.clone() { | |
println!("Embedding book: {}", book.title); | |
let embeddings = model.encode(&[book.clone().summary])?; | |
let embedding = to_array(embeddings[0].as_slice()); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
impl EmbeddedBook { | |
fn topic(embeddings: [f32; 384]) -> Self { | |
Self { | |
title: None, | |
author: None, | |
summary: None, | |
embeddings: embeddings, | |
} | |
} | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fn main() -> anyhow::Result<()> { | |
let model = SentenceEmbeddingsBuilder::remote(SentenceEmbeddingsModelType::AllMiniLmL12V2).create_model()?; | |
let json = fs::read_to_string("data/books.json")?; | |
let library: Library = serde_json::from_str(&json)?; | |
let mut embeddedbooks = Vec::new(); | |
for book in library.books.clone() { | |
println!("Embedding book: {}", book.title); | |
let embeddings = model.encode(&[book.clone().summary])?; | |
let embedding = to_array(embeddings[0].as_slice()); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
impl KdPoint for EmbeddedBook { | |
type Scalar = f32; | |
type Dim = typenum::U2; // 2 dimensional tree. | |
fn at(&self, k: usize) -> f32 { | |
self.embeddings[k] | |
} | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fn main() -> anyhow::Result<()> { | |
+ let model = SentenceEmbeddingsBuilder::remote(SentenceEmbeddingsModelType::AllMiniLmL12V2).create_model()?; | |
let json = fs::read_to_string("data/books.json")?; | |
let library: Library = serde_json::from_str(&json)?; | |
+ let mut embeddedbooks = Vec::new(); | |
for book in library.books.clone() { | |
- println!("Embedding book: {}", book.title); | |
+ let embeddings = model.encode(&[book.clone().summary])?; | |
+ let embedding = to_array(embeddings[0].as_slice()); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#[derive(Debug)] | |
pub struct EmbeddedBook { | |
pub title: String, | |
pub author: String, | |
pub summary: String, | |
pub embeddings: [f32; 384], | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fn main() -> anyhow::Result<()> { | |
let json = fs::read_to_string("data/books.json")?; | |
let library: Library = serde_json::from_str(&json)?; | |
for book in library.books.clone() { | |
println!("Embedding book: {}", book.title); | |
} | |
Ok(()) | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
use serde::{Deserialize}; | |
#[derive(Debug, Deserialize)] | |
pub struct Library { | |
pub books: Vec<Book>, | |
} | |
#[derive(Debug, Deserialize, Clone)] | |
pub struct Book { | |
pub title: String, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"books": [ | |
{ | |
"title": "The Great Gatsby", | |
"author": "F. Scott Fitzgerald", | |
"summary": "The story primarily concerns the young and mysterious millionaire Jay Gatsby and his quixotic passion and obsession with the beautiful former debutante Daisy Buchanan." | |
}, | |
{ | |
"title": "The Catcher in the Rye", | |
"author": "J. D. Salinger", |
NewerOlder