Skip to content

Instantly share code, notes, and snippets.

View tazarov's full-sized avatar
🚀
Work

Trayan Azarov tazarov

🚀
Work
View GitHub Profile
@tazarov
tazarov / wal_clean.py
Last active January 18, 2024 12:27
Clean ChromaDB WAL (⚠️ This script has been superseded by https://github.com/amikos-tech/chromadb-ops ⚠️)
#!/usr/bin/env python3
# Call the script: python wal_clean.py ./chroma-test-compact
#!/usr/bin/env python3
# Call the script: python wal_clean.py ./chroma-test-compact
import argparse
import importlib
import os
import sqlite3
import typer
@tazarov
tazarov / cleanup.sh
Created January 9, 2024 13:56
Clean up ChromaDB defunct binary indices.
#!/bin/bash
if ! command -v sqlite3 &> /dev/null; then
echo "sqlite3 could not be found. Please install sqlite3 and try again."
exit 1
fi
PERSIST_DIR=$(realpath "$1")
BACKUP_DIR=$(realpath "$2")
@tazarov
tazarov / docker-compose.yaml
Last active April 19, 2025 23:32
Minimal Docker Compose for Chroma
version: '3.9'
networks:
net:
driver: bridge
services:
chromadb:
image: chromadb/chroma:latest
volumes:
- ./chromadb:/chroma/chroma
@tazarov
tazarov / llama_embeddings_for_chroma.py
Last active December 13, 2023 13:19
Chroma and LlamaIndex both offer embedding functions which are wrappers on top of popular embedding models. Unfortunately Chroma and LI's embedding functions are not compatible with each other. Below we offer an adapters to convert LI embedding function to Chroma one.
from llama_index.embeddings import OpenAIEmbedding
from llama_index.embeddings.base import BaseEmbedding
import chromadb
from chromadb.api.types import EmbeddingFunction
class LlamaIndexEmbeddingAdapter(EmbeddingFunction):
def __init__(self,ef:BaseEmbedding):
self.ef = ef
def __call__(self, input: Documents) -> Embeddings:
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@tazarov
tazarov / recent_entries.ipynb
Created November 29, 2023 09:20
Query collection with date filter
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@tazarov
tazarov / keyword_search.py
Created November 24, 2023 12:55
Keyword Search in Chroma
import chromadb
from chromadb.config import Settings
client = chromadb.PersistentClient(path="test", settings=Settings(allow_reset=True))
client.reset()
col = client.get_or_create_collection("test")
col.add(ids=["1", "2", "3"], documents=["He is a technology freak and he loves AI topics", "AI technology are advancing at a fast pace", "Innovation in LLMs is a hot topic"])
col.query(query_texts=["technology"], where_document={"$or":[{"$contains":"technology"}, {"$contains":"freak"}]})
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@tazarov
tazarov / cq_batching_with_lc.ipynb
Created September 20, 2023 14:15
Chroma Batching with Langchain
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@tazarov
tazarov / chroma_no_docs.py
Created August 15, 2023 13:44
This gist illustrates how to store vectors of your documents in chroma without providing your actual text documents. Useful if your docs contain sensitive info or you are mindful of the storage.
import uuid
from chromadb.utils import embedding_functions
import chromadb
ef = embedding_functions.DefaultEmbeddingFunction()
docs = ["Article by john", "Article by Jack", "Article by Jill"]
client = chromadb.Client()
embeddings = ef(docs)
collection = client.get_or_create_collection("test-where-list")