Karmak23 Karmak23

LLM Wiki

A pattern for building personal knowledge bases using LLMs.

This is an idea file, it is designed to be copy pasted to your own LLM Agent (e.g. OpenAI Codex, Claude Code, OpenCode / Pi, or etc.). Its goal is to communicate the high level idea, but your agent will build out the specifics in collaboration with you.

The core idea

Most people's experience with LLMs and documents looks like RAG: you upload a collection of files, the LLM retrieves relevant chunks at query time, and generates an answer. This works, but the LLM is rediscovering knowledge from scratch on every question. There's no accumulation. Ask a subtle question that requires synthesizing five documents, and the LLM has to find and piece together the relevant fragments every time. Nothing is built up. NotebookLM, ChatGPT file uploads, and most RAG systems work this way.

	# coding=UTF-8
	import nltk
	from nltk.corpus import brown

	# This is a fast and simple noun phrase extractor (based on NLTK)
	# Feel free to use it, just keep a link back to this post
	# http://thetokenizer.com/2013/05/09/efficient-way-to-extract-the-main-topics-of-a-sentence/
	# Create by Shlomi Babluki
	# May, 2013

	# -- coding: utf-8 --

	"""
	Whoosh backend for haystack that implements character folding, as per
	http://packages.python.org/Whoosh/stemming.html#character-folding .

	Tested with Haystack 2.4.0 and Whooch 2.7.0

	To use, put this file on your path and add it to your haystack settings, eg.

	#!/bin/bash

	### USAGE
	###
	### ./ElasticSearch.sh 1.5.0 will install Elasticsearch 1.5.0
	### ./ElasticSearch.sh 1.4.4 will install Elasticsearch 1.4.4
	### ./ElasticSearch.sh will fail because no version was specified (exit code 1)
	###
	### CLI options Contributed by @janpieper
	### Check http://www.elasticsearch.org/download/ for latest version of ElasticSearch

	#!/bin/bash

	#
	# lxc: linux Container library

	# Authors:
	# Daniel Lezcano <daniel.lezcano@free.fr>

	# This library is free software; you can redistribute it and/or
	# modify it under the terms of the GNU Lesser General Public

	import json
	import datetime
	from decimal import Decimal
	from mongoengine.queryset import queryset_manager
	from mongoengine.queryset import QuerySet
	from mongoengine.base import BaseList, BaseDict, ObjectId


	def list_encoder(inst, obj, field, force_string=False):
	"""