Martin Milan martin12333

Segmentation 101, part 1: Why your strategy matters

I recent did some more exploring with a local LLM tool that would import your documents into a vector store. Given the promising initial results with a handful of docs I wanted to see how it handled more / different data. I decided to copy over the text files containing Expanse trivia and answers I use as a regression suite to test my own "Q&A over documents" process. I wanted to see what types of questions it could answer from that content...

The Problem With Generic Segmentation

The strategy employed by this tool used double newlines as their segmentation boundary condition. A strategy that works well for many types of content however for this content that was a terrible choice as the text in the files are formatted with numbered questions followed by their answers like this:

1. Long winded question with establishing context

Each of these commands will run an ad hoc http static server in your current (or specified) directory, available at http://localhost:8000. Use this power wisely.

Discussion on reddit.

Python 2.x

$ python -m SimpleHTTPServer 8000

	$ori = @{}
	Try {
	$i = 0

	# Loading .env files
	if(Test-Path $args[0]) {
	foreach($line in (Get-Content $args[0])) {
	if($line -Match '^\s*$' -Or $line -Match '^#') {
	continue
	}

	C:\Users\user\Desktop>cscript //E:{1b7cd997-e5ff-4932-a7a6-2a9e636da385} repl.js
	Microsoft (R) Windows Script Host Version 5.812
	Copyright (C) Microsoft Corporation. All rights reserved.

	> if(true){\
	> 1\
	> }
	1
	> var a = 10
	undefined

	So we need people to have weird new
	ideas ... we need more ideas to break it
	and make it better ...

	Use it. Break it. File bugs. Request features.

	- Soledad Penadés, Real time front-end alchemy, or: capturing, playing,
	altering and encoding video and audio streams, without
	servers or plugins!

	/*
	feidlambda v0.4.0 - LOGIC / UTILITIES FUNCTIONS BY FIAKO ENGINEERING
	OFFICIAL GIST (feidlambda v0.4.x):
	https://gist.github.com/taruma/b4df638ecb7af48ab63691951481d6b2
	REPOSITORY:
	https://github.com/fiakoenjiniring/feidlambda
	CONTRIBUTOR: @taruma, @iingLK
	TESTED: Microsoft Excel 365 v2304
	*/

	// ==UserScript==
	// @name jQuery For Chrome (A Cross Browser Example)
	// @namespace jQueryForChromeExample
	// @include *
	// @author Erik Vergobbi Vold
	// @description This userscript is meant to be an example on how to use jQuery in a userscript on Google Chrome.
	// ==/UserScript==

	// a function that loads jQuery and calls a callback function when jQuery has finished loading
	function addJQuery(callback) {

	--------------------------------------------
	Version: 1.45.1
	Commit: 5763d909d5f12fe19f215cbfdd29a91c0fa9208a
	Date: 2020-05-14T08:33:47.663Z
	Electron: 7.2.4
	Chrome: 78.0.3904.130
	Node.js: 12.8.1
	V8: 7.8.279.23-electron.0
	OS: Darwin x64 18.5.0
	-------------------------------------------

	#!/bin/bash

	# Locate the history file in your profile, and copy it to the same folder as this script.
	# On Mac: ~/Library/Application\ Support/Google/Chrome/Default/History
	# On Windows: C:\Users\YOUR USER NAME\AppData\Local\Google\Chrome\User Data\Default\History

	sqlite3 History <<!
	.headers on
	.mode csv
	.output out.csv