Skip to content

Instantly share code, notes, and snippets.

View ljs904123126's full-sized avatar
🥑

lym ljs904123126

🥑
1 follower · 22 following
View GitHub Profile
@ljs904123126
ljs904123126 / chunking-regex.ts
Created August 15, 2024 03:01 — forked from hanxiao/testRegex.js
Use regex to do chunking by using all semantic cues
// Used in https://jina.ai/tokenizer (Aug. 14th version)
// Define variables for magic numbers
const MAX_HEADING_LENGTH = 6;
const MAX_HEADING_CONTENT_LENGTH = 200;
const MAX_HEADING_UNDERLINE_LENGTH = 200;
const MAX_HTML_HEADING_ATTRIBUTES_LENGTH = 100;
const MAX_LIST_ITEM_LENGTH = 200;
const MAX_NESTED_LIST_ITEMS = 5;
const MAX_LIST_INDENT_SPACES = 7;
const MAX_BLOCKQUOTE_LINE_LENGTH = 200;