Skip to content

Instantly share code, notes, and snippets.

View Black-Milk's full-sized avatar
🐇
turtles all the way down…

Edwin M. Black-Milk

🐇
turtles all the way down…
View GitHub Profile
@Black-Milk
Black-Milk / testRegex.js
Created May 30, 2025 22:26
Regex for chunking by using all semantic cues
// Updated: Aug. 20, 2024
// Run: node testRegex.js whatever.txt
// Live demo: https://jina.ai/tokenizer
// LICENSE: Apache-2.0 (https://www.apache.org/licenses/LICENSE-2.0)
// COPYRIGHT: Jina AI
const fs = require('fs');
const util = require('util');
// Define variables for magic numbers
const MAX_HEADING_LENGTH = 7;