Logical word count is a cheap, deterministic, tokenizer-free measure of text length that stays stable across natural languages, code, math, and mixed content, and stays roughly proportional to LLM token counts — while remaining equal to ordinary word count for plain English prose.
It is defined in two rules:
- Wide (East Asian) characters count as ½ logical word each. This covers Chinese characters, Japanese kana, Korean Hangul, and CJK punctuation.
- Everything else is split on whitespace, and the word count is clamped so the
average word length stays between 3 and 6 characters. Text with very long “words”
(compounds, identifiers, URLs, minified code) is counted as
chars / 6; text with very short words is counted aschars / 3; typical prose passes through unchanged.
One number, two uses:
- A word count that survives any format. For ordinary English prose it equals the literal word count. On text where literal word counts break — code, mathematical notation, CJK, URLs, minified or structured data — it keeps measuring content volume instead of skewing by an order of magnitude, because average characters per logical word is confined to a narrow range by construction.
- A token estimate. Multiply by 1.6 to approximate o200k-family (GPT-5.x) token counts within roughly ±25% on any of those formats, a bound no fixed multiplier achieves on literal word counts or raw character counts (see Token Estimation).
A reference implementation is ~15 lines with no dependencies (see Reference Implementation). Empirically, tokens per logical word stays within a ~2× band for every tokenizer tested across ten languages plus source code and machine formats, versus a 24× spread for literal word count on the same corpus (see Validation).
Word count is the standard unit for measuring text length, but a whitespace-delimited word is a meaningful unit only for prose in spaced alphabetic scripts. The moment text departs from that format, literal word counts skew — sometimes by an order of magnitude, and in different directions for different formats.
Symbolic text breaks word counts in two opposite directions at once:
Symbols are rarely space-separated. The expression a+b*clamp(x,lo,hi) is one
whitespace-delimited “word” containing eight semantic tokens, and the LaTeX fragment
\frac{-b\pm\sqrt{b^2-4ac}}{2a} is one “word” carrying an entire quadratic formula.
Minified JavaScript averages 44 characters per whitespace-delimited “word” in our test
corpus; CSV rows and JSON average 10–14.
Identifiers vary wildly in length. x and calculateAverageResponseTime are both
one word, but the latter contains five semantic units.
Because the two failure modes skew in opposite directions, no fixed correction factor repairs word counts for symbolic text. Logical word count bounds both at the source: the clamp pins average characters per logical word to 3–6, so an expression, a formula, and a paragraph of prose with the same number of characters receive counts of the same order.
CJK text has few or no word-delimiting spaces; Unicode’s own segmentation standard notes that word boundaries in Chinese, Japanese, or Thai require dictionary lookup.1 A 500-character Chinese paragraph counts as a handful of whitespace-delimited “words” despite carrying the information of roughly 250 English words. In our test corpus, an 8,000-character Chinese Wikipedia extract contains just 165 whitespace-delimited tokens.
German compounds like Donaudampfschifffahrtsgesellschaft2 pack five lexical units into one “word”; agglutinative languages like Finnish average 8+ characters per whitespace-delimited word versus ~5.5 for English. Identical content, materially different word counts.
Documents that combine prose, code blocks, tables, links, and CJK text have no single word-counting strategy that works across all sections. A README with badges, URLs, and code samples measures 11.7 characters per “word” — double typical prose.
Token counts solve the normalization problem by construction, but they are not stable across model providers, they require a tokenizer vocabulary, and they are orders of magnitude more expensive to compute than a character scan.
Token counts vary across providers — and across model generations within a provider. The same 8,000-character samples measured against the latest production tokenizers (July 2026):3
| Sample | GPT-5.6 | Claude Opus 4.8 / Fable 5 | DeepSeek V4 | max deviation |
|---|---|---|---|---|
| English Wikipedia | 1,657 | 2,471 | 1,664 | 49% |
| German Wikipedia | 1,816 | 3,695 | 2,068 | 103% |
| Chinese Wikipedia | 5,351 | 7,348 | 4,333 | 70% |
| Japanese Wikipedia | 5,097 | 6,403 | 4,343 | 47% |
| Korean Wikipedia | 4,529 | 7,916 | 5,026 | 75% |
| Thai Wikipedia | 3,520 | 7,107 | 3,238 | 119% |
| Python code | 1,853 | 2,810 | 1,893 | 52% |
| TypeScript code | 1,907 | 3,518 | 2,004 | 84% |
Even for English the tokenizers diverge by 49%, and for German by 103%. Nor are counts stable over time: Anthropic’s newest tokenizer (shared by Claude Opus 4.8, Fable 5, and Sonnet 5) spends ~37% more tokens on the same English text than the Opus 4.6 tokenizer did, and ~2× on Thai. Academic work documents far larger gaps across languages: the same content can take up to 15× more tokens in some languages than in English.4 A “token count” is therefore not a property of a text — it is a property of a (text, model) pair. Any model-independent measure has this cross-tokenizer variance as a floor on its achievable accuracy, and that is fine: the goal is a stable measure of content volume that tokens track within a predictable band, not a replacement for a tokenizer when exact counts matter.
Token counts also require a vocabulary. Computing them means shipping or calling the target model’s tokenizer — a dependency that may not exist where length is measured (a database check constraint, a CLI, an editor plugin, a non-Python runtime) and that changes when models change.
Given a text string:
- Wide characters. Count the characters whose Unicode East Asian Width is Wide or
Fullwidth5 — Han ideographs, Hiragana, Katakana, Hangul syllables, CJK
punctuation, and fullwidth forms.
Call this
wide. Each contributes ½ logical word. For splitting purposes, treat each wide character as if it were surrounded by spaces. - Remaining text. Split what remains on whitespace.
Let
wordsbe the number of resulting tokens andcharsthe total count of their characters (Unicode code points). Clamp the word count so that average word length stays in[L, U] = [3, 6]:
clamped = clamp(words, chars / 6, chars / 3)
= max(chars / 6, min(chars / 3, words))
logical_words = round(wide / 2 + clamped)
Empty or whitespace-only text has 0 logical words.
The clamp activates only when average characters per word falls outside [3, 6]. Most
English and European prose runs 4.5–5.5 characters per whitespace-delimited word
(punctuation attached), so for ordinary prose logical word count equals literal word
count — the measure is backward-compatible where word count already works.
Long-word text (German and Finnish compounds, identifiers, URLs, minified code, CSV
rows) is counted as chars / 6; degenerate short-word text (e.g. tables of single
digits) is capped at chars / 3.
In practice the upper bound does nearly all the work; the lower bound is a safety rail that real content rarely hits.
- Primary definition: East_Asian_Width ∈ {W, F} (available in Python’s
unicodedata, ICU, and most Unicode libraries). Where the property is unavailable, a range-based approximation — the Han and Hangul scripts plus the CJK punctuation-through-Katakana blocks (U+3000–30FF), Katakana phonetic extensions (U+31F0–31FF), and fullwidth forms (U+FF00–FFEF) — produces identical counts on all our CJK test samples (kana must be matched by block, not script property: ー U+30FC and ・ U+30FB are Script=Common but Wide). Rare code points (halfwidth Katakana, some symbols) may still differ between the two definitions; the effect on real text is well under 1%. - Korean: Hangul is space-delimited, but each syllable block encodes 2–3 jamo and tokenizers price Korean like Chinese and Japanese (~1.4 chars/token). Weighting Hangul syllables at ½ overrides the space-based count and empirically aligns Korean with every other language (K = 1.57 vs. 2.54 without the rule; see Validation). The localization industry’s GMX-V standard likewise groups Korean with the logographic scripts.6
- Emoji are East_Asian_Width Wide, so they count ½ each. Tokenizers typically spend 1–3 tokens per emoji; at realistic emoji densities the error is negligible.
Independent conventions, developed for different purposes, converge on roughly two CJK characters per English-word-equivalent:
| Convention | Effective rate | Purpose |
|---|---|---|
| Translation-industry rule of thumb | ~1,000 Japanese chars ≈ 400–500 English words; 1,000 Chinese chars ≈ 650–750 English words7 | Quoting/billing |
| Japanese 原稿用紙 (genkō yōshi) manuscript page | 400 chars ≈ 200–250 English words8 | Publishing |
| Twitter/X character limit | CJK weighted 2× (280-unit limit = 140 CJK chars)9 | Length limiting |
| Medium and Hugo reading speed | 500 CJK chars/min vs. ~212–265 words/min ≈ 2–2.4 chars per word10 | Reading time |
| ICU dictionary segmentation | Chinese words average ~1.5–2 chars11 | Linguistic segmentation |
| Modern LLM tokenizers (measured) | one CJK char ≈ 0.7 tokens ≈ half the ~1.35 tokens of an English word | Token accounting |
The tokenizer alignment is the one this spec is calibrated to: weighting CJK characters at ½ word puts Chinese, Japanese, and Korean at 1.54–1.57 tokens per logical word under o200k — statistically indistinguishable from French (1.50) and German (1.57).
GMX-V, the localization industry’s published volume standard, uses somewhat higher divisors for source-word counts (Chinese ÷ 2.8, Japanese ÷ 3.0, Korean ÷ 3.3)6 — it estimates the number of words a native segmentation would find, whereas logical word count (like the billing ratios above) targets volume equivalence. Simple word-counting tools mostly land on 1× instead: Microsoft Word and memoQ count each CJK character as one word,12 and Unicode’s default word segmentation breaks between every pair of ideographs.1 Half a word per character sits deliberately between “one word per character” (too high: overweights CJK ~2× in any cross-language comparison) and GMX-V’s ~⅓ (too low for token tracking).
BPE tokenizers merge frequent byte sequences until each token carries roughly one “meaning-sized” chunk.13 Measured on large samples, modern large-vocabulary tokenizers (o200k, cl100k, Llama 3, Gemma 2) all land near 4.2–4.3 characters per token for English prose and 4.0–4.8 for source code;14 vendor rules of thumb agree (OpenAI: ~4 chars or ¾ of a word per token; Anthropic: ~3.5 chars; Google: ~4 chars).15 (Anthropic’s newest tokenizer is denser — ~2.7 characters per token on our English sample — but uniformly so across content types, which shifts K by a constant factor without widening the band the clamp is designed to hold.)
The bounds bracket that range without a vocabulary table:
- U = 6 is the point where long-word content (compounds, identifiers, URLs) stops
gaining count from packing more characters into each word.
chars / 6tracks the ~4–5 chars/token that tokenizers actually spend on such content, at a stable ratio of ~1.3–1.5 tokens per logical word. - L = 3 caps degenerate short-word content. Real prose almost never averages below 3 (even Korean, with dense syllable blocks, averages 3.5); the bound exists so that pathological inputs (single-character tables, heavily spaced tokens) cannot inflate the count more than ~1.5× over prose.
- Everything between 3 and 6 chars/word passes through, preserving
logical_words = literal wordsfor ordinary prose in English, French, Spanish, Russian, Hebrew, and most other spaced languages.
Tightening the bounds (e.g. [3, 5]) narrows the token-tracking band another ~10% but
sacrifices exact backward compatibility with English word count; loosening them (e.g.
[2, 7]) admits outliers without benefit.
[3, 6] with weight ½ is the recommended convention; all three constants are parameters
in the reference implementation for applications that want to recalibrate.
Python (uses the East_Asian_Width property directly):
import unicodedata
def logical_word_count(text: str, lower: float = 3.0, upper: float = 6.0,
wide_weight: float = 0.5) -> int:
wide = 0
rest = []
for c in text:
if unicodedata.east_asian_width(c) in ("W", "F"):
wide += 1
rest.append(" ")
else:
rest.append(c)
words = "".join(rest).split()
chars = sum(len(w) for w in words)
clamped = max(chars / upper, min(chars / lower, len(words)))
return round(wide * wide_weight + clamped)TypeScript (script-range approximation of East_Asian_Width; identical results on real-world text):
const WIDE =
/[\p{Script=Han}\p{Script=Hangul}\u3000-\u30FF\u31F0-\u31FF\uFF00-\uFFEF]/u;
function logicalWordCount(
text: string, lower = 3, upper = 6, wideWeight = 0.5,
): number {
let wide = 0;
const rest: string[] = [];
for (const ch of text) { // iterates code points, not UTF-16 units
if (WIDE.test(ch)) {
wide++;
rest.push(" ");
} else {
rest.push(ch);
}
}
const words = rest.join("").split(/\s+/).filter(Boolean);
const chars = words.reduce((n, w) => n + [...w].length, 0);
const clamped = Math.max(chars / upper, Math.min(chars / lower, words.length));
return Math.round(wide * wideWeight + clamped);
}Both are O(n), allocation-light, and dependency-free.
Measured with the companion script (validate_lwc.py) on 8,000-character samples:
Wikipedia extracts in ten languages (Hebrew and Thai are reported under
Limitations), real source files, minified JS, CSV, JSON, and a
mixed-content README. Token counts are GPT-5.6 via the OpenAI API (the o200k tokenizer
family); K = tokens / logical words.
| Sample | chars | literal words | logical words | tokens | K |
|---|---|---|---|---|---|
| English Wikipedia | 6,775 | 1,219 | 1,219 | 1,657 | 1.36 |
| French Wikipedia | 6,756 | 1,222 | 1,222 | 1,838 | 1.50 |
| German Wikipedia | 6,921 | 1,063 | 1,154 | 1,816 | 1.57 |
| Russian Wikipedia | 6,998 | 995 | 1,166 | 1,939 | 1.66 |
| Finnish Wikipedia | 7,104 | 868 | 1,184 | 2,309 | 1.95 |
| Chinese Wikipedia | 7,807 | 165 | 3,400 | 5,351 | 1.57 |
| Japanese Wikipedia | 7,642 | 300 | 3,300 | 5,097 | 1.54 |
| Korean Wikipedia | 6,181 | 1,781 | 2,888 | 4,529 | 1.57 |
| Python code | 5,709 | 862 | 952 | 1,853 | 1.95 |
| TypeScript code | 6,800 | 865 | 1,133 | 1,907 | 1.68 |
| Markdown README | 7,314 | 627 | 1,219 | 2,178 | 1.79 |
| Minified JavaScript | 7,825 | 176 | 1,304 | 2,889 | 2.22 |
| CSV data | 7,462 | 539 | 1,244 | 2,786 | 2.24 |
| JSON | 7,037 | 560 | 1,173 | 2,168 | 1.85 |
Prose, code, and CJK all land in K = 1.36–1.95 (a 1.43× band). Punctuation-dense machine formats run hotter, up to ~2.2. Compare the alternatives on the identical corpus:
| Measure | tokens ÷ measure, across samples | spread |
|---|---|---|
| logical words | 1.36 – 2.24 | 1.65× |
| literal words | 1.36 – 32.4 | 24× |
| chars ÷ 4 | 0.98 – 2.9 | 3.0× |
The same holds for the other tokenizer families. On the identical corpus (including the Hebrew and Thai limitation cases), tokens per logical word spans a 2.0× band for GPT-5.6, 2.1× for DeepSeek V4, and 2.9× for the Claude Opus 4.8 / Fable 5 tokenizer — each centered on its own K — while literal word count spans 19–24× on every one of them.
The remaining variation is largely irreducible: the tokenizers themselves disagree by 37–119% on identical text (see the table in Motivation), so no model-free measure can track any single model much more tightly than models track each other.
The count itself is the word estimate: a normalized, prose-equivalent word count that needs no per-format correction. A single multiplier turns it into a token estimate for o200k-family tokenizers (GPT-5.x; DeepSeek V4 runs ~10% higher):
estimated_tokens ≈ logical_words × 1.6
This lands within roughly ±25% across prose (any script), code, and markdown. Useful refinements:
- Clean English prose: use ×1.35 — which is the familiar “1 word ≈ 4⁄3 tokens” vendor rule,15 recovered exactly because logical words equal literal words for such text.
- Punctuation-dense machine formats (minified code, CSV, logs): use ×2.2, or accept under-estimation.
- Model-specific correction: apply a per-tokenizer factor — DeepSeek V4 runs ~1.1× o200k, and Anthropic’s newest tokenizer (Claude Opus 4.8 / Fable 5 / Sonnet 5) runs ~1.5–2× o200k depending on content, i.e. tokens ≈ logical_words × 2.75 overall. This correction is roughly constant per model, unlike the correction needed for raw word count, which varies by an order of magnitude with text type.
When exact counts matter (billing, hard context limits), use the model’s tokenizer; logical word count is for the much larger class of uses where a stable, cheap, model-free number is worth ±25%.
- Format-stable: one measure covers prose, code, math, markup, and data. Average characters per logical word is confined to a narrow range by construction (3–6 for spaced text, 2 for CJK), so no format skews the count by more than a small factor.
- Cheap: O(n) single pass, no vocabulary, no dependencies.
- Deterministic and model-free: same input, same count, forever.
- Backward-compatible: equals literal word count for ordinary English (and most European-language) prose.
- Script-aware but language-blind: uses only character properties; no language detection, dictionaries, or segmentation models.
- Monotonic: appending non-whitespace content never decreases the count.
- Additive: splitting text at whitespace boundaries and summing the parts’ counts approximates the whole (within rounding), so counts can be computed incrementally or in parallel.
- Bounded token ratio: every tokenizer tested spends a near-constant number of tokens per logical word — within a ~2–3× band across all content types, versus ~20–24× for literal word count.
- Unspaced non-CJK scripts (Thai, Lao, Khmer, Myanmar) are not covered by the
wide-character rule; they fall to the clamp’s
chars / 6floor. That happens to match GMX-V’s convention for Thai (chars ÷ 6),6 but o200k still spends ~2.75 tokens per logical word on Thai (Anthropic’s newest tokenizer, ~5.6) — high fertility for these scripts is a property of the tokenizers, not of the measure. Applications centered on these scripts should add a per-script weight (Thai ≈ ¼–⅓ word/char) by the same method used here for CJK. - Non-Latin alphabetic scripts (Hebrew, Greek, Devanagari, …) count correctly as volume — their word counts are true word counts — but tokenizers charge them 1.9–2.3 tokens per logical word versus 1.35 for English (o200k; other tokenizer families show the same skew at their own scale). This is the well-documented tokenizer-fairness gap,4 16 and logical word count deliberately does not bend the volume measure to compensate for it.
- Machine formats (minified code, CSV, dense JSON) run ~1.4× hotter in tokens than prose. The clamp keeps them bounded (literal word count is off by 24× here), but a single multiplier will under-estimate their token cost.
- It is not a linguistic word count. For Chinese, ½ word per character approximates
dictionary segmentation (Chinese words average 1.5–2 characters), but no claim is made
that
logical_wordsmatches what a human or a segmenter would count as words in any specific language. It is a normalized volume unit, like a “standard page” — not a lexical analysis.
No existing term or formula was found for this specific construction (a chars-per-word clamp plus a wide-character weight), and the name “logical word count” appears to be unclaimed. ("Standard word" already means 5 keystrokes in typing measurement;17 “weighted word count” means fuzzy-match discounts in CAT tools.) The pieces have long precedents:
- GMX-V (LISA → ETSI GS LIS 004) — the localization industry’s word/character count standard — is the closest precedent. Version 1.0 (2007) refused to segment unspaced scripts and required character counts; version 2.0 (2012) added normative per-script divisors to convert characters to words: Chinese ÷ 2.8, Japanese ÷ 3.0, Korean ÷ 3.3, Thai ÷ 6, described as “acknowledged best practice within the Localization Industry.”6 Logical word count generalizes the same move — chars ÷ factor — and extends it to code and mixed content via the clamp.
- CJK character-as-word conventions: Microsoft Word’s “Words” statistic counts each East Asian character as a word (with a separate Asian-characters statistic); memoQ computes source words as Asian characters plus non-Asian words; WordPress locale packs switch Japanese and Chinese to character counting outright.12
- Weighted length limits: Twitter/X’s 280-unit limit weights code points beyond a small Latin-ish set (roughly through U+10FF plus general punctuation) at 2, so CJK posts max out at 140 characters — a widely deployed precedent for weighting wide characters at 2× Latin.9
- Reading-time estimators: Medium uses ~265 words/min for spaced languages and 500
characters/min for CJK; Hugo and the
reading-timenpm package (used by Gatsby, Ghost analogues) count CJK per-character against a ~2.4× higher rate.10 - Character-based billing units: Germany’s court-interpreting law (§ 11 JVEG) prices translation per 55 keystrokes ("je angefangene 55 Anschläge"); publishing standard pages (Normseite: 1,500–1,800 chars) and typing’s 5-character “standard word” all define length by characters when word counts are unreliable or unfair.18 17 Amazon’s KENPC normalizes e-book length to a synthetic page unit for paying authors.19
- Character length as a text statistic: the Automated Readability Index (1967) used chars/word as a machine-friendly complexity proxy precisely because characters were cheap to count.20
- Tokenizer research: subword fertility (tokens per word) is the standard metric for tokenizer efficiency;21 the unfairness of token counts across languages is documented up to 15×.4 16 BPE itself13 solves normalization by construction, at the cost of a vocabulary and model dependence.
- Tokenizer-free token estimators (the
tokenxnpm package; LangChain’scount_tokens_approximately; Semantic Kernel’s chars ÷ 4) estimate tokens for a specific model family from character ratios.22 Logical word count differs in intent: it is a stable text measure first — human-legible, word-count compatible — that happens to track tokens within a known band, rather than a model-specific predictor.
All measurements in this document come from the companion script validate_lwc.py
(published alongside this document), which fetches the public samples, computes literal
words, logical words, and exact token counts via provider APIs, and prints the
comparison tables:
uv run validate_lwc.py # o200k locally; other providers with API keys
uv run validate_lwc.py --lower 3 --upper 6 --wide-weight 0.5
uv run validate_lwc.py --list-samples
With no API keys, OpenAI counts fall back to local tiktoken (o200k_base — measured
identical to the API counts modulo ~6 tokens of message framing).
Set OPENAI_API_KEY, ANTHROPIC_API_KEY, and/or DEEPSEEK_API_KEY to reproduce the
cross-model table with exact provider counts; Gemini and xAI models are supported via
--models with their keys.
The script’s dependencies are pinned under a two-week cooling-off policy (uv’s
exclude-newer).
Footnotes
-
Unicode Standard Annex #29, “Unicode Text Segmentation,” §4 Word Boundaries: default rules break between every pair of ideographs (rule WB999), and “reliable detection of word boundaries in languages such as Thai, Lao, Chinese, or Japanese requires the use of dictionary lookup or other mechanisms.” https://unicode.org/reports/tr29/ ↩ ↩2
-
Danube–steam–ship–journey–company; the Erste Donau-Dampfschiffahrts-Gesellschaft (founded 1829) is the textbook example of German nominal compounding. https://en.wikipedia.org/wiki/Donaudampfschiffahrtsgesellschaft ↩
-
Measured 2026-07-12 with
validate_lwc.pyon 8,000-character truncations of Wikipedia articles and open-source code. GPT-5.6 = OpenAI chat completions APIusage.prompt_tokens(the three GPT-5.6 variants — luna, sol, terra — return identical counts, matching tiktoken’s o200k_base within ~6 tokens of message framing); Claude = Anthropiccount_tokensAPI (Claude Opus 4.8, Fable 5, and Sonnet 5 return identical counts — one shared tokenizer); DeepSeek = OpenAI-compatible completions API (deepseek-v4-pro and v4-flash identical). An earlier five-provider run that included Gemini 3.1 Pro and Grok-4 showed the same picture (35–79% spread). ↩ -
Petrov, La Malfa, Torr, and Bibi, “Language Model Tokenizers Introduce Unfairness Between Languages,” NeurIPS 2023. arXiv:2305.15425. “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases” (Shan vs. English under cl100k; even the cheapest languages carry a ~50% premium over English). https://arxiv.org/abs/2305.15425 ↩ ↩2 ↩3
-
Unicode Standard Annex #11, “East Asian Width.” https://www.unicode.org/reports/tr11/ ↩
-
ETSI GS LIS 004 V2.0.0 (2012-07), “GMX-V: Global Information Management Metrics eXchange — Volume” (successor to the LISA standard), clause 4.2.13 “Logographic Scripts”: word counts for unspaced scripts are derived from character counts via fixed factors — Chinese 2.8, Japanese 3.0, Korean 3.3, Thai 6.0 — “based on acknowledged best practice within the Localization Industry.” https://www.etsi.org/deliver/etsi_gs/lis/001_099/004/02.00.00_60/gs_lis004v020000p.pdf ↩ ↩2 ↩3 ↩4
-
E.g. AnyCount, “Word count in Oriental languages”: 1,000 Chinese characters ≈ 650–750 English words (https://www.anycount.com/word-count-worldwide/word-count-in-oriental-languages/); 1StopAsia gives the same Chinese figures and recommends character counts for Chinese/Japanese/Thai (https://www.1stopasia.com/blog/word-count-vs-character-count-for-asian-languages/). ↩
-
SWET (Society of Writers, Editors and Translators, Tokyo), “What’s in a Page?”: the 400-character genkō yōshi manuscript page corresponds to roughly 200–250 English words. https://swet.jp/articles/article/whats_in_a_page/_C31 ↩
-
X developer documentation, “Counting characters,” and the open-source twitter-text library config (v3.json): code points U+0000–10FF plus selected general-punctuation ranges weigh 1; everything else (all CJK, Hangul, emoji) weighs 2; URLs count as 23. https://docs.x.com/fundamentals/counting-characters, https://github.com/twitter/twitter-text/blob/master/config/v3.json ↩ ↩2
-
Medium Help Center, “Read time”: “roughly 265 WPM… For posts in Chinese, Japanese and Korean, it’s a function of number of characters (500 characters/min).” Hugo (
hugolib/page__content.go) uses the same 500 chars/min for CJK vs. ~213 wpm otherwise, and counts runes per CJK token whenhasCJKLanguageis set. Thereading-timenpm package counts each CJK character as one word. https://help.medium.com/hc/en-us/articles/214991667, https://github.com/gohugoio/hugo, https://github.com/ngryman/reading-time ↩ ↩2 -
ICU segments Chinese and Japanese with a frequency dictionary (cjdict) rather than Unicode default rules; dictionary-segmented Chinese words average roughly 1.5–2 characters. https://unicode-org.github.io/icu/userguide/boundaryanalysis/ ↩
-
Microsoft Word: “In East Asian languages, each character counts as a word” (NumChars documentation), with a separate
wdStatisticFarEastCharactersstatistic (https://learn.microsoft.com/en-us/office/vba/api/word.wdstatistic). memoQ: “Source words = Asian characters + non-Asian words” for Chinese/Japanese sources (https://docs.memoq.com/current/en/Workspace/statistics.html). WordPress: locale packs translate the word-count type to character counting for Japanese and Chinese (WP_Locale::get_word_count_type()). ↩ ↩2 -
Sennrich, Haddow, and Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL 2016 — the original BPE-for-NLP paper. https://aclanthology.org/P16-1162/ ↩ ↩2
-
Measured on ~300k characters of English prose (Pride and Prejudice) and ~165k characters of Python stdlib source: cl100k_base 4.24/4.81 chars per token (prose/code), o200k_base 4.28/4.77, Llama 3 4.25/4.81, Gemma 2 4.29/4.03. Shorter, punctuation- or link-heavy samples run lower (3.1–3.6 in the validation corpus). ↩
-
OpenAI: “1 token ≈ 4 characters… ≈ ¾ of a word; 100 tokens ≈ 75 words” (https://help.openai.com/en/articles/4936856) and “1 token is approximately 4 characters or 0.75 words for English text” (https://developers.openai.com/api/docs/concepts). Anthropic: “a token approximately represents 3.5 English characters” (https://platform.claude.com/docs/en/docs/resources/glossary). Google: “a token is equivalent to about 4 characters… 100 tokens is equal to about 60-80 English words” (https://ai.google.dev/gemini-api/docs/tokens). ↩ ↩2
-
Ahia et al., “Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models,” EMNLP 2023. arXiv:2305.13707. Documents ~5× token-count and API-cost premiums for non-Latin-script languages. https://aclanthology.org/2023.emnlp-main.614/ ↩ ↩2
-
Words-per-minute measurement standardizes a “word” as 5 characters or keystrokes (including spaces and punctuation). https://en.wikipedia.org/wiki/Words_per_minute ↩ ↩2
-
§ 11 JVEG (Justizvergütungs- und -entschädigungsgesetz): translation honoraria “für jeweils angefangene 55 Anschläge des schriftlichen Textes” (per commenced 55 keystrokes). https://www.gesetze-im-internet.de/jveg/__11.html. Normseite: 30 lines × 60 keystrokes = 1,800 characters (publishing convention); VG Wort uses 1,500. https://de.wikipedia.org/wiki/Normseite ↩
-
Amazon KDP, “Royalties in Kindle Unlimited”: KENPC (Kindle Edition Normalized Page Count) normalizes page count using standard formatting settings; the formula is unpublished (author analyses estimate ~170–200 words per page). https://kdp.amazon.com/en_US/help/topic/G201541130 ↩
-
Senter and Smith, “Automated Readability Index,” AMRL-TR-66-220, 1967. Formula: ARI = 4.71 × (chars/words) + 0.5 × (words/sentences) − 21.43. https://apps.dtic.mil/sti/tr/pdf/AD0667273.pdf ↩
-
Ács, “Exploring BERT’s Vocabulary” (2019) defined subword fertility — average subwords per word — borrowing the term from statistical MT (Brown et al. 1993); Rust et al., “How Good is Your Tokenizer?” (ACL 2021) established it as the standard tokenizer-efficiency metric. https://juditacs.github.io/2019/02/19/bert-tokenization-stats.html, https://aclanthology.org/2021.acl-long.243/ ↩
-
tokenx: character-ratio estimation with per-language overrides, “~96% accuracy” (https://github.com/johannschopplich/tokenx). LangChain
count_tokens_approximately: chars ÷ 4 plus per-message overhead. Microsoft Semantic Kernel TextChunker: length ÷ 4. ↩