Commit: https://github.com/hosmada/emr/commit/2c27eb5381d082e0a9745085d384b8fbe74e83e6
What changed:
- Renamed two
Token#token_typevalues for clarity:word→japanese_wordandforeign→latin_word. - Added
SuikaTokenizer.join(tokens), a shared helper that reconstructs text from tokens correctly instead of naively joining with a fixed separator. It only inserts a space between two consecutive "space-delimited" tokens (Latin words, abbreviations, and numbers) — Japanese tokens never get a space. - Wired
Meeting#utterancesandReviewWorkflow#phi_review_datato use this new helper instead oftokens.map(&:text).join(" "). - Updated the token test factory to auto-classify
token_typefromtextvia the real tokenizer instead of hardcoding a default.
Why:
Our transcripts are Japanese, and the old .join(" ") inserted a literal space between every token regardless of script. That's correct for English but wrong for Japanese (no inter-word spaces) and also silently dropped real spacing in things like "2026 01 02" (which collapsed to "20260102"). The new logic gets both cases right, while leaving punctuation-delimited text like "2026-01-02" untouched.
Examples (before → after):
| Tokens | Old .join(" ") |
New SuikaTokenizer.join |
|---|---|---|
今回, は (Japanese) |
"今回 は" ❌ |
"今回は" ✅ |
MRI (abbrev), 検査 (Japanese) |
"MRI 検査" ❌ |
"MRI検査" ✅ |
Hello, there (Latin) |
"Hello there" ✅ |
"Hello there" ✅ (unchanged) |
2026, 01, 02 (numbers) |
"2026 01 02" ✅ (accidentally) |
"2026 01 02" ✅ (correctly, by rule) |
2026, -, 01 (number/punctuation) |
"2026 - 01" ❌ |
"2026-01" ✅ |
Risk/impact: Existing PHI/review text derived from tokens will now render with more accurate spacing.