Skip to content

Instantly share code, notes, and snippets.

@bogdan
Last active July 16, 2026 11:04
Show Gist options
  • Select an option

  • Save bogdan/46d72a231b9778cadf1198c545fd7af1 to your computer and use it in GitHub Desktop.

Select an option

Save bogdan/46d72a231b9778cadf1198c545fd7af1 to your computer and use it in GitHub Desktop.
Rename token types and centralize space-aware token joining

Rename token types and centralize space-aware token joining

Commit: https://github.com/hosmada/emr/commit/2c27eb5381d082e0a9745085d384b8fbe74e83e6

What changed:

  • Renamed two Token#token_type values for clarity: wordjapanese_word and foreignlatin_word.
  • Added SuikaTokenizer.join(tokens), a shared helper that reconstructs text from tokens correctly instead of naively joining with a fixed separator. It only inserts a space between two consecutive "space-delimited" tokens (Latin words, abbreviations, and numbers) — Japanese tokens never get a space.
  • Wired Meeting#utterances and ReviewWorkflow#phi_review_data to use this new helper instead of tokens.map(&:text).join(" ").
  • Updated the token test factory to auto-classify token_type from text via the real tokenizer instead of hardcoding a default.

Why: Our transcripts are Japanese, and the old .join(" ") inserted a literal space between every token regardless of script. That's correct for English but wrong for Japanese (no inter-word spaces) and also silently dropped real spacing in things like "2026 01 02" (which collapsed to "20260102"). The new logic gets both cases right, while leaving punctuation-delimited text like "2026-01-02" untouched.

Examples (before → after):

Tokens Old .join(" ") New SuikaTokenizer.join
今回, (Japanese) "今回 は" "今回は"
MRI (abbrev), 検査 (Japanese) "MRI 検査" "MRI検査"
Hello, there (Latin) "Hello there" "Hello there" ✅ (unchanged)
2026, 01, 02 (numbers) "2026 01 02" ✅ (accidentally) "2026 01 02" ✅ (correctly, by rule)
2026, -, 01 (number/punctuation) "2026 - 01" "2026-01"

Risk/impact: Existing PHI/review text derived from tokens will now render with more accurate spacing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment