bee-san · August 4, 2026 18:02
diff --git a/gistfile1.txt b/gistfile1.txt
  # Small Experimental Hoshidicts Reader and Mining Integration

  ## Summary

  - Add Hoshidicts behind enable_experimental_features && enable_hoshidicts; both default off. Changing the Hoshidicts toggle requires restarting the overlay.
  - When enabled, Hoshidicts handles Japanese term scanning and the Hoshi Reader popup. Yomitan scanning/popups are disabled, but its GSM bridge remains loaded for minimal Anki mining. When
    disabled, existing Yomitan behavior is unchanged.

  - V1 includes dictionary import/removal, scheduled updates, Japanese word lookup, and one mouse-driven mining button. Exclude controllers, hotkeys, kanji lookup, nested popup scanning,
    word audio, dictionary media, frequency/pitch export, note browsing, and multiple mining formats.

  - Pin the currently verified bases: Hoshidicts main (https://github.com/Manhhao/hoshidicts/tree/01630e8648153ef160c39d92ec2838e90a0168c4), C bindings
    (https://github.com/Manhhao/hoshidicts/tree/fefcb4f4ba975e9bb86f3221cabd7f2705011cde), Hoshi Reader popup
    (https://github.com/Manhhao/Hoshi-Reader/tree/c31c9d0ce376ff83bf6a91d908bf9f8e0fb4947b/Features/Popup), and yomitan-gsm
    (https://github.com/bpwhelan/yomitan-gsm/tree/006dd464a50a468c71093dc8a8311f6110bf1996).

  ## Dependency and Native Boundary

  - Rebase the existing C-bindings work onto Hoshidicts main instead of recreating it. Add only the missing Lookup::lookup ABI, with an owned result handle exposing matched, deinflected,
    deinflection trace, expression, reading, rules, glossary text, dictionary name, and term/definition tags.

  - Do not add C accessors for updater metadata. After import, read revision, isUpdatable, indexUrl, downloadUrl, and sourceLanguage directly from the generated index.json.
  - Add a library-only CMake option so GSM does not build Hoshidicts CLI or benchmarks. Vendor it as a recursively checked-out submodule under the Rust overlay server and statically link it
    into gsm_overlay_server; no extra DLL or resident process.

  - Treat cross-platform path work specifically as Windows Unicode-path correctness: convert UTF-8 C inputs to native std::filesystem::path, stop rebuilding paths with string separators,
    and test Japanese user-profile, ZIP, and dictionary-title paths.

  - Replace the remaining SSH nested-submodule URL with HTTPS. Validate dictionary titles before creating directories, rejecting empty, absolute, traversal, or separator-containing titles.
  - Use compiler capability checks instead of compiler-version assumptions:
      - Keep std::ranges::to whenever __cpp_lib_ranges_to_container is available; use a tiny local fallback only when the standard library feature test fails.
      - Keep #embed when the compile probe succeeds; otherwise generate the equivalent byte header in the build directory.

  - Add hoshidicts to the overlay server’s optional feature leases. Construct and map dictionaries only while leased.
  - Add request-correlated, read-only WebSocket operations:
      - hoshidicts_lookup { requestId, text, scanLength: 10, maxResults: 16 }
      - hoshidicts_lookup_result { requestId, results, error? }
      - hoshidicts_reload { requestId }, which only reloads the already-committed manifest in memory.

  - Add a one-shot import subcommand to the same binary for trusted Electron-main use. Dictionary installation, removal, downloads, and scheduling must not be exposed through the
    unauthenticated overlay WebSocket.

  ## Settings, Dictionaries, and Updates

  - Add one localized Hoshidicts card under GSM’s Experimental settings. It contains the enable toggle, ZIP import, installed dictionary list, removal, “check now,” update schedule, last-
    check time, and status—so scheduling is in the same window as importing.

  - Installed rows show title, revision, language, and update eligibility. All Electron renderer text uses t(...), with keys added to en.json, ja.json, and ukr.json.
  - Store dictionaries as immutable generations under the GSM data directory with an atomically replaced manifest containing stable IDs, order, active generation, schedule, and last-check
    time.

  - Parse each active generation’s index.json as the metadata source. Interpret sourceLanguage as the displayed language; accept ja, accept a missing value as a legacy Japanese dictionary,
    reject an explicit non-Japanese language, and reject imports containing no term entries.

  - Import and update into staging, validate there, atomically switch the manifest, request a native reload, then remove the previous generation. On Windows file-lock failure, defer old-
    generation cleanup until the next startup.

  - Importing another ZIP with the same title atomically replaces that dictionary while preserving its stable ID and position.
  - Support one global schedule: off, daily, weekly, or monthly, defaulting to off; monthly means 30 days. While GSM is running, check once at startup and hourly for whether the interval is
    due.

  - A dictionary is updateable only when its installed metadata has isUpdatable: true, indexUrl, and downloadUrl. Fetch the remote HTTPS index, compare revision strings, validate Japanese
    language, use its downloadUrl, and run the normal staged import. Failures retain the installed generation and record the attempt status without retrying every hour.

  ## Overlay Reader and Mining

  - Scan forward from the Shift-hovered Japanese character within the current OCR block, debounce for 20 ms, and let the newest request win. Reuse the existing overlay positioning, click-
    through, focus, Magpie, topmost, and popup-close behavior.

  - Adapt only the basic Hoshi Reader design: ruby expression/reading header, deinflection and term tags, rounded collapsible dictionary cards, and a right-aligned 28×28 plus-square mining
    button.

  - Theme it with the pinned GSM Yomitan default dark variables and spacing. Render definitions through DOM nodes and textContent; recursively extract only textual structured-content nodes
    and ignore images, styles, custom dictionary CSS, and media.

  - Keep the Yomitan content bridge active but suppress its scanner and popup whenever Hoshidicts is effective. Make all Yomitan changes in the yomitan-gsm/ext source checkout and
    regenerate GSM’s overlay copy with local-build-chrome-overlay.ps1; never edit GSM_Overlay/yomitan/ directly.

  - Add one restricted Yomitan bridge action:
      - hoshidictsMining { operation: "status" } reports Anki readiness and whether the active profile has a usable term format.
      - hoshidictsMining { operation: "mine", requestId, entry, context } accepts the selected expression, reading, matched/deinflected forms, escaped plain glossaries, full OCR-block
        sentence, and match offset.

  - Server-side, select the first valid term card format from the active Yomitan profile. The renderer cannot supply a deck, model, API key, or arbitrary field names.
  - Support only the text markers needed for basic mining: expression, reading/simple furigana, glossary variants, dictionary, tags, sentence, and cloze prefix/body/suffix. Resolve audio,
    screenshot, clipboard, dictionary-media, frequency, pitch, URL, and other unsupported markers to empty values.

  - Reuse the selected format’s deck, model, static field mappings, profile tags, duplicate scope, and AnkiConnect configuration. Append overlay so AnkiBeacon or polling feeds the new note
    into GSM’s existing screenshot/audio enhancement pipeline.

  - Respect duplicate modes new and prevent; return a clear unsupported error for overwrite rather than implementing note searching/editing in V1.
  - Restrict the mutating bridge operation at both layers to the trusted GSM file: overlay document, require event.source === window, validate structured fields and size bounds, and never
    expose Anki credentials.

  - The mine button has only idle, pending/disabled, success, and retryable-error states. Prevent double submission. On success, dispatch exactly one existing gsm-anki-note-added event for
    hide-after-mine UI behavior; backend enhancement still comes only from AnkiBeacon/polling, avoiding double processing.

  ## Tests and Review Shape

  - Hoshidicts tests: C ABI ownership/errors, conjugated lookup and deinflection, raw glossary/tag data, recursive submodule checkout, unsafe titles, Japanese Windows paths, library-only
    builds, and both compiler-feature branches.

  - Rust tests: feature lease lifecycle, lazy loading/unloading, manifest reload, request correlation, malformed requests, result caps, import JSON output, and no lookup initialization
    while disabled.

  - Electron tests: effective experimental gating, import/replace/remove, metadata parsing, Japanese-only validation, atomic rollback, fake-clock schedule intervals, update failures, and
    localized renderer output.

  - Overlay/Yomitan tests: latest-request-wins scanning, safe plain-text rendering, Hoshi layout, first-format selection, marker substitution, sender restrictions, duplicate modes, double-
    click suppression, offline/error recovery, and exactly one success event.

  - Python regression test: an overlay-tagged Hoshidicts seed note follows the existing update_single_card path, receives replay audio/screenshot, and retains its glossary fields.
  - Run .venv/bin/pytest, uv run ruff format GameSentenceMiner tests scripts after Python changes, Hoshidicts CTest, Rust tests, npm run test:ts, npm run build, the Yomitan fork checks/
    rebuild, and Windows/Linux/macOS package builds. Manually smoke-test a Unicode-path ZIP through conjugated lookup, mining, AnkiBeacon, and media enhancement.

  - Keep dependency preparation as isolated Hoshidicts and Yomitan-fork commits. The GSM PR remains four reviewable commits: native/protocol, experimental dictionary manager/updater, Hoshi
    reader overlay, and minimal mining plus attribution. Do not include unrelated refactors or generated-file hand edits.

  ## Assumptions

  - “Optional” means runtime-disabled with no native initialization, CPU, or memory cost; the statically linked bytes remain in packaged builds.
  - No automatic fallback to Yomitan occurs while Hoshidicts is enabled; disabling the experimental toggle restores Yomitan.
  - Scheduled updates require GSM to be running and do not install an OS-level scheduler.
  - V1 mining is one mouse button using one Yomitan term format; everything else remains a follow-up.
	# Small Experimental Hoshidicts Reader and Mining Integration

	## Summary

	- Add Hoshidicts behind enable_experimental_features && enable_hoshidicts; both default off. Changing the Hoshidicts toggle requires restarting the overlay.
	- When enabled, Hoshidicts handles Japanese term scanning and the Hoshi Reader popup. Yomitan scanning/popups are disabled, but its GSM bridge remains loaded for minimal Anki mining. When
	disabled, existing Yomitan behavior is unchanged.

	- V1 includes dictionary import/removal, scheduled updates, Japanese word lookup, and one mouse-driven mining button. Exclude controllers, hotkeys, kanji lookup, nested popup scanning,
	word audio, dictionary media, frequency/pitch export, note browsing, and multiple mining formats.

	- Pin the currently verified bases: Hoshidicts main (https://github.com/Manhhao/hoshidicts/tree/01630e8648153ef160c39d92ec2838e90a0168c4), C bindings
	(https://github.com/Manhhao/hoshidicts/tree/fefcb4f4ba975e9bb86f3221cabd7f2705011cde), Hoshi Reader popup
	(https://github.com/Manhhao/Hoshi-Reader/tree/c31c9d0ce376ff83bf6a91d908bf9f8e0fb4947b/Features/Popup), and yomitan-gsm
	(https://github.com/bpwhelan/yomitan-gsm/tree/006dd464a50a468c71093dc8a8311f6110bf1996).

	## Dependency and Native Boundary

	- Rebase the existing C-bindings work onto Hoshidicts main instead of recreating it. Add only the missing Lookup::lookup ABI, with an owned result handle exposing matched, deinflected,
	deinflection trace, expression, reading, rules, glossary text, dictionary name, and term/definition tags.

	- Do not add C accessors for updater metadata. After import, read revision, isUpdatable, indexUrl, downloadUrl, and sourceLanguage directly from the generated index.json.
	- Add a library-only CMake option so GSM does not build Hoshidicts CLI or benchmarks. Vendor it as a recursively checked-out submodule under the Rust overlay server and statically link it
	into gsm_overlay_server; no extra DLL or resident process.

	- Treat cross-platform path work specifically as Windows Unicode-path correctness: convert UTF-8 C inputs to native std::filesystem::path, stop rebuilding paths with string separators,
	and test Japanese user-profile, ZIP, and dictionary-title paths.

	- Replace the remaining SSH nested-submodule URL with HTTPS. Validate dictionary titles before creating directories, rejecting empty, absolute, traversal, or separator-containing titles.
	- Use compiler capability checks instead of compiler-version assumptions:
	- Keep std::ranges::to whenever __cpp_lib_ranges_to_container is available; use a tiny local fallback only when the standard library feature test fails.
	- Keep #embed when the compile probe succeeds; otherwise generate the equivalent byte header in the build directory.

	- Add hoshidicts to the overlay server’s optional feature leases. Construct and map dictionaries only while leased.
	- Add request-correlated, read-only WebSocket operations:
	- hoshidicts_lookup { requestId, text, scanLength: 10, maxResults: 16 }
	- hoshidicts_lookup_result { requestId, results, error? }
	- hoshidicts_reload { requestId }, which only reloads the already-committed manifest in memory.

	- Add a one-shot import subcommand to the same binary for trusted Electron-main use. Dictionary installation, removal, downloads, and scheduling must not be exposed through the
	unauthenticated overlay WebSocket.

	## Settings, Dictionaries, and Updates

	- Add one localized Hoshidicts card under GSM’s Experimental settings. It contains the enable toggle, ZIP import, installed dictionary list, removal, “check now,” update schedule, last-
	check time, and status—so scheduling is in the same window as importing.

	- Installed rows show title, revision, language, and update eligibility. All Electron renderer text uses t(...), with keys added to en.json, ja.json, and ukr.json.
	- Store dictionaries as immutable generations under the GSM data directory with an atomically replaced manifest containing stable IDs, order, active generation, schedule, and last-check
	time.

	- Parse each active generation’s index.json as the metadata source. Interpret sourceLanguage as the displayed language; accept ja, accept a missing value as a legacy Japanese dictionary,
	reject an explicit non-Japanese language, and reject imports containing no term entries.

	- Import and update into staging, validate there, atomically switch the manifest, request a native reload, then remove the previous generation. On Windows file-lock failure, defer old-
	generation cleanup until the next startup.

	- Importing another ZIP with the same title atomically replaces that dictionary while preserving its stable ID and position.
	- Support one global schedule: off, daily, weekly, or monthly, defaulting to off; monthly means 30 days. While GSM is running, check once at startup and hourly for whether the interval is
	due.

	- A dictionary is updateable only when its installed metadata has isUpdatable: true, indexUrl, and downloadUrl. Fetch the remote HTTPS index, compare revision strings, validate Japanese
	language, use its downloadUrl, and run the normal staged import. Failures retain the installed generation and record the attempt status without retrying every hour.

	## Overlay Reader and Mining

	- Scan forward from the Shift-hovered Japanese character within the current OCR block, debounce for 20 ms, and let the newest request win. Reuse the existing overlay positioning, click-
	through, focus, Magpie, topmost, and popup-close behavior.

	- Adapt only the basic Hoshi Reader design: ruby expression/reading header, deinflection and term tags, rounded collapsible dictionary cards, and a right-aligned 28×28 plus-square mining
	button.

	- Theme it with the pinned GSM Yomitan default dark variables and spacing. Render definitions through DOM nodes and textContent; recursively extract only textual structured-content nodes
	and ignore images, styles, custom dictionary CSS, and media.

	- Keep the Yomitan content bridge active but suppress its scanner and popup whenever Hoshidicts is effective. Make all Yomitan changes in the yomitan-gsm/ext source checkout and
	regenerate GSM’s overlay copy with local-build-chrome-overlay.ps1; never edit GSM_Overlay/yomitan/ directly.

	- Add one restricted Yomitan bridge action:
	- hoshidictsMining { operation: "status" } reports Anki readiness and whether the active profile has a usable term format.
	- hoshidictsMining { operation: "mine", requestId, entry, context } accepts the selected expression, reading, matched/deinflected forms, escaped plain glossaries, full OCR-block
	sentence, and match offset.

	- Server-side, select the first valid term card format from the active Yomitan profile. The renderer cannot supply a deck, model, API key, or arbitrary field names.
	- Support only the text markers needed for basic mining: expression, reading/simple furigana, glossary variants, dictionary, tags, sentence, and cloze prefix/body/suffix. Resolve audio,
	screenshot, clipboard, dictionary-media, frequency, pitch, URL, and other unsupported markers to empty values.

	- Reuse the selected format’s deck, model, static field mappings, profile tags, duplicate scope, and AnkiConnect configuration. Append overlay so AnkiBeacon or polling feeds the new note
	into GSM’s existing screenshot/audio enhancement pipeline.

	- Respect duplicate modes new and prevent; return a clear unsupported error for overwrite rather than implementing note searching/editing in V1.
	- Restrict the mutating bridge operation at both layers to the trusted GSM file: overlay document, require event.source === window, validate structured fields and size bounds, and never
	expose Anki credentials.

	- The mine button has only idle, pending/disabled, success, and retryable-error states. Prevent double submission. On success, dispatch exactly one existing gsm-anki-note-added event for
	hide-after-mine UI behavior; backend enhancement still comes only from AnkiBeacon/polling, avoiding double processing.

	## Tests and Review Shape

	- Hoshidicts tests: C ABI ownership/errors, conjugated lookup and deinflection, raw glossary/tag data, recursive submodule checkout, unsafe titles, Japanese Windows paths, library-only
	builds, and both compiler-feature branches.

	- Rust tests: feature lease lifecycle, lazy loading/unloading, manifest reload, request correlation, malformed requests, result caps, import JSON output, and no lookup initialization
	while disabled.

	- Electron tests: effective experimental gating, import/replace/remove, metadata parsing, Japanese-only validation, atomic rollback, fake-clock schedule intervals, update failures, and
	localized renderer output.

	- Overlay/Yomitan tests: latest-request-wins scanning, safe plain-text rendering, Hoshi layout, first-format selection, marker substitution, sender restrictions, duplicate modes, double-
	click suppression, offline/error recovery, and exactly one success event.

	- Python regression test: an overlay-tagged Hoshidicts seed note follows the existing update_single_card path, receives replay audio/screenshot, and retains its glossary fields.
	- Run .venv/bin/pytest, uv run ruff format GameSentenceMiner tests scripts after Python changes, Hoshidicts CTest, Rust tests, npm run test:ts, npm run build, the Yomitan fork checks/
	rebuild, and Windows/Linux/macOS package builds. Manually smoke-test a Unicode-path ZIP through conjugated lookup, mining, AnkiBeacon, and media enhancement.

	- Keep dependency preparation as isolated Hoshidicts and Yomitan-fork commits. The GSM PR remains four reviewable commits: native/protocol, experimental dictionary manager/updater, Hoshi
	reader overlay, and minimal mining plus attribution. Do not include unrelated refactors or generated-file hand edits.

	## Assumptions

	- “Optional” means runtime-disabled with no native initialization, CPU, or memory cost; the statically linked bytes remain in packaged builds.
	- No automatic fallback to Yomitan occurs while Hoshidicts is enabled; disabling the experimental toggle restores Yomitan.
	- Scheduled updates require GSM to be running and do not install an OS-level scheduler.
	- V1 mining is one mouse button using one Yomitan term format; everything else remains a follow-up.
No results found