This article is intended for those who want to get involved in the development of NVDA, an open-source screen reader for Windows, and its derivative product, the Japanese version of NVDA (nvdajp), or for those who want to read and understand its source code.
The following is a quote from the description of version 2012.3 in "NVDA Latest Information".
NVDA can now support Asian character input using IME and text service input methods in all applications, Including:
- Reporting and navigation of candidate lists;
- Reporting and navigation of composition strings; and
- Reporting of reading strings.
The support for "Asian languages" in the official version was carried out from summer to autumn of 2012 with the support of Chinese (China, Taiwan, Hong Kong) users. At that time, I participated in the meetings online as an observer. I tested the Japanese Kana-Kanji conversion (IME) and gave feedback as much as possible.
Because we judged that many of the tasks carried out by the official version would be useful for Japanese IME support, we discarded many of the experimental implementations of Kana-Kanji conversion that we had done in activities such as the "NVDA Japanese Localization Project" in the past, and started development anew. As a result, in the "NVDA Japanese version 2013.1jp" released in May 2013, Japanese input became practically usable.
The Japanese IMEs supported by the Japanese version of NVDA are as follows.
- Microsoft Japanese IME (currently from Windows 7 to 11)
- JustSystems ATOK (verified with ATOK Passport version from 2012 to the present)
In the new Microsoft IME introduced since Windows 10 version 2004 in 2020, compatibility with the traditional NVDA IME support was lost. The Japanese version of NVDA 2022.4jp has implemented support for the new Microsoft IME.
The processing of the candidate list itself of the input method is found in the CandidateList and CandidateItem classes in NVDAObjects/inputComposition.py. These processes define the differences in behavior from regular lists and list items.
The implementation of CandidateItemBehavior is the CandidateItem class in NVDAObjects/behaviors.py. Here, there are methods specific to candidate list items such as getFormattedCandidateName.
There are also processes that depend on the implementation of individual input methods. For example, the Microsoft IME (which was the target of the official version as of 2012) controls behaviors in detail for each version in NVDAObjects/IAccessible/mscandui.py.
These processes do things like:
- Reading out the candidate list items in detail, not just reading them out, such as "Sunday's 'Ni' from 'Nichiyoubi', 'Hon' from 'Hondana'"
- Not reading out redundant information such as being a list, being a list item, etc.
The association of mscandui classes and displayed windows is done in NVDAObjects/IAccessible/__init__.py.
Specifically, mscandui.findExtraOverlayClasses is executed under conditions such as when the windowClassName starts with 'mscandui' in findOverlayClasses().
Support for ATOK is done with NVDAObjects/IAccessible/atok.py, which was uniquely added in the Japanese version of NVDA, and associates it with the ATOK support class when the first 5 characters of windowClassName are ATOK2 or ATOK3, mimicking the aforementioned implementation. In a few years, we will need to add a case for ATOK4...
For the "new Microsoft IME" after 2020, because the API used is UIA, it is implemented as a subclass of NVDAObjects.UIA in appModules/windowsinternal_composableshell_experiences_textinput_inputapp.py. This is an application module.
In the AppModule of this module, the window and implementation are associated.
The Japanese version of NVDA has added a separate file called _jp.py because the official implementation does not work well with Japanese IME.
Some of the processing for composition strings is in NVDAObjects/inputComposition.py. The InputComposition class is a subclass of NVDAObjects.window.Window.
However, to retrieve events and information about input and conversion, APIs other than windows are required. Event handling for keystrokes themselves is not sufficient for input method support.
For example, using the (original) unconverted key function in ATOK allows you to switch input modes like "Half-width alphanumeric" and "Hiragana Romanji". This is read out by NVDA. This process is handleInputConversionModeUpdate.
This code is run from handleIMEConversionModeUpdate in nvdaHelper\remote\ime.cpp.
IMN_SETCONVERSIONMODE is a notification message related to the Windows API's Input Method Manager (IMM). This message is sent to the target window when the conversion mode of the IME changes.
ime.cpp contains the initial settings for NVDA to receive messages and the process to call NVDAHelper's method when received.
TSF (Text Services Framework) is a newer Windows API that manages collaboration between applications and IME, and efficiently supports multilingual input and text processing. TSF has been used by Microsoft IME for a long time, and recently ATOK also supports TSF. TSF-compatible processing is written in tsf.cpp.
For example, the processes that call nvdaControllerInternal_inputCompositionUpdate differ in imm.cpp and tsf.cpp, but they are commonized in NVDAHelper.py.
NVDAHelper.py also contains processes that call the inputComposition code in various situations.
In preparation for reading the source code, I haven't delved into the details. Also, in reality, the 'fine-tuning' for individual IMEs and apps is scattered across various modules.
Nonetheless, the main version of NVDA has implemented these processes efficiently with respect to character input for Asian languages (mainly Chinese and Korean).
As mentioned earlier, the main version also includes processes for Japanese IMEs, such as handleInputConversionModeUpdate.
The differences that NVDA Japanese version has made regarding the 'lack of minute functions' and 'differences in expected behavior' that Japanese screen reader users expect are not significant, but it's a regret that they have not been merged into the main version.
On the other hand, even if they were merged, I'm starting to think it would be a problem if there was no one in Japan to suggest how to fix it or communicate what's broken when it gets broken due to future refactoring or extension.
This article was written by @nishimotz (aka @24motz) as study material for a study session conducted jointly by Shuaruta Inc., which is responsible for the development of the NVDA Japanese version, and the AccessibleToolsLaboratory.
I hope to continue to organize technical information about NVDA from time to time.
These are materials that were helpful when I was doing this work or explaining it to people about 10 years ago.