String Localization Merging process

Localization key value merging.md

This sort of documents the hopefully one-time process of merging some keys and values for localized strings

There are four things:

1. a source of truth key/value file from Android (./RWApp/Base.lproj/Localizable.strings) called Master below.
2. the existing English "base" localization file (./RWApp/en.lproj/Localizable.strings)
3. non-localized Swift strings (extracted with genstrings)
4. non-localized ObjC strings (extracted with genstrings)

The source of truth file is generated from a different series of tools documented here: https://github.com/ridewithgps/RWGPS-app-common/blob/develop/translation/README.md

The goal here is to rewrite the file ./RWApp/en.lproj/Localizable.strings with a new set of key/value pairs that represents the merged set of all things so that it's keys are taken from the source of truth, which I've called "Master" and it's values are also from "Master". We need to update all the localization call sites to use the right keys.

A script "merge_strings" was used to create the key/value substitutions list and list of unmerged values - values that had no matching key in Master. The --ofile in merge_string specifies an output file to write ed(1) commands to that will be used to translate the old keys to the new keys in both the localization strings file and also source files.

Merge Master into the existing en localizable

Generate the substitutions $ ./merge_strings.py --ofile Localizable.strings.ed --master ./RWApp/Base.lproj/Localizable.strings --merge ./RWApp/en.lproj/Localizable.strings > Localizable.strings.diff

Apply the changes to the target localized strings file. cat Localizable.strings.ed | ed -s ./RWApp/en.lproj/Localizable.strings

Add the new Android-sourced key/values to our target localized strings file. cat ./RWApp/en.lproj/Localizable.strings ./RWApp/Base.lproj/Localizable.strings | uniq > foo mv foo ./RWApp/en.lproj/Localizable.strings

Swift Files

First we need to replace the calls to .localize with RWLocalizedString(). Use Xcode's find/replace to do that using the Regular Expression for find and set your scope to just swift files (there are a couple of Objc files that will otherwise be incorrectly changed).

Filter on Swift Files
Replace ("[^"]+")\.localized
With RWLocalizedString($1, comment: "auto")

Find the Swift Strings that didn't have keys yet

$ find . -name "*.swift" -print0 | xargs -0 genstrings -s RWLocalizedString -o /tmp

$ find . -name "*.swift" -print0 | xargs -0 genstrings -s RWLocalizedString -o /tmp
genstrings: error: bad entry in file ./RWApp/BLEMicroRadar/BLETestViewController.swift (line = 64): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Walkthrough/WalkthroughViewModel.swift (line = 24): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Walkthrough/WalkthroughViewModel.swift (line = 25): Argument is not a literal string.
Key "Update Failed" used with multiple values. Value "Profile not updated." kept. Value "There was a problem updating your Profile, code: \(httpResponse.statusCode). Please try again or contact RWGPS Support if you are still having difficulty." ignored.
Key "Update Failed" used with multiple values. Value "Profile not updated." kept. Value "Update Failed" ignored.
Key "Update Failed" used with multiple values. Value "Profile not updated." kept. Value "Update Failed" ignored.
genstrings: error: bad entry in file ./RWApp/Controllers/Login/RWIntroController.swift (line = 225): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Controllers/Login/RWIntroController.swift (line = 226): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Controllers/Login/RWIntroController.swift (line = 227): Argument is not a literal string.

Convert to plain text

$ iconv -f UTF-16 -t UTF-8 /tmp/Localizable.strings | sed -e '/\*/d' -e '/^$/d' -e 's/;$//' > Localizable.swift.strings

Run the merge with Android keys as base, and generate substitutions and unmerged list

$ ./merge_strings.py --swift --ofile Localizable.swift.strings.sed --master ./RWApp/en.lproj/Localizable.strings --merge Localizable.swift.strings > Localizable.swift.strings.diff

Apply patch to swift files

find . -name *.swift -exec sed -i '' -f Localizable.swift.strings.sed {} \;

ObjC Files

Find the Objc Strings

$ find . -name "*.m" -print0 | xargs -0 genstrings -s RWLocalizedString -o /tmp

genstrings: error: bad entry in file ./RWApp/Filters/RWDataFormatter.m (line = 593): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Filters/RWDataFormatter.m (line = 601): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Upsell/RWUpgradeDialogManager.m (line = 31): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Upsell/RWUpgradeViewController.m (line = 150): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/RWConvert.m (line = 349): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Controllers/Settings/RWIntervalSetupController.m (line = 237): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Controllers/Home/RWDrawerItems.m (line = 83): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Controllers/RWLoggingController.m (line = 2518): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Controllers/RWViewTrouteController/RWViewTrouteController.m (line = 781): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Controllers/Login/RWLoginController.m (line = 109): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Controllers/RWSearchController.m (line = 309): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Views/RWMetricsGridView.m (line = 203): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Views/RWPlotView.m (line = 604): Argument is not a literal string.
genstrings: error: bad entry in file ./RWApp/Categories/UIButton+RWLogin.m (line = 20): Argument is not a literal string.

Convert to plain text

$ iconv -f UTF-16 -t UTF-8 /tmp/Localizable.strings | sed -e '/\*/d' -e '/^$/d' -e 's/;$//' > Localizable.m.strings

Run the merge with Android keys as base, and generate substitutions and unmerged list

$ ./merge_strings.py --objc --ofile Localizable.m.strings.sed --master ./RWApp/en.lproj/Localizable.strings --merge Localizable.m.strings > Localizable.m.strings.diff

Make substitutions in the Objc files

find . -name *.m -exec sed -i '' -f Localizable.m.strings.sed {} \;

Merge all this goop back to the One Ring

Next steps... There are a lot of unmerged strings due to differences between Android and iOS in case, spaces, etc. One could tune the merge_strings script more to do soft matches, but then you have keep the original strings for actual pattern matching in the sed scripts. We might need to merge the results from genstrings from Swift and Objc back into the en strings file too. Fix all the places that had non-literal strings from the above error output listings. Verify all the things.