Solution for locale and character encoding issues in Japanese games on Steam Deck

ja_JP.UTF-8.md

Problem 1

Cannot access filenames containing non-ASCII characters.

Steam Deck's SteamOS 3 provides only one locale named en_US.utf8 (as configured in /etc/locale.gen) but /etc/locale.conf sets LANG to en_US.UTF-8. They don't match, so the effective locale for programs is C and the character set becomes limited to ASCII.

Set the game's launch options in Steam to LANG=en_US.utf8 %command% to solve this.

To fix this for all applications running in desktop mode (eg. to correctly extract games), create a file named ~/.config/plasma-workspace/env/utf8.sh containing (restart desktop mode to apply):

#!/bin/sh
export LANG=en_US.utf8

Problem 2

Japanese text displays as mojibake or question marks. Some games may even completely fail to load due to script compilation errors (common in visual novels). The Windows ANSI codepage must be set to SHIFT-JIS to fix this.

Wine uses the current system locale (set via LANG) or the LC_ALL environment variable to determine which ANSI codepage to use. This is equivalent to using Locale Emulator on Windows.

Steam Deck's SteamOS 3 doesn't provide a ja_JP.UTF-8 locale so this leaves LC_ALL.

However Steam may override it, so instead set the game's launch options to HOST_LC_ALL=ja_JP.UTF-8 %command% to have Proton set LC_ALL to the desired value.

Problem 3

The simple solutions shown above to problem 1 and 2 are unfortunately mutually exclusive.

ja_JP.UTF-8 is not provided by SteamOS 3 so requesting it makes the effective Unix locale fallback to C (ASCII) again.

Complete solution

This sets the Wine codepage to Japanese and the Unix character set to UTF-8 without requiring root access or unlocking the readonly system partition of the Steam Deck.

1. Extract the archive steam-deck-jp-locale-utf8.tar.zst using Ark into the home directory on the Steam Deck (/home/deck).
2. Double-check that /home/deck/locales/run.sh is executable (right click > properties > permissions).
3. Set the game's launch options to LANG=ja_JP.UTF-8 ~/locales/run.sh %command% to fix locale and encoding issues.

Alternative steps

Follow these steps instead to generate the files manually. A full install of Arch Linux is required (commands may need adjustments to work in other distributions).

1. On an Arch Linux install:
   1. Uncomment ja_JP.UTF-8 in /etc/locale.gen
   2. Run the following commands:

      sudo locale-gen
      
      mkdir locales
      export LOCPATH="$HOME/locales"
      localedef -f UTF-8 -i ja_JP "$LOCPATH/ja_JP.UTF-8"

2. Transfer the locales directory created above into /home/deck on your Steam Deck (SteamOS 3).
3. Create a file named run.sh inside that folder containing:

   #!/bin/bash
   # https://gist.github.com/cfillion/4394c3b8cd051fb45721187053e92296
   newlocpath=$(dirname "$(realpath "$0")")
   script='
     BEGIN { FS=OFS=" '"'--'"' " }
     NF > 1 { for(i = 1; i < NF; ++i) printf "%s", $i OFS; }
     { print "env LOCPATH='"'"'" newlocpath "'"'"' " $NF }
   '
   command="$(awk -v newlocpath="$newlocpath" "$script" <<< "${@@Q}")"
   eval "$command"

4. Make run.sh executable: chmod +x ~/locales/run.sh
5. Set the game's launch options to LANG=ja_JP.UTF-8 ~/locales/run.sh %command% to fix locale and encoding issues.

Sources and tips

• https://github.com/wine-mirror/wine/blob/120ca2ff52d145faa925a41d494035842d47a967/dlls/ntdll/unix/env.c#L804-L805
• https://github.com/wine-mirror/wine/blob/120ca2ff52d145faa925a41d494035842d47a967/dlls/ntdll/unix/env.c#L715-L719
• https://github.com/ValveSoftware/Proton/blob/4221d9ef07cc38209ff93dbbbca9473581a38255/proton#L1081-L1087
• PROTON_LOG=1 WINEDEBUG=+file,+nls,+msgbox %command% to troubleshoot file accesses, locales/character set conversions and message box contents.

steam-deck-jp-locale-utf8.tar.zst

Thanks, I've tried it (first result on Nyaa). Out of the box, without the script, I got this after logging in:

With LANG=ja_JP.UTF-8 ~/locales/run.sh %command%:

So my locale script does appear to be working with this game. However textures are indeed broken with Proton. The logos were corrupted and the title screen was a complete mess:

I saw this line in the Proton log that seemed like it could be related:

02d8:fixme:d3dx:D3DXCreateTextureFromFileInMemoryEx Unrecognized file format, returning failure.

Googled it, and as per this post I tried installing d3dx9 using ProtonTricks. It fixed the logos and title screen:

Hope this helps a little bit. (I tried starting a game, but it failed after character creation due to the server connection getting lost. Not sure if it's another Proton problem or just me not knowing how to connect to the community-run servers.)

Hm weird, the problems you have i never had seen before, the actual problems happen after login (aka when you are on a server)
The game got revived by the community a while back, do you have discord by any chance? Would be easier to communicate

do you have discord by any chance? Would be easier to communicate

Sure: <EDIT: removed>. Won't spend too much time this but sounds like it might be a fun troubleshoot for a little while.

Amazing, thanks alot, friend request sent

ありがとう

Is there a Solution for zh_CN.utf8 ?

Yes, the manual steps can be used to generate the locale files for any language on an Arch Linux install then transfer them to your Steam Deck.

Also SteamOS 3.5 is apparently going to include multiple locales, so hopefully the solutions described in this page won't be needed anymore in the future! source

Thanks a lot. I'll try it !! Christian Fillion ***@***.***> 于2023年7月3日周一 06:15写道：

…

***@***.**** commented on this gist. ------------------------------ Yes, the manual steps <https://gist.github.com/cfillion/4394c3b8cd051fb45721187053e92296#alternative-steps> can be used to generate the locale files for any language on an Arch Linux install then transfer them to your Steam Deck. Also SteamOS 3.5 is apparently going to include multiple locales, so hopefully the solutions described in this page won't be needed anymore in the future! source <ValveSoftware/SteamOS#794 (comment)> — Reply to this email directly, view it on GitHub <https://gist.github.com/cfillion/4394c3b8cd051fb45721187053e92296#gistcomment-4616968> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHSYQGIS23T6N2R7A2NRCM3XOHXIJBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTCNZZGUYTGMBYU52HE2LHM5SXFJTDOJSWC5DF> . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

agree, not working

Which game (and what is the problem with it)? So far it has been working with all Japanese visual novels I've tried.

Thanks you verry much!! 😀
On my Steamdeck each Unity games has this problem.
And now all is working so well to get my prefered language (maternal)
I've just put on steamlauch options: HOST_LC_ALL=fr_FR.UTF-8
And before that off course a little clean on locale.gen follow by sudo locale-gen

I will forward your operationnal mode if I see somes posts on that issue

Hello, as mentioned in an earlier comment SteamOS 3.5 will ship a bunch of locales in addition to en_US.UTF-8 (here is the list). You can try installing it from the preview channel. If you do and notice any issues please let me know.

Noticing that the steam menu in some games also garbles japanese characters e.g. when copy-pasting.

Steam Deck's SteamOS 3 provides only one locale named en_US.utf8 (as configured in /etc/locale.gen) but /etc/locale.conf sets LANG to en_US.UTF-8. They don't match, so the effective locale for programs is C and the character set becomes limited to ASCII.

I'm curious about why this arose to begin with; glibc should be normalizing UTF-8 references to utf8 making them functionally equivalent. Not sure if that could lead to my issue but it does sound like OP's first problem/fix suggests there may be a bug? See also ValveSoftware/SteamOS#1178

Steam Deck's SteamOS 3 provides only one locale named en_US.utf8 (as configured in /etc/locale.gen) but /etc/locale.conf sets LANG to en_US.UTF-8.

This might have been true in the past, but it's not true anymore. Multiple locales are configured in /etc/locale.gen and they all have the UTF-8 suffix, instead of the mentioned utf8.

This might have been true in the past, but it's not true anymore. Multiple locales are configured in /etc/locale.gen and they all have the UTF-8 suffix, instead of the mentioned utf8.

Indeed, this only applies to SteamOS prior to 3.5.