BiDirectional Text

BiDiSupport.md

This gist will show the support of BiDirectional text in the terminal emulators and console programs. You can read more about the standardization efforts at the dedicated page of FreeDesktop Terminal BiDi working group.

How to test

Logical Order	◀ ◀ ◀ RTL	LTR ▶ ▶ ▶
WHAT IS UNICODE؟ in arabic	in arabic ؟EDOCINU SI TAHW	؟EDOCINU SI TAHW in arabic
	ما هو الترميز الموحد يونيكود؟ in Arabic	ما هو الترميز الموحد يونيكود؟ in Arabic

Supporting BiDirectional text

According to a new standard

Please note, that specification is preliminary yet, and VTE implementation is still work in progress:

• all libvte based terminals - https://gitlab.gnome.org/GNOME/vte/issues/53
  • libvte-based Gnome Terminal - https://bugs.launchpad.net/ubuntu/+source/vte/+bug/263822
  • libvte-based sakura
  • libvte-based xfce4-terminal
  • libvte-based Terminator
  • libvte-based Tilix - written in D. Similar user interface as for Terminator.
  • libvte-based Lilyterm - https://github.com/Tetralet/LilyTerm/
  • libvte-based ROXTerm
  • libvte-based evilvte https://github.com/caleb-/evilvte
  • libvte-based Termit
  • libvte-based Tilda
  • libvte-based tinyterm
  • libvte-based Pantheon Terminal
  • libvte-based lxterminal

Incompatible and incomplete implementations

• mlterm
• konsole
• qterminal
• mintty Cygwin and MSYS/MSYS2 - Windows platform

Supporting BiDirectional text programs, but incompatible

• neatvi - Vi-like editor aligrudi/neatvi#8

Not supporting BiDirectional text

• another libvte-based Termite
• st (from suckless)
• xst fork of st gnotclub/xst#55
• iterm2
• alacritty - written in Rust alacritty/alacritty#663
• kitty - uses OpenGL kovidgoyal/kitty#2109
• cool-retro-term
• Termux - Android platform
• Black Screen - crossplatform, HTML/CSS/JS-based
• Hyper.app - crossplatform, HTML/CSS/JS-based (Electron)
• hterm - HTML/CSS/JS-based (ChromeOS)
• Windows Terminal - Windows platform, standard terminal microsoft/terminal#538
• Tera Term - Windows platform
• ConEmu - Windows platform Maximus5/ConEmu#1665
• FinalTerm - abandoned, iTerm2 borrowing it's ideas and features.
• MacTerm - Mac OS X platform
• Windows 10 bash and cmd.exe console - https://wpdev.uservoice.com/forums/266908-command-prompt-console-windows-subsystem-for-l/suggestions/34937857-bidirectional-text-support

Terminal multiplexers

• tmux - not yet tmux/tmux#1936
• screen - not yet
• pymux - not yet prompt-toolkit/pymux#77
• dvtm - not yet martanne/dvtm#99

Not supporting BiDirectional text programs

• neovim - neovim/neovim#553
• vim - vim/vim#204
• radare2 - https://github.com/radare/radare2/issues/2032
• irssi - irssi/irssi#1133

I've spent the last few months studying and evaluating the only document about BiDi in terminals that I could locate, as well as existing implementations. Unfortunately I have severe problems with all of them.

The most important problem with all the self-claimed BiDi-aware terminals is that they always apply the Unicode BiDi algorithm on the contents, giving no way for applications to switch it off. This way they literally, mathematically provably become a platform on top of which it's impossible to implement proper BiDi-aware text viewing/editing experience. (In most terminals you can disable BiDi in the settings, but it's an unacceptable user experience if people have to do so every time they start up or quit a text editor.)

They also work on lines of text rather than paragraphs, lack support for overall RTL direction, and suffer from various one-off problems as well.

Thus I'd make the bold claim that the terminals mentioned above as supporting BiDi actually don't. Supporting BiDi doesn't mean unconditionally doing it, since it does about as much harm as good. Supporting BiDi means (among plenty of other things) knowing when to do it and when not to.

I came up with a draft specification on how BiDi should be done in terminal emulators, it's at https://terminal-wg.pages.freedesktop.org/bidi/ . Feedback about this spec is welcome over there. I'm making a somewhat bold, but well supported claim that only terminals that implement BiDi according to this spec should be called BiDi-aware.

Recently Mintty did a couple of improvements, partially along the lines of the new specs. I haven't yet had a chance to evaluate what it supports now and what it still doesn't.

Rxvt-unicode with fribidi (Perl) plugin supporting bidirectional text

Just an update - was recently implemented in libvte by @egmontkob: https://terminal-wg.pages.freedesktop.org/bidi/implementations.html#vte

https://github.com/aligrudi/neatvi a fork of VI works great and may be included in the list of programs.

I suppose Termite does not support bidirectional text rendering because their code is not compatible with vanilla libvte.

Please Remove Termite from the list it doesn't support bidirectional text they use a different vte library.
Tested on arch linux.

Removed Termite, thanks!

How may someone contribute to one of these terminals to add BiDi support? if anybody can reference a starting point, may it be how to shape fonts, how to implement BiDi, or even the part of code of these libvte based terminals that makes it able to view BiDi text, that would be appreciated.

Vim actually supports bidirectional text, all you have to do is use a terminal that supports it (or BiCon) and put :set arabicshape! in your .vimrc

WezTerm has premliminary BiDi support in recent releases: wez/wezterm#784 (comment)

Compiled programs support for Bidi in this ArchWiki article.

There's finally some movement on this; Unicode Technical Committee №175 approved a Working Group to be called the Terminal Complex Script Support Working Group (TCSS WG) after Microsoft asked for it in L2/23-107.¹

Microsoft wants to do it in a libvte-incompatible way so I plan to attend every meeting of the WG I can just to discuss…why, it seems they may not have even known about prior art. My document is L2/23-153.²

Footnotes

• ¹ L2/23-107
  Li, Renzhi et al. Proper Complex Script Support in Text Terminals. 2023-04-24.
• ² L2/23-153
  Brennan, Fredrick R. Opposition to and Comment on L2/23-107. 2023-06-30.

@ctrlcctrlv : Re: "Opposition to and Comment on L2/23-107", the BiDi in Terminal Emulators draft proposal is by Egmont Koblinger.

Yes, sorry, my mistake; @XVilka told me this already via email; but the Unicode office is closed on weekends :-) I'll submit a revised (“R”) version to L2 via docsubmit@ next week.

what about wezterm ?

What about it?

This output is wrong:

For «Wikipedia», ויקיפדיה is the correct writing order, not הידפיקיו.

It fails the two major tests of the debu.tanuki.agency test:

It accepts the fribidi'd Hebrew as-is meaning it's on par with kitty (i.e. broken) and mangles the Devanagari worse than Konsole or GNOME Terminal but slightly better than Kitty yet far behind mlterm.

@ctrlcctrlv could you share with me how to run the tests you ran with wezterm?

@wez Sure!

Do:

ssh [email protected]

No password. That's not how I ssh to デブ.狸.agency and is there as a joke against would-be crackers and skiddies, but it's useful for this too! (As that message is just my MOTD.)

So is there a terminal that supports bidi yet? Gnome terminal isn't that good. I've tried to run Neovim inside it, and got the wrong behavior. mlterm seems to be the one with the best behavior out of all the ones that I've tried, but its documentation is lacking and it doesn't look very nice out of the box.

that is the state of the art. mlterm does the best job.

PuTTY (which is more famous in Windows but also runs on linux, known as `pterm`) also has usefully good bidi 👍 - [x] Aligns RTL lines to the right side, processing them as RTL base-direction text. It seems to be using first-strong heuristic to decide line direction. - [x] Treats each line as "paragraph" for purpose of bidi algorithm. - [ ] Long line wrapping creates separate "paragraphs". This is sub-optimal as a long logical line may get broken into paragraphs of different base direction [depending on break points](https://terminal-wg.pages.freedesktop.org/bidi/prior-work/terminals.html#common-problems). AFAICT wrapping is permanent — resizing window to increase width doesn't reflow to join them, they remain broken at original wrap point; resizing to reduce width _hides_ text without reflowing — but it's not lost, resizing back reveals it. This mixed model is a bit weird. - [x] Supports "paired bracket" addition of newer bidi algorithm, so that e.g. `HEBREW f(x) TEXT` looks like `TXET f(x) WERBEH` not `TXET (f(x WERBEH` as it tended to look once upon the time.

perhaps bidi better on win32 but i see dotted circles

fails both deva and hebr

cosmic-term supports bidirectional text and ligatures via cosmic-text: https://fosstodon.org/@soller/111602927867586005

Konsole is actually doing a great job at bidi, in fact it is the only terminal that renders the RTL text from the right edge of the terminal window.

See this screenshot from Konsole displaying English and Persian bible text in one window.

I'm not sure if others can render LTR and RTL text in such fashion.

The settings window

I'd say all terminals I've seen to do bidi do it that way - how else? To check, however, please upload your test file.

The konsole feature "Table characters BiDi mode override" might be interesting but unfortunately it is not really specified (which characters? examples?); maybe I'll submit them an issue.

@wolandark

In any VTE-based terminal, echo '\e[?2501h' to enable auto-direction, and you should get the same behavior.

This comes from terminal-wg's BiDI draft proposal.

@MoSal Neat! I didn't know that. After running the escape sequence, XFCE terminal also behaves the same as Konsole does.

@mintty here is a sample of the text (though what I shared in the screenshot was being fetched from an API).

As mentioned above, doing as MoSal suggested, makes XFCE terminal (which is VTE) behave the same way as Konsole, so I guess my defense of Konsole is relevant, though in my defense there is no need to run any escape sequence in Konsole to get this behavior.

though in my defense there is no need to run any escape sequence in Konsole to get this behavior.

FWIW, LTR instead of AUTO as a default paragraph direction was a design choice which I, as a native speaker of an RTL language, tend to agree with.

The konsole feature "Table characters BiDi mode override" might be interesting

It appears that mintty already has a mode that seems to render table layout like konsole: echo -e "\e[?2501l".