Backwards Unicode and Shell Support

Flipping strings

So, I couldn’t find any obvious tools for reversing text on the command line, but fortunately if you set your “hackishness” filter to allow solving a problem with BASH, you find yourself not too far away from just about any crazy idea that enters your head. With a portion of my mind always worrying about someone typing in some bizarre combination of characters and breaking one of my websites, I was soon inspired to think about Right To Left (RTL) languages. I can’t be the only person who’s seen a chat client get confused (or at least make the most of a difficult situation) while trying to render a line which contains a mix of Arabic and ASCII or Hebrew and Hangul. All I needed, I thought, was a single character from an RTL language, and to put the rest of the string after that. I even dared hope that there would be some bug in some terminal emulator where removing the RTL character after the flip would keep the string reversed.

Somewhere along the line I remembered the amusing upside-down text trick, but I also learnt about the non-printing Unicode character U+202e that makes text go backwards, which I suppose wasn’t difficult to Google upon accidentally. Everything that needs to be said about it has probably been said already, except that Google does not filter for this character in search results, leading to an amusing result (if the Google dance hasn’t interfered with that page).

Unicode Support

The first problem in using this trick to solve the problem was getting a console program to output the desired character. echo does support specifying characters with octal escape sequences, but only within the ASCII range, which meant the next command to try was printf. “Obviously”, though, there are two printfs, and the default one is the one that doesn’t work. To understand this a bit better, try which printf, man printf, help printf and remember that BASH has built-in commands. With this understood, it was relatively easy to write

/usr/bin/printf "\u202E foo"

and expect to see oof as the result. It’s here that things start to get a bit murky though.

My first attempt was running this code in Konsole, but it seemed to just ignore the Unicode character. This was disappointing, as it managed to print Arabic and Hebrew text correctly, albeit in a left to right manner. For the purposes of proving the abilities of BASH, I thought it was fair to use any terminal emulator software at my disposal, so I installed mlterm which prides itself on supporting “various character sets and encodings in the world and complex characters such as double width for East Asian, combining for Thai, Vietnamese, and so on, and bi-direction for Arabic and Hebrew”. I have to admit, when I ran the above command I did get oof returned, but with the rest of the line (indicating my username, host and path) reversed too.

This was good enough to get started, though, and with help from another colleague, I came up with a version which relied on subshells to capture stdin. Like semicolons, however, I consider subshells to break the principles of a one-liner, so after some more research I discovered:

echo "ysae" | xargs -I foo /usr/bin/printf "\u202Efoo\n"

which produced a left-aligned line containing simply easy and a nasty hollow square at the beginning representing the non-printing character. The BASH trick used here is the -I argument to xargs which does string replacement in the arguments to the command using the piped text, rather than merely appending the text as further arguments.

Playing with mlterm, and testing its much vaunted support for Unicode, I tried pasting some strings into it and seeing how it would react. Disappointingly, I found that when a Hebrew string was pasted, it appeared with an unnecessary @ sign at the end and prompted a console beep due to an error. The string could be echoed correctly, however, and even appeared right-justified, but when I tried the same trick on an Arabic string, which worked perfectly well in Konsole, I got a series of hollow squares followed by the @ sign and beep. Pasting in some Greek produced even worse results, however, with the text appearing invisibly, not even moving the cursor along like a space would. I could only confirm the characters were there by echoing the string and running it through wc.

Perhaps some of these bugs could be fixed by fighting with the myriad of font settings, but it seems a shame that a program which prides itself on multi-lingual support behaves worse than one which lists multi-lingual support as just another great feature. I did try to fix the problem, even though I had no particular interest in Greek and Arabic text. I installed fonts and an extra configuration program, and specified options on the command line before invoking the terminal, but nothing worked. All that happened was I got more confused by all the permutations. Did I need to “Process received strings via Unicode”? Hebrew seemed to be working, so I was sure I was sending the strings correctly. Was my “XIM locale” correct? Again, I had to assume it was.

Conclusion

Unicode support is difficult for all sorts of applications, and I accept that; but when dealing with a console-based program, it feels like the straw that breaks that camel’s back and highlights how badly the whole console software stack has kept up with developments in technology compared to modern complementary environments. More on this next time…

Did I prove that BASH could do whatever Perl could do, for people who don’t mind ugly hacks? That depends, are you running the script 10 times in a row?