Using UTF-8 console on Linux

Post by Rui Santos
In my "quest", I'd like to use UTF-8 in all consoles. I almost did
it, except for a little detail: I cannot use any kind of accents with
any of my letters. Here is what I do
loadkeys /usr/share/kbd/compose.winkeys
loadkeys /usr/share/kbd/compose.latin1.add

Composing characters don't work with utf-8, but a patch exsts (not tested by
me). It was mentioned last week on kernel list:
http://marc.info/?l=linux-kernel&m=118531371404736&w=2

Standalone accents should work, but you may need to pass a -u/--unicode
option to loadkeys. (Not mentioned in loadkeys manual, but printed by
loadkeys --help.)

I'm using something like this; this should work work you too:
echo -en '\033%G'
kbd_mode -u
setfont lat2-16 -m 8859-2
loadkeys -u hu
If you press the keys, some accented vowels should appear.

Furthermore, brand new in 2.6.22: there's a file called default_utf8 (or
something similar) somewhere under /proc, echo 1 to it and your newly
allocated or reseted terminals will automatically be UTF-8 so you won't need
that \033%G.

bye,

Egmont

Simos Xenitellis

2007-07-31 16:22:09 UTC

Στις 31-07-2007, ημέρα Τρι, και ώρα 17:56 +0200, ο/η Egmont Koblinger

Composing characters don't work with utf-8, but a patch exsts (not tested by
http://marc.info/?l=linux-kernel&m=118531371404736&w=2

There has been a discussion on this at this list, a summary of which is
at
http://www.mail-archive.com/linux-***@nl.linux.org/msg04900.html

Hope this helps,
Simos

Post by Egmont Koblinger
Standalone accents should work, but you may need to pass a -u/--unicode
option to loadkeys. (Not mentioned in loadkeys manual, but printed by
loadkeys --help.)
echo -en '\033%G'
kbd_mode -u
setfont lat2-16 -m 8859-2
loadkeys -u hu
If you press the keys, some accented vowels should appear.
Furthermore, brand new in 2.6.22: there's a file called default_utf8 (or
something similar) somewhere under /proc, echo 1 to it and your newly
allocated or reseted terminals will automatically be UTF-8 so you won't need
that \033%G.
bye,
Egmont
--
Linux-UTF8: i18n of Linux on all levels
Archive: http://mail.nl.linux.org/linux-utf8/

Rui Santos

2007-07-31 17:27:29 UTC

Post by Simos Xenitellis
Στις 31-07-2007, ημέρα Τρι, και ώρα 17:56 +0200, ο/η Egmont Koblinger

Composing characters don't work with utf-8, but a patch exsts (not tested by
http://marc.info/?l=linux-kernel&m=118531371404736&w=2

There has been a discussion on this at this list, a summary of which is
at

Thank you for the link

Post by Simos Xenitellis
Hope this helps,
Simos

Rui

Post by Simos Xenitellis

--
Linux-UTF8: i18n of Linux on all levels
Archive: http://mail.nl.linux.org/linux-utf8/

--
Cumprimentos

*Rui Santos*
Dep. Testes

*GrupoPIE Portugal, S.A.*
Tel: +351 252 290 600
Fax: +351 252 290 601

Email: ***@grupopie.com <mailto:***@grupopie.com>
Web: www.grupopie.com <http://www.grupopie.com/>

/WinREST /EVERYWHERE

Rui Santos

2007-07-31 17:26:47 UTC

Composing characters don't work with utf-8, but a patch exsts (not tested by
http://marc.info/?l=linux-kernel&m=118531371404736&w=2

Could you please point me to where the patch is located ?

No change in my setup. I have a few questions though:
- That has to be non-unicode? Or it doesn't matters ?
- I hope it should also work with 8859-15
- I suppose that hu from loadkeys is your keymap ( pt-latin1/9 for me ) ?
- Just to be on the safe side: I do not have locale installed. Is it
needed ?

Post by Egmont Koblinger
Furthermore, brand new in 2.6.22: there's a file called default_utf8 (or
something similar) somewhere under /proc, echo 1 to it and your newly
allocated or reseted terminals will automatically be UTF-8 so you won't need
that \033%G.

I assume you meant /sys/module/vt/parameters/default_utf8
I didn't knew that. Anyway, I tested it and it didn't work. I always
need the echo.

Post by Egmont Koblinger
bye,

Thanks,

Post by Egmont Koblinger
Egmont

Rui

Post by Egmont Koblinger
--
Linux-UTF8: i18n of Linux on all levels
Archive: http://mail.nl.linux.org/linux-utf8/

Egmont Koblinger

2007-08-01 10:48:59 UTC

Post by Egmont Koblinger
http://marc.info/?l=linux-kernel&m=118531371404736&w=2

Could you please point me to where the patch is located ?

It's just one click from the URL I've mentioned above, I hope you can find
it. But as said I haven't tried this patch. By the way, that page gives a
link to a patch to the kbd package, and this link is no longer valid. I
guess you should contact the sender of that mail (Jirka Bohac) and ask him.

Post by Egmont Koblinger
echo -en '\033%G'
kbd_mode -u
setfont lat2-16 -m 8859-2
loadkeys -u hu

- That has to be non-unicode? Or it doesn't matters ?

^^^^
What do you refer to by "That"? I'm using an UTF-8 only setup.

Post by Rui Santos
- I hope it should also work with 8859-15

Sure. Just make sure the font you load contains a Unicode map, or you supply
one. The "-m" option is not necessary, but makes gpm behave better (but yet
another kernel patch is needed to make gpm work with unicode anyway. We'll
return to this if you're ready with the other steps and you need gpm.)

Post by Rui Santos
- I suppose that hu from loadkeys is your keymap ( pt-latin1/9 for me ) ?

Yes. Does pt-latin1/9 contain direct accented letters, or composing/dead
characters? In other words: do you expect a single keypress (maybe with a
modifier like Shift or AltGr) to produce an accented letter, or would you
need more keypresses?

Just for curiosity: please try the exact commands I gave you above, then run
the command "cat" and press the "main" letters on your keyboard. You should
see something like this:
0123456789öüó
qwertzuiopőú
asdfghjkléáű
íyxcvbnm,.-
Do you get these accented vowels there?

Do you see accented letters if they are printed by an application, not from
keyboard? E.g. try this:
echo -e 'x\0303\0241y\0303\0251z'
The expected output is:
xáyéz
("a" and "e" have an accent on them). Is it okay?

Post by Rui Santos
- Just to be on the safe side: I do not have locale installed. Is it
needed ?

If you launch the command "cat" (to get into cooked terminal mode) and then
just type letters, then no locale is needed. The kbd_mode, setfont, loadkeys
etc. command don't make use of locales.

However, if you run any more complicated utilities (including the shell, any
readline or ncurses apps) then most likely they need to have an UTF-8 locale
set up, otherwise they'll mishandle non-ascii letters. So you'd better have
at least one UTF-8 locale (either C.UTF-8 or en_US.UTF-8) and set (export)
your LANG variable to this value before starting that application. (Altering
LANG within a shell probably doesn't change the behavior of that particular
shell, only its children.)

And finally, if you're all done with this and accented letters all work, try
the following: launch "cat", press four accented letters, press backspace
twice (now you see two accented letters) and press Enter. Probably you'll
see three accented letters being echoed back in the next line, which is
wrong. In this case, the command "stty iutf8" is the solution.

bye,
Egmont

Rui Santos

2007-08-01 11:32:37 UTC

Post by Egmont Koblinger
http://marc.info/?l=linux-kernel&m=118531371404736&w=2

Could you please point me to where the patch is located ?

That thread is from early 2005... Does it still applies to the current
kernel ?

Post by Egmont Koblinger
echo -en '\033%G'
kbd_mode -u
setfont lat2-16 -m 8859-2
loadkeys -u hu

- That has to be non-unicode? Or it doesn't matters ?

^^^^
What do you refer to by "That"? I'm using an UTF-8 only setup.

You use lat2-16 fonf. That means that it does not have an utf8 mapping.
Usually the ones that end with 'u', like lat9u-16.psfu.gz already have
it. Am I correct ?

Post by Rui Santos
- I hope it should also work with 8859-15

Post by Rui Santos
- I suppose that hu from loadkeys is your keymap ( pt-latin1/9 for me ) ?

I retrieved it from my own system. I have pt-latin1 and
compose.latin1.add with all includes.

Post by Egmont Koblinger
Just for curiosity: please try the exact commands I gave you above, then run
the command "cat" and press the "main" letters on your keyboard. You should
0123456789öüó
qwertzuiopőú
asdfghjkléáű
íyxcvbnm,.-
Do you get these accented vowels there?

No, I do not get any accented characters.

Post by Egmont Koblinger
Do you see accented letters if they are printed by an application, not from
echo -e 'x\0303\0241y\0303\0251z'
xáyéz
("a" and "e" have an accent on them). Is it okay?

The front is working with accented characters. I have a utf8 encode
text-file on the filesystem and the text is outputed correctly.
That 'echo' also works as it should.

Post by Rui Santos
- Just to be on the safe side: I do not have locale installed. Is it
needed ?

If you launch the command "cat" (to get into cooked terminal mode) and then
just type letters, then no locale is needed. The kbd_mode, setfont, loadkeys
etc. command don't make use of locales.
However, if you run any more complicated utilities (including the shell, any
readline or ncurses apps) then most likely they need to have an UTF-8 locale
set up, otherwise they'll mishandle non-ascii letters. So you'd better have
at least one UTF-8 locale (either C.UTF-8 or en_US.UTF-8) and set (export)
your LANG variable to this value before starting that application. (Altering
LANG within a shell probably doesn't change the behavior of that particular
shell, only its children.)
And finally, if you're all done with this and accented letters all work, try
the following: launch "cat", press four accented letters, press backspace
twice (now you see two accented letters) and press Enter. Probably you'll
see three accented letters being echoed back in the next line, which is
wrong. In this case, the command "stty iutf8" is the solution.
bye,
Egmont
--
Linux-UTF8: i18n of Linux on all levels
Archive: http://mail.nl.linux.org/linux-utf8/

Egmont Koblinger

2007-08-01 13:00:09 UTC

Post by Rui Santos
That thread is from early 2005... Does it still applies to the current
kernel ?

I don't know.

Post by Rui Santos
You use lat2-16 fonf. That means that it does not have an utf8 mapping.
Usually the ones that end with 'u', like lat9u-16.psfu.gz already have
it. Am I correct ?

setfont automatically adds the extension. This file is actially
lat2-16.psfu, so it does contain Unicode mapping.

Post by Rui Santos
I retrieved it from my own system. I have pt-latin1 and
compose.latin1.add with all includes.

Okay, but do you know what keyboard layout these files are supposed to
implement? I bet you do have expectations like "this or that key should
produce this or that letter". Or rather in the reverse way: "I need these
accented letters, and they should be accessible by these particular keys or
key sequences...". So I'm still curious: do you really plan to use dead keys
or combining keys, or just simple keys for accented letters?

No, I do not get any accented characters.

Now this is strange, and at this moment I have no idea where to look at.

Did the commands all succeed? Are you root? Are you executing these from the
real console (not from mc, screen, su...)? Please confirm that you really
typed these letters to the command "cat".

What happens when you hit those keys that are supposed to print accented
letters? Is simply this keystroke ignored? Or a question mark printed? Or
what else?

--
Egmont

Rui Santos

2007-08-01 14:13:17 UTC

Post by Rui Santos
That thread is from early 2005... Does it still applies to the current
kernel ?

I don't know.

Post by Rui Santos
You use lat2-16 fonf. That means that it does not have an utf8 mapping.
Usually the ones that end with 'u', like lat9u-16.psfu.gz already have
it. Am I correct ?

setfont automatically adds the extension. This file is actially
lat2-16.psfu, so it does contain Unicode mapping.

Post by Rui Santos
I retrieved it from my own system. I have pt-latin1 and
compose.latin1.add with all includes.

Okay, but do you know what keyboard layout these files are supposed to
implement?

Yes.

Post by Egmont Koblinger
I bet you do have expectations like "this or that key should
produce this or that letter". Or rather in the reverse way: "I need these
accented letters, and they should be accessible by these particular keys or
key sequences...". So I'm still curious: do you really plan to use dead keys
or combining keys, or just simple keys for accented letters?

I was planning to use dead keys in combination with the simple keys to
produce the accented key I want. If possible, all of them.

No, I do not get any accented characters.

Sorry. It does work. I was issuing the command on bash, and not on cat.
So it works :)

Post by Egmont Koblinger
What happens when you hit those keys that are supposed to print accented
letters? Is simply this keystroke ignored? Or a question mark printed? Or
what else?

Rui Santos

2007-08-01 14:27:03 UTC

Post by Rui Santos
That thread is from early 2005... Does it still applies to the current
kernel ?

I don't know.

Post by Rui Santos
You use lat2-16 fonf. That means that it does not have an utf8 mapping.
Usually the ones that end with 'u', like lat9u-16.psfu.gz already have
it. Am I correct ?

setfont automatically adds the extension. This file is actially
lat2-16.psfu, so it does contain Unicode mapping.

Post by Rui Santos
I retrieved it from my own system. I have pt-latin1 and
compose.latin1.add with all includes.

Okay, but do you know what keyboard layout these files are supposed to
implement?

Yes.

I was planning to use dead keys in combination with the simple keys to
produce the accented key I want. If possible, all of them.

No, I do not get any accented characters.

Sorry. It does work. I was issuing the command on bash, and not on cat.
So it works :)

Also I tested with the cat command with my current setup and also works.
All characters are typed correctly, even the accented characters
produced with dead keys.
Why is in not working on bash ?

I also tested with stty iutf8

Post by Egmont Koblinger
What happens when you hit those keys that are supposed to print accented
letters? Is simply this keystroke ignored? Or a question mark printed? Or
what else?

Egmont Koblinger

2007-08-01 15:31:04 UTC

Post by Rui Santos
Sorry. It does work. I was issuing the command on bash, and not on cat.
So it works :)

Okay, good to hear it, so we're not completely lost :) So it's simply an
application error, we need to fix bash.

To make things work in bash, you'll need some of these. I don't know which
ones are exactly needed, so the safest is to go for sure and do all of
these:

- A sufficiently recent readline and bash: at least 5.1/3.1, but rather
5.2/3.2, plus preferably the patches from
ftp://ftp.gnu.org/gnu/{readline,bash}

- A ~/.inputrc file or $INPUTRC pointing to a file that contains this:
set meta-flag On
set convert-meta Off
set input-meta On
set output-meta On
IIRC bash 3.2 automatically chekcs /etc/inputrc too, but I'm not sure.

- Have at least one UTF-8 locale, e.g. en_US.UTF-8

- Have your LANG variable point to that UTF-8 locale, e.g. execute
"export LANG=en_US.UTF-8" in your startup stript prior to starting the
interactive bash in which you'll be editing command line with accents.

--
Egmont

Lefteris Dimitroulakis

2007-08-02 19:04:11 UTC

Post by Egmont Koblinger
set meta-flag On
set convert-meta Off
set input-meta On
set output-meta On

According to bash docs
meta-flag is synonymous with input-meta,
so we only need to set either of them.

regards

Ken Moffat

2007-08-03 21:31:31 UTC

Composing characters don't work with utf-8, but a patch exsts (not tested by
http://marc.info/?l=linux-kernel&m=118531371404736&w=2

I'm sorry to be pedantic on my first post to this list, but it's
"only" the non-latin-1 composing and dead keys which don't work. Not
trying to minimise the scope of the problem, I'd love to be able to
type in more languages at the console.

I've put an example British keymap at
http://homepage.ntlworld.com/zarniwhoop/uk-utf.map - if a dead key
in the standard xorg layout can be made to work on the console, it
uses it (so, for example, dead acute works on a,e,i,o,u only), and
there are some other variations.

ĸen

--
das eine Mal als Tragödie, das andere Mal als Farce

Simos Xenitellis

2007-08-03 23:16:18 UTC

Post by Ken Moffat

Composing characters don't work with utf-8, but a patch exsts (not tested by
http://marc.info/?l=linux-kernel&m=118531371404736&w=2

I'm sorry to be pedantic on my first post to this list, but it's
"only" the non-latin-1 composing and dead keys which don't work. Not
trying to minimise the scope of the problem, I'd love to be able to
type in more languages at the console.
I've put an example British keymap at
http://homepage.ntlworld.com/zarniwhoop/uk-utf.map - if a dead key
in the standard xorg layout can be made to work on the console, it
uses it (so, for example, dead acute works on a,e,i,o,u only), and
there are some other variations.

That should be
http://homepage.ntlworld.com/zarniwhoop/console/uk-utf.map

The keymap references a couple of files, unicode.map and compose.latin1.
Are they part of a distribution of console-data (using Ubuntu)?

During the last attempt to get compose support in the kernel at LKML,
the response was that the kernel console support was supposed to provide
facilities for emergency usage only (serial debugger, access to fsck,
etc). Therefore, the patch described at
http://www.advogato.org/person/simosx/diary.html?start=2
although it solved the problem, got rejected. (Also the patch was
somewhat of a hack in the way it solved the problem).

Simos

Ken Moffat

2007-08-04 03:18:25 UTC