Debian Bug report logs - #404980
mawk: UTF-8 multibyte characters are not handled properly

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Teemu Likonen <tlikonen@ulapland.fi>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: gawk: UTF-8 multibyte characters are not handled properly

Date: Mon, 13 Jun 2005 17:42:48 +0300

Package: gawk
Version: 1:3.1.4-2
Severity: important


gawk does not handle UTF-8 multibyte characters properly. Here's an
example:


$ cat example.txt

A Only_a_singlebyte_character_here_(UTF-8:_41)
Ö A_letter_which_takes_two_bytes_(UTF-8:_c3_96)
€ A_currency_symbol_which_takes_three_bytes_(UTF-8:_e2_82_ac)


$ cat example.txt | awk '{ printf "%-5s%s\n",$1, $2 }'

A    Only_a_singlebyte_character_here_(UTF-8:_41)
Ö   A_letter_which_takes_two_bytes_(UTF-8:_c3_96)
€  A_currency_symbol_which_takes_three_bytes_(UTF-8:_e2_82_ac)


As we can see the format specifier %-5s does not calculate field widths
correctly when string contains multibyte characters. Unfortunately this
makes gawk's field widths mostly unusable with UTF-8 locale.


-- System Information:
Debian Release: 3.1
  APT prefers testing
  APT policy: (850, 'testing'), (800, 'unstable')
Architecture: i386 (i686)
Kernel: Linux 2.6.8-2-k7
Locale: LANG=fi_FI.UTF-8, LC_CTYPE=fi_FI.UTF-8 (charmap=UTF-8)

Versions of packages gawk depends on:
ii  libc6                       2.3.2.ds1-22 GNU C Library: Shared libraries an

-- no debconf information

Message #14 received at 313411@bugs.debian.org (full text, mbox, reply):

From: Teemu Likonen <tlikonen@iki.fi>

To: control@bugs.debian.org, 313411@bugs.debian.org

Subject: This also affects mawk

Date: Fri, 29 Dec 2006 23:55:30 +0200

clone 313411 -1
retitle -1 mawk: UTF-8 multibyte characters are not handled properly
thanks

/usr/bin/mawk seems to be default awk interpreter in Etch. The same
UTF-8 bug is in mawk too.

Message #27 received at 404980@bugs.debian.org (full text, mbox, reply):

From: Francesco Poli <frx@firenze.linux.it>

To: 313411@bugs.debian.org, 404980@bugs.debian.org

Subject: What's the status of this bug?

Date: Thu, 18 Sep 2008 23:49:27 +0200

[Message part 1 (text/plain, inline)]

Hi!

This bug is a show-stopper whenever one wants to set a field width with
the %s format specifier.
It has been reported quite some time ago and I cannot see any activity.
Can this bug at least be forwarded upstream, please?


BTW, I experienced this bug while trying to center lines of text inside
an 80-column container:

$ echo 'hello world' | awk '{ w = int((80 + length())/2); printf "%" w "s\n", $0; }'
                                  hello world
$ echo 'hèllo wörld' | awk '{ w = int((80 + length())/2); printf "%" w "s\n", $0; }'
                                hèllo wörld

Do you happen to know of a command-line tool that can read text lines
from stdin and write them centered to stdout?


Thanks for any help.

-- 
 http://frx.netsons.org/doc/index.html#nanodocs
 The nano-document series is here!
..................................................... Francesco Poli .
 GnuPG key fpr == C979 F34B 27CE 5CD8 DC12  31B5 78F4 279B DD6D FCF4

[Message part 2 (application/pgp-signature, inline)]

Message #34 received at 404980-done@bugs.debian.org (full text, mbox, reply):

From: Jonathan Nieder <jrnieder@gmail.com>

To: Teemu Likonen <tlikonen@ulapland.fi>, control@bugs.debian.org, 404980-done@bugs.debian.org

Cc: Francesco Poli <frx@firenze.linux.it>, Thomas E Dickey <dickey@invisible-island.net>

Subject: Re: mawk: UTF-8 multibyte characters are not handled properly

Date: Mon, 1 Mar 2010 14:09:38 -0600

clone 404980 -1
retitle -1 mawk: Please add a function wrapping wcswidth()
severity -1 wishlist
tags -1 + upstream
thanks

Hi Teemu,

Teemu Likonen wrote:

> $ cat example.txt
> 
> A Only_a_singlebyte_character_here_(UTF-8:_41)
> Ö A_letter_which_takes_two_bytes_(UTF-8:_c3_96)
> € A_currency_symbol_which_takes_three_bytes_(UTF-8:_e2_82_ac)
> 
> 
> $ cat example.txt | awk '{ printf "%-5s%s\n",$1, $2 }'
> 
> A    Only_a_singlebyte_character_here_(UTF-8:_41)
> Ö   A_letter_which_takes_two_bytes_(UTF-8:_c3_96)
> €  A_currency_symbol_which_takes_three_bytes_(UTF-8:_e2_82_ac)
> 
> 
> As we can see the format specifier %-5s does not calculate field widths
> correctly when string contains multibyte characters.

This behavior is shared with C printf, and sadly it is is required.
POSIX is clear about this: the numeric argument to a %s format is a
number of bytes.  See the target of the “File Format Notation” link in
http://www.opengroup.org/onlinepubs/9699919799/utilities/awk.html#tag_20_06_13_10
So closing.

On the other hand, the functionality you are asking for would be very
nice to have in some form.

> Unfortunately this
> makes gawk's field widths mostly unusable with UTF-8 locale.

In C, it is understandable why it was chosen to use number of bytes,
to avoid nonobvious buffer overflow bugs with sprintf().  That problem
does not apply to awk, so maybe it would be possible to convince the
Open Group people to change the behavior (or add a new function)?

See http://unix.org/2008edition/ for the latest standards,
http://austingroupbugs.net/main_page.php to contact the standards
bodies.

Jonathan

Send a report that this bug log contains spam.