Debian Bug report logs - #319952
bsdmainutils: [col] doesn't handle UTF-8

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Bas Zoetekouw <bas@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: locales not take into account when output is redirected

Date: Tue, 26 Jul 2005 00:09:04 +0200

Package: man-db
Version: 2.4.3-1
Severity: normal

It looks like man outputs only ASCII7 text if the output is redirected
to a file or a pipe.  This even happens if the locale is explicitly
specified on the command line.

For example, set your locale to de_DE.UTF-8, and compare the outputs
of "man chsh", "man chsh|less" and "man -L de_DE.UTF-8|less".  The
first one outputs nice utf8 German umlaut chars, but these chars are
simply dropped in the latter two cases.

This breaks programs such as pinfo, which uses man as a backend for
displaying man pages.

I would suggest that man just looks at the current locale (or adding a
command line param that forces such behaviour).  Anyone who
wants a clean ASCII7 text can simply set the correct locale, or
specify -7 on the command line.

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.9-ac6
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)

Versions of packages man-db depends on:
ii  bsdmainutils                6.1.2        collection of more utilities from 
ii  debconf [debconf-2.0]       1.4.52       Debian configuration management sy
ii  dpkg                        1.13.10      Package maintenance system for Deb
ii  groff-base                  1.18.1.1-8   GNU troff text-formatting system (
ii  libc6                       2.3.2.ds1-22 GNU C Library: Shared libraries an
ii  libgdbm3                    1.8.3-2      GNU dbm database routines (runtime

man-db recommends no packages.

-- debconf information:
  man-db/build-database: true
* man-db/rebuild-database: true
* man-db/install-setuid: true

Message #10 received at 319952@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: Bas Zoetekouw <bas@debian.org>, 319952@bugs.debian.org

Subject: Re: Bug#319952: locales not take into account when output is redirected

Date: Tue, 26 Jul 2005 00:03:31 +0100

On Tue, Jul 26, 2005 at 12:09:04AM +0200, Bas Zoetekouw wrote:
> Package: man-db
> Version: 2.4.3-1
> Severity: normal
> 
> It looks like man outputs only ASCII7 text if the output is redirected
> to a file or a pipe.  This even happens if the locale is explicitly
> specified on the command line.
> 
> For example, set your locale to de_DE.UTF-8, and compare the outputs
> of "man chsh", "man chsh|less" and "man -L de_DE.UTF-8|less".  The
> first one outputs nice utf8 German umlaut chars, but these chars are
> simply dropped in the latter two cases.
> 
> This breaks programs such as pinfo, which uses man as a backend for
> displaying man pages.
> 
> I would suggest that man just looks at the current locale (or adding a
> command line param that forces such behaviour).  Anyone who
> wants a clean ASCII7 text can simply set the correct locale, or
> specify -7 on the command line.

Actually, all locale support is still present. The problem is that man
pages being output to a file or a pipe are now filtered through 'col
-b', the intent of which was simply:

        o When stdout is not a terminal, man pages will be formatted in
          plain text without the use of backspace or ANSI formatting
          characters.

Unfortunately, and unintentionally, this clobbers characters that aren't
printable ASCII. Bah. 'col -b -p -x' is much better, but exhibits some
minor corruption ("ÜBERSICHT" at the start of a line has some garbage
before it, probably because col can't handle the overstruck "Ü").

Changing the arguments to col seems to be obviously the right thing to
do, but beyond that I'm not sure what to do about this. I could reassign
to bsdmainutils in the hope that col can be changed to deal with UTF-8
in UTF-8 locales, or I could just disable the col command in the
pipeline for multibyte locales. The latter would be a shame given that
it means everyone has to go back to putting explicit 'col -b' in
makefiles and things again, and the breakage pending a col fix is
confined to just a few places ...

Cheers,

-- 
Colin Watson                                       [cjwatson@debian.org]

Message #15 received at 319952@bugs.debian.org (full text, mbox, reply):

From: Bas Zoetekouw <bas@debian.org>

To: Colin Watson <cjwatson@debian.org>

Cc: 319952@bugs.debian.org, control@bugs.debian.org

Subject: Re: Bug#319952: locales not take into account when output is redirected

Date: Tue, 26 Jul 2005 16:16:13 +0200

retitle 319952 UTF8 chars butchered by piping man output through col
thanks

Hi Colin!

You wrote:

> Actually, all locale support is still present. The problem is that man
> pages being output to a file or a pipe are now filtered through 'col
> -b', the intent of which was simply:
>         o When stdout is not a terminal, man pages will be formatted in
>           plain text without the use of backspace or ANSI formatting
>           characters.
> Unfortunately, and unintentionally, this clobbers characters that aren't
> printable ASCII. Bah. 'col -b -p -x' is much better, but exhibits some
> minor corruption ("ÜBERSICHT" at the start of a line has some garbage
> before it, probably because col can't handle the overstruck "Ü").

Ah, I see.  

> Changing the arguments to col seems to be obviously the right thing to
> do, but beyond that I'm not sure what to do about this. I could reassign
> to bsdmainutils in the hope that col can be changed to deal with UTF-8
> in UTF-8 locales, or I could just disable the col command in the
> pipeline for multibyte locales. The latter would be a shame given that
> it means everyone has to go back to putting explicit 'col -b' in
> makefiles and things again, and the breakage pending a col fix is
> confined to just a few places ...

I think the best way to solve this would indeed be to fix col.  Or maybe
it can be replaced by a sed script that rips out the ANSI codes?  

-- 
Kind regards,
+--------------------------------------------------------------------+
| Bas Zoetekouw              | GPG key: 0644fab7                     |
|----------------------------| Fingerprint: c1f5 f24c d514 3fec 8bf6 |
| bas@o2w.nl, bas@debian.org |              a2b1 2bae e41f 0644 fab7 |
+--------------------------------------------------------------------+ 

Message #22 received at 319952@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: Bas Zoetekouw <bas@debian.org>, 319952@bugs.debian.org

Cc: bsdmainutils@packages.debian.org, control@bugs.debian.org

Subject: Re: Bug#319952: locales not take into account when output is redirected

Date: Tue, 30 Aug 2005 14:06:26 +0100

reassign 319952 bsdmainutils
retitle 319952 col doesn't handle UTF-8
thanks

On Tue, Jul 26, 2005 at 04:16:13PM +0200, Bas Zoetekouw wrote:
> You wrote:
> > Actually, all locale support is still present. The problem is that man
> > pages being output to a file or a pipe are now filtered through 'col
> > -b', the intent of which was simply:
> >         o When stdout is not a terminal, man pages will be formatted in
> >           plain text without the use of backspace or ANSI formatting
> >           characters.
> > Unfortunately, and unintentionally, this clobbers characters that aren't
> > printable ASCII. Bah. 'col -b -p -x' is much better, but exhibits some
> > minor corruption ("ÜBERSICHT" at the start of a line has some garbage
> > before it, probably because col can't handle the overstruck "Ü").
> 
> Ah, I see.  
> 
> > Changing the arguments to col seems to be obviously the right thing to
> > do, but beyond that I'm not sure what to do about this. I could reassign
> > to bsdmainutils in the hope that col can be changed to deal with UTF-8
> > in UTF-8 locales, or I could just disable the col command in the
> > pipeline for multibyte locales. The latter would be a shame given that
> > it means everyone has to go back to putting explicit 'col -b' in
> > makefiles and things again, and the breakage pending a col fix is
> > confined to just a few places ...
> 
> I think the best way to solve this would indeed be to fix col.  Or maybe
> it can be replaced by a sed script that rips out the ANSI codes?  

man-db (2.4.3-2) unstable; urgency=low

  * Use 'col -b -p -x' rather than just 'col -b' when stdout is not a
    terminal. Partly fixes #319952, but col still needs to be fixed to cope
    with UTF-8 input.
  * Use www-browser as default HTML pager, and suggest the virtual
    www-browser package (closes: #321769).
  * Update debian/copyright with the FSF's new address.

 -- Colin Watson <cjwatson@debian.org>  Tue, 30 Aug 2005 13:37:35 +0100

I think the rest of this bug now belongs to col; at a minimum it should
avoid corrupting UTF-8 text when being run in a UTF-8 locale. Ideally it
would also handle other multibyte encodings correctly.

Cheers,

-- 
Colin Watson                                       [cjwatson@debian.org]

Message #33 received at 319952@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: Denis Barbier <barbier@linuxfr.org>

Cc: Norbert Preining <preining@logic.at>, Christian Perrier <bubulle@debian.org>, 348032@bugs.debian.org, Vincent Lönngren <bice77@malmo2.net>, Bas Zoetekouw <bas@debian.org>, 319952@bugs.debian.org, bsdmainutils@packages.debian.org, control@bugs.debian.org

Subject: Re: Bug#348032: dpkg: Some characters are replace with questionmarks in the Swedish man page when UTF-8 locale is used

Date: Sun, 26 Aug 2007 14:23:37 +0100

reassign 348032 bsdmainutils
severity 348032 normal
merge 319952 348032
# 'patch' if a link to CVS counts; I wasn't sure that 'fixed-upstream'
# was quite right either.
tags 319952 patch
thanks

On Mon, Jan 16, 2006 at 11:59:27AM +0100, Denis Barbier wrote:
> you are right, output is not valid utf8 when redirected into a file,
> reassigning to man-db.  Here is another example:
>   $ LC_ALL=en_US.UTF-8 man -L C man > man.out
> messes up some hyphens (0x2010) and apostrophs (0x2019); it looks
> like a multibyte sequence of n bytes is prepended by the (n-1) first
> bytes, but not for all occurences.

This is the same bug as #319952, and is a bug in col(1).

I discovered that this was fixed some time ago in FreeBSD. See revision
1.18 here:

  http://www.freebsd.org/cgi/cvsweb.cgi/src/usr.bin/col/col.c

I built the current version from FreeBSD CVS (removing the __FBSDID line
so that it would compile on Debian) and confirmed that it fixes this
bug. Daniel, could you please upgrade bsdmainutils to include this new
version of col? As the man-db maintainer I would greatly appreciate it.

Thanks,

-- 
Colin Watson                                       [cjwatson@debian.org]

Message #42 received at 319952@bugs.debian.org (full text, mbox, reply):

From: Daniel Baumann <daniel@debian.org>

To: Colin Watson <cjwatson@debian.org>, 319952@bugs.debian.org

Subject: Re: Bug#319952: Bug#348032: dpkg: Some characters are replace with questionmarks in the Swedish man page when UTF-8 locale is used

Date: Sun, 26 Aug 2007 15:31:50 +0200

Colin Watson wrote:
> Daniel, could you please upgrade bsdmainutils to include this new
> version of col? As the man-db maintainer I would greatly appreciate it.

i'm currently working on porting all debian changes to proper patches
against freebsd; however, this needs another weekend time.. but i'm
definitely on it.

-- 
Address:        Daniel Baumann, Burgunderstrasse 3, CH-4562 Biberist
Email:          daniel.baumann@panthera-systems.net
Internet:       http://people.panthera-systems.net/~daniel-baumann/

Message #49 received at 319952@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: Matt Wozniski <mjw@drexel.edu>, 505086@bugs.debian.org

Cc: 319952@bugs.debian.org

Subject: Re: Bug#505086: man-db: man can emit invalid utf-8

Date: Wed, 12 Nov 2008 12:42:13 +0000

On Sun, Nov 09, 2008 at 04:48:33AM -0500, Matt Wozniski wrote:
> As you already know, man uses 'col -b -p -x' in its pipeline when piping
> the man page to a child process, unless $MAN_KEEP_FORMATTING is
> specified, and col doesn't handle UTF-8 properly.  Because of this, some
> man pages are invalid UTF-8 by the time they're done being handled, such
> as "man xterm | iconv".  However, even if -b isn't passed to col, it
> reorders things so that each character is immediately followed by the
> backspace that erases it, if any - so, "abc^H^H^Hxyz" becomes
> "a^Hxb^Hyc^Hz".  I'd like to suggest that we use this to fix the
> problem, by changing the ending of the pipeline from '| col -b -p -x' to
> '| col -p -x | sed -e "s/^\x08*//" -e "s/.\x08//g"'.  I don't think
> this would ever be likely to be accepted upstream, since it depends on
> (I think?) an implementation detail of 'col', but since we have a col
> that doesn't support multibyte, but rearranges the text to be easily
> postprocessed, and a sed that does support multibyte, this seems like
> a quick-and-dirty way to fix the problem.

Thanks for this suggestion and your patch.

With regard to acceptance upstream, since I am upstream obviously I am
looking for something acceptable to myself to some extent. :-)


I have to say that to some extent I wonder why we can't just get #319952
fixed rather than having to work around it further in man. Daniel, in
August 2007 you said:

  "i'm currently working on porting all debian changes to proper patches
  against freebsd; however, this needs another weekend time.. but i'm
  definitely on it."

Have you had a chance to make any further progress on this? That said, I
can see that this workaround with sed would be a more plausible
candidate for late inclusion in lenny than full UTF-8 support in col
would be.


Given that we know that col is broken on some systems, I wonder if the
correct upstream solution wouldn't simply be to reimplement the part of
col that we need, and call it as an internal pipeline-processing
function. I'm not a huge fan of duplication in general, but it's not as
if col is hugely complex nor as if it changes much.

-- 
Colin Watson                                       [cjwatson@debian.org]

Message #54 received at 319952@bugs.debian.org (full text, mbox, reply):

From: <john.lindgren@tds.net>

To: cjwatson@debian.org, 319952@bugs.debian.org

Subject: Re: col doesn't handle UTF-8

Date: Fri, 6 Mar 2009 0:08:38 -0600

As a Debian user annoyed by this problem, I would be willing to do some coding or patching work toward fixing it, though I can't do packaging or uploading. What is the status of this bug, and is there any way I can help?

John

Message #59 received at 319952@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: john.lindgren@tds.net

Cc: 319952@bugs.debian.org

Subject: Re: col doesn't handle UTF-8

Date: Mon, 9 Mar 2009 12:00:22 +0000

On Fri, Mar 06, 2009 at 12:08:38AM -0600, john.lindgren@tds.net wrote:
> As a Debian user annoyed by this problem, I would be willing to do
> some coding or patching work toward fixing it, though I can't do
> packaging or uploading. What is the status of this bug, and is there
> any way I can help?

I can't comment on the status of this bug in bsdmainutils.

As for the workaround in man-db suggested in my most recent mail to this
bug, I do intend to work on this for man-db 2.5.5, but have not yet
started on it. If you want to help,
http://man-db.nongnu.org/development.html has some starting points.

Cheers,

-- 
Colin Watson                                       [cjwatson@debian.org]

Message #64 received at 319952@bugs.debian.org (full text, mbox, reply):

From: <john.lindgren@tds.net>

To: 319952@bugs.debian.org, 348032@bugs.debian.org, 484579@bugs.debian.org, cjwatson@debian.org, daniel@debian.org

Cc: bas@debian.org, bice77@malmo2.net, tsuchiya@namazu.org

Subject: Re: col doesn't handle UTF-8

Date: Mon, 9 Mar 2009 16:31:28 -0500

[Message part 1 (text/plain, inline)]

Attached is a patch to make col UTF8-aware. Be warned that the patch has
not been thoroughly tested.

John

[col.c.patch (text/x-diff, attachment)]

Message #77 received at 319952-close@bugs.debian.org (full text, mbox, reply):

From: Michael Meskes <meskes@debian.org>

To: 319952-close@bugs.debian.org

Subject: Bug#319952: fixed in bsdmainutils 8.0.1

Date: Sun, 08 Nov 2009 11:02:07 +0000

Source: bsdmainutils
Source-Version: 8.0.1

We believe that the bug you reported is fixed in the latest version of
bsdmainutils, which is due to be installed in the Debian FTP archive:

bsdmainutils_8.0.1.dsc
  to main/b/bsdmainutils/bsdmainutils_8.0.1.dsc
bsdmainutils_8.0.1.tar.gz
  to main/b/bsdmainutils/bsdmainutils_8.0.1.tar.gz
bsdmainutils_8.0.1_amd64.deb
  to main/b/bsdmainutils/bsdmainutils_8.0.1_amd64.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 319952@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Michael Meskes <meskes@debian.org> (supplier of updated bsdmainutils package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Sat, 07 Nov 2009 12:31:56 +0100
Source: bsdmainutils
Binary: bsdmainutils
Architecture: source amd64
Version: 8.0.1
Distribution: unstable
Urgency: low
Maintainer: Debian Bsdmainutils Team <pkg-bsdmainutils@teams.debian.net>
Changed-By: Michael Meskes <meskes@debian.org>
Description: 
 bsdmainutils - collection of more utilities from FreeBSD
Closes: 264996 280176 293689 315664 319952 335087 337311 348032 358609 361223 367299 368384 381114 382638 388153 401091 413900 415057 431930 446547 455248 472355 474600 484579 485809 487985 493759 497014 498232 503276 507602 516271 525925 528657 542229 543833 547622 554561
Changes: 
 bsdmainutils (8.0.1) unstable; urgency=low
 .
   * Added patch system to be able to track upstream's source more
     directly.
   * Updated ncal to the latest upstream version. (Closes: #415057)
   * New maintainers. (Closes: #543833)
   * Bumped Standards version to 3.8.3, no update needed.
   * Made ncal find the first day of the week automatically.
     (Closes: #472355)
   * Added -3 option to cal and ncal. (Closes: #497014)
   * Made ncal cope with longer month names. (Closes: #528657)
   * Document that cal always prints 8 lines. (Closes: #367299)
   * Made ncal catch incorrect year parameter. (Closes: #431930)
   * Made ncal use locale information for knowing how to display week.
     (Closes: #361223)
   * Updated col to the latest FreeBSD version.
     (Closes: #319952, #348032, #484579)
   * Patched col to recognize a single non-empty line without newline.
     (Closes: #335087)
   * Updated colrm to the latest FreeBSD version. (Closes: #516271)
   * Updated column to latest FreeBSD version. (Closes: #368384)
   * Made column react more gracefully upon reading empty fields.
     (Closes: #382638)
   * Re-added '-n' option to column and documented it as a Debian
     extension. (Closes: #485809)
   * Updated colcrt to the latest FreeBSD version.
   * Updated banner to latest FreeBSD version.
   * Renamed banner to printerbanner. (Closes: #315664)
   * Updated hexdump to its latest FreeBSD version.
   * Bumped debhelper compat level.
   * Added patch to prevent segfault in case an empty repetition is
     given. (Closes: #498232)
   * Added patch to make hd ignore -C option. (Closes: #487985)
   * Updated from to latest FreeBSD version.
   * Updated ul to latest FreeBSD version but kept our changes
   * Updated lorder to latest version from OpenBSD.
   * Re-added patch that makes lorder use signal names instead of
     numbers.
   * Updated look to latest version from FreeBSD. (Closes: #547622)
   * Reimplemented and documented -b option for look. (Closes: #264996)
   * Update write to the latest FreeBSD version.
   * Allow writing from a terminal that has mesg n set. (Closes: #455248)
   * Updated calendar binary to latest version from OpenBSD. (Closes: #503276)
   * Patched new calendar sources to use wide-character functions.
   * Created patch to put Debian specific options back into calendar
     tool. (Closes: #293689)
   * Updated all calendars from their FreeBSD sources.
     (Closes: #388153, #413900, #446547, #542229)
   * Added/Fixed some calendar entries as reported by Debian users.
     (Closes: #554561, #337311, #525925, #493759, #381114, #280176))
   * Added Kazakhstan holiday calendar. (Closes: #358609)
   * Updated Ubuntu calendar. (Closes: #474600)
   * Removed double word in manpage. (Closes: #401091)
   * Changed patch of source.data.gz mentioned in README. (Closes: #507602)
   * Added lintian override for setgid write binary.
   * Bumped version to 8.0.1 because we are using sources from FreeBSD 8 now.
Checksums-Sha1: 
 fbc70c465f992b86cb41fd7b8f8c076468a10edf 857 bsdmainutils_8.0.1.dsc
 9ff212232441080e8eb5d8268e718cefa951f318 250897 bsdmainutils_8.0.1.tar.gz
 dcc1d82282d6e0d87fa812d25ace60b094eb0354 197298 bsdmainutils_8.0.1_amd64.deb
Checksums-Sha256: 
 9dcf752d8b3cdab0078eb18977714eda4794c07673cd0f9f452ec573b0011383 857 bsdmainutils_8.0.1.dsc
 d7ef9be8cb16ed2180bed496a41f74511956a4a42303c71710817144f48f902a 250897 bsdmainutils_8.0.1.tar.gz
 14eb9eb4fbd71704787a809fd4a1a7ddfca80631e79c9b50b415529dc297c2c2 197298 bsdmainutils_8.0.1_amd64.deb
Files: 
 1145378013bdfdb9d29b43f5040b85de 857 utils important bsdmainutils_8.0.1.dsc
 46147db6db21d4cff5efc5278691cf2d 250897 utils important bsdmainutils_8.0.1.tar.gz
 bc13d296b6b0ef822efa53eca4850197 197298 utils important bsdmainutils_8.0.1_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iD8DBQFK9qBRVkEm8inxm9ERAqUqAJ9CkXY2NNi95SEeQGLLSRVVGO+HtwCeLRWq
J/wmeSCwMPzuPc1BbzP3Imc=
=cw6W
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.