Debian Bug report logs - #373704
sort -k does not count fields the same as gnu sort

version graph

Package: busybox; Maintainer for busybox is Debian Install System Team <debian-boot@lists.debian.org>; Source for busybox is src:busybox.

Reported by: Joey Hess <joeyh@debian.org>

Date: Thu, 15 Jun 2006 07:33:02 UTC

Severity: important

Tags: d-i

Found in versions busybox/1:1.1.3-1, 1:1.1.3-3

Fixed in version busybox/1:1.1.3-4

Done: Bastian Blank <waldi@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Joey Hess <joeyh@debian.org>:
New Bug report received and forwarded. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Joey Hess <joeyh@debian.org>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: sort -k does not count fields the same as gnu sort
Date: Thu, 15 Jun 2006 03:03:15 -0400
[Message part 1 (text/plain, inline)]
Package: busybox
Version: 1:1.1.3-1
Severity: normal
Tags: d-i

joey@kodama:~>cat foo
/usr/lib/finish-install.d/1
/usr/lib/prebaseconfig.d/2
/usr/lib/finish-install.d/4
/usr/lib/prebaseconfig.d/6
joey@kodama:~>for n in 3 4 5; do echo busybox $n; busybox sort -n -k$n -t/ foo; echo gnu $n; sort -n -k$n -t/ foo;done
busybox 3
/usr/lib/finish-install.d/1
/usr/lib/finish-install.d/4
/usr/lib/prebaseconfig.d/2
/usr/lib/prebaseconfig.d/6
gnu 3
/usr/lib/finish-install.d/1
/usr/lib/finish-install.d/4
/usr/lib/prebaseconfig.d/2
/usr/lib/prebaseconfig.d/6
busybox 4
/usr/lib/finish-install.d/1
/usr/lib/prebaseconfig.d/2
/usr/lib/finish-install.d/4
/usr/lib/prebaseconfig.d/6
gnu 4
/usr/lib/finish-install.d/1
/usr/lib/finish-install.d/4
/usr/lib/prebaseconfig.d/2
/usr/lib/prebaseconfig.d/6
busybox 5
/usr/lib/finish-install.d/1
/usr/lib/finish-install.d/4
/usr/lib/prebaseconfig.d/2
/usr/lib/prebaseconfig.d/6
gnu 5
/usr/lib/finish-install.d/1
/usr/lib/prebaseconfig.d/2
/usr/lib/finish-install.d/4
/usr/lib/prebaseconfig.d/6

So, busybox sorts them numerically at field 4, while gnu does at
field 5. I think gnu must count field 1 as the zero-width field before the
first delimeter in this case. If the lines don't start with the delimiter,
gnu and busybox both behave the same:

busybox 4
root/usr/lib/finish-install.d/1
root/usr/lib/finish-install.d/4
root/usr/lib/prebaseconfig.d/2
root/usr/lib/prebaseconfig.d/6
gnu 4
root/usr/lib/finish-install.d/1
root/usr/lib/finish-install.d/4
root/usr/lib/prebaseconfig.d/2
root/usr/lib/prebaseconfig.d/6
busybox 5
root/usr/lib/finish-install.d/1
root/usr/lib/prebaseconfig.d/2
root/usr/lib/finish-install.d/4
root/usr/lib/prebaseconfig.d/6
gnu 5
root/usr/lib/finish-install.d/1
root/usr/lib/prebaseconfig.d/2
root/usr/lib/finish-install.d/4
root/usr/lib/prebaseconfig.d/6

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.16-2-686
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)

Versions of packages busybox depends on:
ii  libc6                         2.3.6-15   GNU C Library: Shared libraries

busybox recommends no packages.

-- no debconf information

-- 
see shy jo
[signature.asc (application/pgp-signature, inline)]

Reply sent to Bastian Blank <waldi@debian.org>:
You have taken responsibility. Full text and rfc822 format available.

Notification sent to Joey Hess <joeyh@debian.org>:
Bug acknowledged by developer. Full text and rfc822 format available.

Message #10 received at 373704-close@bugs.debian.org (full text, mbox):

From: Bastian Blank <waldi@debian.org>
To: 373704-close@bugs.debian.org
Subject: Bug#373704: fixed in busybox 1:1.1.3-2
Date: Thu, 29 Jun 2006 13:32:09 -0700
Source: busybox
Source-Version: 1:1.1.3-2

We believe that the bug you reported is fixed in the latest version of
busybox, which is due to be installed in the Debian FTP archive:

busybox-static_1.1.3-2_amd64.deb
  to pool/main/b/busybox/busybox-static_1.1.3-2_amd64.deb
busybox-udeb_1.1.3-2_amd64.udeb
  to pool/main/b/busybox/busybox-udeb_1.1.3-2_amd64.udeb
busybox_1.1.3-2.diff.gz
  to pool/main/b/busybox/busybox_1.1.3-2.diff.gz
busybox_1.1.3-2.dsc
  to pool/main/b/busybox/busybox_1.1.3-2.dsc
busybox_1.1.3-2_amd64.deb
  to pool/main/b/busybox/busybox_1.1.3-2_amd64.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 373704@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Bastian Blank <waldi@debian.org> (supplier of updated busybox package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.7
Date: Thu, 29 Jun 2006 19:36:19 +0000
Source: busybox
Binary: busybox-floppy-udeb busybox busybox-udeb busybox-static
Architecture: source amd64
Version: 1:1.1.3-2
Distribution: unstable
Urgency: low
Maintainer: Debian Install System Team <debian-boot@lists.debian.org>
Changed-By: Bastian Blank <waldi@debian.org>
Description: 
 busybox    - Tiny utilities for small and embedded systems
 busybox-static - Standalone rescue shell with tons of builtin utilities
 busybox-udeb - Tiny utilities for the debian-installer (udeb)
Closes: 373704
Changes: 
 busybox (1:1.1.3-2) unstable; urgency=low
 .
   * coreutils/sort.c:
     - Fix sorting of input which begins with the seperator. (closes: #373704)
Files: 
 2b31bc93591781fe140638a2539ea97b 692 - optional busybox_1.1.3-2.dsc
 43a5446992bd80862476c05135ab95c9 19247 - optional busybox_1.1.3-2.diff.gz
 d1464e31b449649b1309caa44cc19dea 325562 utils optional busybox_1.1.3-2_amd64.deb
 80e2c8b32a336de1ac4449c828b93d81 914814 shells optional busybox-static_1.1.3-2_amd64.deb
 b3c4e3cdb3cdff5ee99d6341c5788453 139936 debian-installer extra busybox-udeb_1.1.3-2_amd64.udeb
Package-Type: udeb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iEYEARECAAYFAkSkMAwACgkQLkAIIn9ODhGVZACg0dexr1ZzOBfz0sNYaowQK5VV
i5kAnjFbFcNvlMGafUIk9wrKr0Njs15u
=S0ce
-----END PGP SIGNATURE-----




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Samuel Thibault <samuel.thibault@ens-lyon.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #15 received at 373704@bugs.debian.org (full text, mbox):

From: Samuel Thibault <samuel.thibault@ens-lyon.org>
To: 373704@bugs.debian.org, control@bugs.debian.org
Cc: 373118@bugs.debian.org
Subject: Re: sort -k does not count fields the same as gnu sort
Date: Wed, 23 Aug 2006 02:19:54 +0200
reopen 373704
thanks

Version: 1:1.1.3-2

Hi,

It doesn't seem to be fixed:

$ sort -k5 -t/ -n
/usr/lib/finish-install.d/99reboot
/usr/lib/prebaseconfig.d/50brltty
^D
/usr/lib/finish-install.d/99reboot
/usr/lib/prebaseconfig.d/50brltty

While it would be expected to see 50brltty coming before 99reboot.

Samuel



Bug reopened, originator not changed. Request was from Samuel Thibault <samuel.thibault@ens-lyon.org> to control@bugs.debian.org. Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to stappers@stappers.nl (Geert Stappers):
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #22 received at 373704@bugs.debian.org (full text, mbox):

From: stappers@stappers.nl (Geert Stappers)
To: Samuel Thibault <samuel.thibault@ens-lyon.org>, 373704@bugs.debian.org
Cc: 373118@bugs.debian.org
Subject: Re: Bug#373704: sort -k does not count fields the same as gnu sort
Date: Thu, 24 Aug 2006 09:46:54 +0200
On Wed, Aug 23, 2006 at 02:19:54AM +0200, Samuel Thibault wrote:
> reopen 373704
> thanks
> 
> Version: 1:1.1.3-2
> 
> Hi,
> 
> It doesn't seem to be fixed:
> 
> $ sort -k5 -t/ -n
                 ^^
> /usr/lib/finish-install.d/99reboot
> /usr/lib/prebaseconfig.d/50brltty
> ^D
> /usr/lib/finish-install.d/99reboot
> /usr/lib/prebaseconfig.d/50brltty
> 
> While it would be expected to see 50brltty coming before 99reboot.

How does it behave when you leave out the "numeric" ( -n ) ?


HtH
GSt



Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Samuel Thibault <samuel.thibault@ens-lyon.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #27 received at 373704@bugs.debian.org (full text, mbox):

From: Samuel Thibault <samuel.thibault@ens-lyon.org>
To: Geert Stappers <stappers@stappers.nl>
Cc: 373704@bugs.debian.org, 373118@bugs.debian.org
Subject: Re: Bug#373704: sort -k does not count fields the same as gnu sort
Date: Thu, 24 Aug 2006 09:58:37 +0200
Geert Stappers, le Thu 24 Aug 2006 09:46:54 +0200, a écrit :
> > It doesn't seem to be fixed:
> > 
> > $ sort -k5 -t/ -n
>                  ^^
> > /usr/lib/finish-install.d/99reboot
> > /usr/lib/prebaseconfig.d/50brltty
> > ^D
> > /usr/lib/finish-install.d/99reboot
> > /usr/lib/prebaseconfig.d/50brltty
> > 
> > While it would be expected to see 50brltty coming before 99reboot.
> 
> How does it behave when you leave out the "numeric" ( -n ) ?

Just the same:

$ sort -k5 -t/
/usr/lib/finish-install.d/99reboot
/usr/lib/prebaseconfig.d/50brltty
^D
/usr/lib/finish-install.d/99reboot
/usr/lib/prebaseconfig.d/50brltty

Samuel



Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Frans Pop <elendil@planet.nl>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #32 received at 373704@bugs.debian.org (full text, mbox):

From: Frans Pop <elendil@planet.nl>
To: 373704@bugs.debian.org
Subject: Bug#373704: sort -k does not count fields the same as gnu sort
Date: Sat, 13 Jan 2007 13:50:15 +0100
[Message part 1 (text/plain, inline)]
severity 373704 critical
found 373704 1:1.1.3-3
thanks

I had meant to look into this much earlier, but unfortunately it slipped 
my mind. Severity critical as this behavior can cause hard to trace 
unpredictable behavior in any script using sort with a delimiter, 
potentially leading to random breakage in unrelated software.

With the following test file:
a c a
b b c
a a b

# sort -k1 test
a a b
a c a
b b c
# sort -k2 test
a a b
b b c
a c a
# sort -k3 test
a c a
a a b
b b c

Conclusion: when testing _without_ delimiter, the sort is correct.

With a slightly modified test file:
a/c/a
b/b/c
a/a/b

# sort -k1 -t/ test
a/a/b
a/c/a
b/b/c
# sort -k2 -t/ test
a/a/b
b/b/c
a/c/a
# sort -k3 -t/ test
a/a/b
b/b/c
a/c/a

Conclusion: when sorting with delimiter on the last field, the 
last-but-one field is sorted instead.

With a leading delimiter in the test file:
/a/c/a
/b/b/c
/a/a/b

# sort -k1 -t/ test
/a/a/b
/a/c/a
/b/b/c
# sort -k2 -t/ test
/a/a/b
/a/c/a
/b/b/c
# sort -k3 -t/ test
/a/a/b
/a/c/a
/b/b/c

Conclusion: with a leading delimiter, the sort is somehow _always_ done on 
the first field.

It looks like sorting with a specific delimiter set is completely broken.
Tests were done inside Debian Installer.

The testcase included in the busybox testsuite seems to work only by 
accident as it is a numeric sort and the last field to be sorted is the 
only numerical data in each line.
[Message part 2 (application/pgp-signature, inline)]

Severity set to `critical' from `normal' Request was from Frans Pop <elendil@planet.nl> to control@bugs.debian.org. Full text and rfc822 format available.

Bug marked as found in version 1:1.1.3-3. Request was from Frans Pop <elendil@planet.nl> to control@bugs.debian.org. Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Frans Pop <elendil@planet.nl>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #41 received at 373704@bugs.debian.org (full text, mbox):

From: Frans Pop <elendil@planet.nl>
To: 373704@bugs.debian.org
Subject: Re: Bug#373704: sort -k does not count fields the same as gnu sort
Date: Sat, 13 Jan 2007 14:38:38 +0100
[Message part 1 (text/plain, inline)]
On Saturday 13 January 2007 13:50, Frans Pop wrote:
> The testcase included in the busybox testsuite seems to work only by
> accident as it is a numeric sort and the last field to be sorted is the
> only numerical data in each line.

It turns out the testcase based on the D-I situation is broken in two 
respects:
- The testing function wants the _expected output_ as its 3rd parameter
  and the _input_ as the 5th parameter.
  In testsuite/sort.tests these are reversed.
- If I run that test with GNU 'sort' (from my regular Etch system), the
  sort only works correctly if I sort with -k5, not -k4.
  This means that GNU 'sort' _does_ count the empty field before the
  leading separator (as also originally reported by Joey).

Attached an extra set of testcases, with the corrected D-I case and a full 
set of cases for sorting with and without leading delimiter.
These all PASS if I use GNU sort.

Cheers,
FJP

[sort.tests_new (text/plain, attachment)]
[Message part 3 (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Frans Pop <elendil@planet.nl>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #46 received at 373704@bugs.debian.org (full text, mbox):

From: Frans Pop <elendil@planet.nl>
To: 373704@bugs.debian.org
Subject: Re: Bug#373704: sort -k does not count fields the same as gnu sort
Date: Sat, 13 Jan 2007 14:53:53 +0100
[Message part 1 (text/plain, inline)]
On Saturday 13 January 2007 13:50, Frans Pop wrote:
> Conclusion: when sorting with delimiter on the last field, the
> last-but-one field is sorted instead.
[...]
> Conclusion: with a leading delimiter, the sort is somehow _always_ done
> on the first field.

The last conclusion is not correct as the sort is not done on the first, 
but on the second field as the leading "null" field should be counted as 
well.

This actually makes the sorting with and without leading delimiter a lot 
more similar: it looks as if sorting past the second field is somehow not 
supported.
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #51 received at 373704@bugs.debian.org (full text, mbox):

From: Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>
To: Debian Bug Tracking System <373704@bugs.debian.org>
Subject: Patch for sort issues
Date: Sat, 13 Jan 2007 19:38:06 +0100
[Message part 1 (text/plain, inline)]
Package: busybox
Version: 1:1.1.3-3
Followup-For: Bug #373704

Attached is a patch for the sort issues. A few have croped up while
testing this. Hopefully busybox sort now behaves like coreutils sort.

About the patch (line numbers of result):

Line 65   : Fix -b -f -i -d and trailing spaces.

Try any of those flags on its own.

Line 75   : Seperators are not skipped (never where)
Line 76-77: Collaps lines:
	     - the while checks for isspace() already
	     - combine str[end] and !key_separator

No code change, just correcting the comment and simplifying.


Line 78-82: Move 'end++' into the loop into the tests
	    for becomes while

The loop in line 74 is supposed to count 'key->range[2*j]+j'
seperators or whitespaces. The old code would advance end up to the
first seperator and then no more. The same seperator got counted again
and again. Only -k1 and -k2 worked. Moving the 'end++' inside the loop
into the test cases causes them to advance past the seperator so the
next loop starts fresh on the next key again.

",x,,,a" gets split into '','x','','','a' now, into 5 key positions.


Line 88 + : The fix in 78-82 included the last whitespace or
            separator, remove it here.

Without this sort -k2,2 -t, and this input

x,a,a
x,a

would use the following strings for comparisons:

a,
a


Line 88 - : Don't skip the first seperator if a key starts with
            seperators.

Frankly that line didn't make any sense. Consider sort -k2 -t,

,a
,b
,,a
,,b
,,,a
,,,b

That gets split into
a
b
,a
,b
,,a
,,b

And then one leading , got ignored giving

a
b
a
b
,a
,b

and last sorted into

,,,a
,,,b
,,a
,a
,,b
,b

It should either ignore all leading seperators or none, not
one. Coreutils ignores none so I removed that line completly.

MfG
	Goswin

-- System Information:
Debian Release: 4.0
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-frosties
Locale: LANG=C, LC_CTYPE=de_DE (charmap=ISO-8859-1)

Versions of packages busybox depends on:
ii  libc6                        2.3.6.ds1-9 GNU C Library: Shared libraries

busybox recommends no packages.

-- no debconf information
[busybox-sort.patch (text/x-c, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Steve Langasek <vorlon@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #56 received at 373704@bugs.debian.org (full text, mbox):

From: Steve Langasek <vorlon@debian.org>
To: Frans Pop <elendil@planet.nl>, 373704@bugs.debian.org
Subject: Re: Bug#373704: sort -k does not count fields the same as gnu sort
Date: Sat, 13 Jan 2007 13:10:25 -0800
On Sat, Jan 13, 2007 at 01:50:15PM +0100, Frans Pop wrote:
> severity 373704 critical
> found 373704 1:1.1.3-3
> thanks

> I had meant to look into this much earlier, but unfortunately it slipped 
> my mind. Severity critical as this behavior can cause hard to trace 
> unpredictable behavior in any script using sort with a delimiter, 
> potentially leading to random breakage in unrelated software.

"Potentially leading to random breakage" doesn't really justify a critical
severity when there are a limited number of packages making use of busybox
(and busybox sort in particular).  Does this bug actually break d-i, and if
so how?

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
vorlon@debian.org                                   http://www.debian.org/



Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Samuel Thibault <samuel.thibault@ens-lyon.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #61 received at 373704@bugs.debian.org (full text, mbox):

From: Samuel Thibault <samuel.thibault@ens-lyon.org>
To: Steve Langasek <vorlon@debian.org>, 373704@bugs.debian.org
Cc: Frans Pop <elendil@planet.nl>
Subject: Re: Bug#373704: sort -k does not count fields the same as gnu sort
Date: Sun, 14 Jan 2007 02:04:11 +0100
Steve Langasek, le Sat 13 Jan 2007 13:10:25 -0800, a écrit :
> "Potentially leading to random breakage" doesn't really justify a critical
> severity when there are a limited number of packages making use of busybox
> (and busybox sort in particular).  Does this bug actually break d-i, and if
> so how?

It used to break the execution order of some scripts.  Currently it
doesn't, but it could very well break again.

Samuel



Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Frans Pop <elendil@planet.nl>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #66 received at 373704@bugs.debian.org (full text, mbox):

From: Frans Pop <elendil@planet.nl>
To: 373704@bugs.debian.org
Cc: debian-release@lists.debian.org, Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>
Subject: Re: Bug#373704: sort -k does not count fields the same as gnu sort
Date: Sun, 14 Jan 2007 03:39:09 +0100
[Message part 1 (text/plain, inline)]
On Saturday 13 January 2007 22:10, Steve Langasek wrote:
> "Potentially leading to random breakage" doesn't really justify a
> critical severity when there are a limited number of packages making
> use of busybox (and busybox sort in particular).  Does this bug
> actually break d-i, and if so how?

Let me start with a little quote:
<ifvoid> fjp: why is it rc?
<fjp> ifvoid: Basically because it can cause unexpected breakage in 
anything that uses busybox sort.
<fjp> ifvoid: My feeling is that the behavior now is even worse than the 
original bug.
<fjp> ifvoid: Not sure if RMs would agree though.
<aj> unexpected breakage in busybox sort sounds horrible
* Maulkin nods

"Critical" is possibly too high if you just consider Debian packages. 
However, if you also consider use of busybox in embedded devices where 
scripts written for GNU sort are being run (which apparently is a big 
market for Debian), it could still be justified.
It can at least potentially break unrelated software.

I think "not suitable for release" is quite justified in this case, 
especially as the sort function in Sarge works better. The original fix 
for this bug has caused sorting with a delimiter (-t option) to work 
worse than it did.
In the past, it was "off by one field" in an edge case (lines starting 
with delimiter). Now sorting with a delimiter is completely broken for 
anything but the first 2 fields.

For D-I this currently only breaks a minor backwards compatibility 
function (correctly merging finish-install.d and prebaseconfig.d hook 
scripts). As this has basically been broken since the start and as far as 
we know there are no _official_ udebs using it anymore, this is no big 
deal.


I've taken a look at Goswin's analysis and patch, and it looks clean to 
me. I've also done a lot of testing with the patched version, and been 
discovering a lot of things about sort in the process (as well as a real 
bug in GNU sort: #406785).

I've adjusted and extended the test suite (new test script attached), as 
some of the tests were based on a misunderstanding of how sort is 
supposed to work (see also #367891).
All tests now give the correct results and are the same as when using GNU 
sort (except for the subrange sort which is broken in GNU sort).

With current busybox sort, three of the tests in my revised set fail:
FAIL: sort key field subrange with reverse option
(This is actually the same error as in GNU sort, so current busybox is 
consistent with GNU in that respect...)
FAIL: sort key edge case with -t
FAIL: sort key with (leading) delimiter

I have also seen a few cases where busybox sort now behaves different from 
GNU sort, but those all involved keys over multiple fields with numerical 
sort option. See #367891 why this is not really supported anyway. The old 
busybox sort also had some differences in these tests.

I'll try to do some additional testing on this, and I would like to 
discuss one point with Goswin.

Cheers,
FJP

P.S. I fully agree that this is not the best time in a release cycle to 
dive into such things :-/
[sort.tests (application/x-shellscript, attachment)]
[Message part 3 (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #71 received at 373704@bugs.debian.org (full text, mbox):

From: Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>
To: Debian Bug Tracking System <373704@bugs.debian.org>
Subject: Updated patch
Date: Mon, 15 Jan 2007 00:18:40 +0100
[Message part 1 (text/plain, inline)]
Package: busybox
Version: 1:1.1.3-3
Followup-For: Bug #373704

Hi,

attached an updated patch for busybox that includes the fix for
  #406785 sort: incorrect result when sorting on subfields
or does not break it if you will.

MfG
	Goswin

-- System Information:
Debian Release: 4.0
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-frosties
Locale: LANG=C, LC_CTYPE=de_DE (charmap=ISO-8859-1)

Versions of packages busybox depends on:
ii  libc6                        2.3.6.ds1-9 GNU C Library: Shared libraries

busybox recommends no packages.

-- no debconf information
[busybox-sort2.patch (text/x-c, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #76 received at 373704@bugs.debian.org (full text, mbox):

From: Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>
To: Debian Bug Tracking System <373704@bugs.debian.org>
Subject: Another patch update
Date: Mon, 15 Jan 2007 01:09:10 +0100
[Message part 1 (text/plain, inline)]
Package: busybox
Version: 1:1.1.3-3
Followup-For: Bug #373704

Hi,

Frans Pop noticed a mistake in the latest patch. The last fix (-2)
changed the way 'end' turns up at the end of the loop for no separator
but still removed one from it. Sort did then crop the end of key by
one without separator.

Patch 3 attached.

MfG
	Goswin

-- System Information:
Debian Release: 4.0
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-frosties
Locale: LANG=C, LC_CTYPE=de_DE (charmap=ISO-8859-1)

Versions of packages busybox depends on:
ii  libc6                        2.3.6.ds1-9 GNU C Library: Shared libraries

busybox recommends no packages.

-- no debconf information
[busybox-sort3.patch (text/x-c, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #81 received at 373704@bugs.debian.org (full text, mbox):

From: Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>
To: Debian Bug Tracking System <373704@bugs.debian.org>
Subject: Once more with feeling
Date: Mon, 15 Jan 2007 02:42:39 +0100
[Message part 1 (text/plain, inline)]
Package: busybox
Version: 1:1.1.3-3
Followup-For: Bug #373704

And once more another patch.

sort -kx,y.z caused busybox to find the end of key y without trailing
separator and then add z-1. And leading whitespaces didn't get
ignored.

I changed the code now to the following:

-kx,y[.0] find start of key y+1 and rewind the separator
-kx,y.z   find start of key y, skip leading spaces if needed, add z

I'm not sure about removing trailing whitespaces. I moved it to after
adjusting end by the offset. Makes no sense to remove trailing
whitespaces when end still points to the start of y. Someone has to
create a few testcases for this and tell me if it differs from
coreutils sort in any way. FLAG_bb gets set with -b and -k1,2b.

MfG
	Goswin

-- System Information:
Debian Release: 4.0
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-frosties
Locale: LANG=C, LC_CTYPE=de_DE (charmap=ISO-8859-1)

Versions of packages busybox depends on:
ii  libc6                        2.3.6.ds1-9 GNU C Library: Shared libraries

busybox recommends no packages.

-- no debconf information
[busybox-sort4.patch (text/x-c, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #86 received at 373704@bugs.debian.org (full text, mbox):

From: Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>
To: Debian Bug Tracking System <373704@bugs.debian.org>
Subject: Wrong patch
Date: Mon, 15 Jan 2007 09:45:18 +0100
[Message part 1 (text/plain, inline)]
Package: busybox
Version: 1:1.1.3-3
Followup-For: Bug #373704

Sorry, I send the wrong patch last night.

MfG
	Goswin

-- System Information:
Debian Release: 4.0
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-frosties
Locale: LANG=C, LC_CTYPE=de_DE (charmap=ISO-8859-1)

Versions of packages busybox depends on:
ii  libc6                        2.3.6.ds1-9 GNU C Library: Shared libraries

busybox recommends no packages.

-- no debconf information
[busybox-sort4.patch (text/x-c, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Frans Pop <elendil@planet.nl>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #91 received at 373704@bugs.debian.org (full text, mbox):

From: Frans Pop <elendil@planet.nl>
To: 373704@bugs.debian.org
Subject: Re: Bug#373704: sort -k does not count fields the same as gnu sort
Date: Wed, 17 Jan 2007 02:42:46 +0100
[Message part 1 (text/plain, inline)]
On Saturday 13 January 2007 13:50, Frans Pop wrote:
> It looks like sorting with a specific delimiter set is completely
> broken. Tests were done inside Debian Installer.

I've tested the final patch by Goswin and can find no more issues or 
differences with GNU sort.
In some cases the use of -s is required to obtain the same output, but 
only in cases where the sort order is undetermined, which -s is exactly 
intended to solve.

Attached is a final version of the sort testsuite script. It includes all 
original tests (with some additional comments and minor changes) and a 
series of new tests that better cover real use cases.
All tests now PASS using both GNU sort and Busybox sort after Goswin's 
final patch has been applied.

Cheers,
FJP

[sort.tests (application/x-shellscript, attachment)]
[Message part 3 (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#373704; Package busybox. Full text and rfc822 format available.

Acknowledgement sent to Steve Langasek <vorlon@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #96 received at 373704@bugs.debian.org (full text, mbox):

From: Steve Langasek <vorlon@debian.org>
To: Samuel Thibault <samuel.thibault@ens-lyon.org>, Frans Pop <elendil@planet.nl>
Cc: 373704@bugs.debian.org, debian-release@lists.debian.org, Goswin Brederlow <brederlo@informatik.uni-tuebingen.de>
Subject: Re: Bug#373704: sort -k does not count fields the same as gnu sort
Date: Fri, 19 Jan 2007 03:19:26 -0800
severity 373704 important
thanks

On Sun, Jan 14, 2007 at 02:04:11AM +0100, Samuel Thibault wrote:
> Steve Langasek, le Sat 13 Jan 2007 13:10:25 -0800, a écrit :
> > "Potentially leading to random breakage" doesn't really justify a critical
> > severity when there are a limited number of packages making use of busybox
> > (and busybox sort in particular).  Does this bug actually break d-i, and if
> > so how?

> It used to break the execution order of some scripts.  Currently it
> doesn't, but it could very well break again.

Well, if it currently doesn't then that seems to not be a reason to treat it
as RC.

On Sun, Jan 14, 2007 at 03:39:09AM +0100, Frans Pop wrote:
> On Saturday 13 January 2007 22:10, Steve Langasek wrote:
> > "Potentially leading to random breakage" doesn't really justify a
> > critical severity when there are a limited number of packages making
> > use of busybox (and busybox sort in particular).  Does this bug
> > actually break d-i, and if so how?

> Let me start with a little quote:
> <ifvoid> fjp: why is it rc?
> <fjp> ifvoid: Basically because it can cause unexpected breakage in 
> anything that uses busybox sort.
> <fjp> ifvoid: My feeling is that the behavior now is even worse than the 
> original bug.
> <fjp> ifvoid: Not sure if RMs would agree though.
> <aj> unexpected breakage in busybox sort sounds horrible
> * Maulkin nods

True, unexpected breakage is horrible, but currently the only material
breakage seems to not only be expected, but worked around.  It may make
busybox sort unsuitable as a POSIX sort replacement, but that's simply not
anything that needs to be release-critical.

> "Critical" is possibly too high if you just consider Debian packages. 
> However, if you also consider use of busybox in embedded devices where 
> scripts written for GNU sort are being run (which apparently is a big 
> market for Debian), it could still be justified.
> It can at least potentially break unrelated software.

- software that invokes busybox isn't unrelated
- folks that are changing the Debian base system by replacing bash and/or
  coreutils with busybox are bound to do their own regression testing when
  deploying such changes; we shouldn't expect that every such bug that may
  affect people who are *modifying* Debian be treated as release-critical
  for Debian itself.

> I think "not suitable for release" is quite justified in this case, 
> especially as the sort function in Sarge works better. The original fix 
> for this bug has caused sorting with a delimiter (-t option) to work 
> worse than it did.

I disagree about it being unsuitable for release, but feel free to NMU for
this, a patch seems to be available now and I have no problem letting such
a fix through into etch.

Thanks,
-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
vorlon@debian.org                                   http://www.debian.org/



Severity set to `important' from `critical' Request was from Steve Langasek <vorlon@debian.org> to control@bugs.debian.org. Full text and rfc822 format available.

Reply sent to Bastian Blank <waldi@debian.org>:
You have taken responsibility. Full text and rfc822 format available.

Notification sent to Joey Hess <joeyh@debian.org>:
Bug acknowledged by developer. Full text and rfc822 format available.

Message #103 received at 373704-close@bugs.debian.org (full text, mbox):

From: Bastian Blank <waldi@debian.org>
To: 373704-close@bugs.debian.org
Subject: Bug#373704: fixed in busybox 1:1.1.3-4
Date: Sun, 21 Jan 2007 15:47:03 +0000
Source: busybox
Source-Version: 1:1.1.3-4

We believe that the bug you reported is fixed in the latest version of
busybox, which is due to be installed in the Debian FTP archive:

busybox-floppy-udeb_1.1.3-4_i386.udeb
  to pool/main/b/busybox/busybox-floppy-udeb_1.1.3-4_i386.udeb
busybox-static_1.1.3-4_i386.deb
  to pool/main/b/busybox/busybox-static_1.1.3-4_i386.deb
busybox-udeb_1.1.3-4_i386.udeb
  to pool/main/b/busybox/busybox-udeb_1.1.3-4_i386.udeb
busybox_1.1.3-4.diff.gz
  to pool/main/b/busybox/busybox_1.1.3-4.diff.gz
busybox_1.1.3-4.dsc
  to pool/main/b/busybox/busybox_1.1.3-4.dsc
busybox_1.1.3-4_i386.deb
  to pool/main/b/busybox/busybox_1.1.3-4_i386.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 373704@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Bastian Blank <waldi@debian.org> (supplier of updated busybox package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.7
Date: Sun, 21 Jan 2007 15:29:02 +0000
Source: busybox
Binary: busybox-floppy-udeb busybox busybox-udeb busybox-static
Architecture: source i386
Version: 1:1.1.3-4
Distribution: unstable
Urgency: low
Maintainer: Debian Install System Team <debian-boot@lists.debian.org>
Changed-By: Bastian Blank <waldi@debian.org>
Description: 
 busybox    - Tiny utilities for small and embedded systems
 busybox-floppy-udeb - Tiny utilities for the debian-installer floppy images (udeb)
 busybox-static - Standalone rescue shell with tons of builtin utilities
 busybox-udeb - Tiny utilities for the debian-installer (udeb)
Closes: 373704
Changes: 
 busybox (1:1.1.3-4) unstable; urgency=low
 .
   * coreutils/sort.c:
     - Try again to fix sorting issues. Patch by Goswin Brederlow.
       (closes: #373704)
Files: 
 5e9a02a1bdba93f487cad64a2d430f28 714 - optional busybox_1.1.3-4.dsc
 d60d3876c7ccf091018e7c23d93f2a79 20971 - optional busybox_1.1.3-4.diff.gz
 8627944f9615f8c20694bbf26a62176f 286364 utils optional busybox_1.1.3-4_i386.deb
 3edd24e8a0b0e4ce9ba169dfc73ca504 672274 shells optional busybox-static_1.1.3-4_i386.deb
 9720bdf3ed2e990ecb78c16dce0f5c01 118904 debian-installer extra busybox-udeb_1.1.3-4_i386.udeb
 06ba09270a638680d6dfe189c43c0590 45926 debian-installer extra busybox-floppy-udeb_1.1.3-4_i386.udeb
Package-Type: udeb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iEYEARECAAYFAkWziHwACgkQLkAIIn9ODhHpUwCgmH5auepwAo0ZqSYOBmCs5rIT
fhkAn3Vi0jfyK3lqtxF7vDvhtDeNgP50
=t0Fe
-----END PGP SIGNATURE-----




Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Sun, 24 Jun 2007 10:20:41 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Wed Apr 16 19:05:56 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.