Debian Bug report logs - #686502
busybox unxz silently fails to decompress multiple compressed streams on input

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: submit@bugs.debian.org

Subject: pxz produces archives broken for busybox's unxz

Date: Sun, 2 Sep 2012 15:18:40 +0200

package: pxz
version: 4.999.9~beta+git537418b-1
severity: important
affects: busybox
x-debbugs-cc: debian-boot@lists.debian.org,xz-utils@packages.debian.org,jnovy@users.sourceforge.net

Hi,

pxz (somtimes) produces archives broken for busybox's unxz, while they 
decompress fine with unxz from xz-utils packges. I noticed when trying
to uncompress pxz compressed initramfs files, while this is an universal
way to reproduce it:

(this is sid, and busybox from squeeze behaves the same.)

# get some big archive:
~/t$ apt-get source typo3
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Picking 'typo3-src' as source package instead of 'typo3'
NOTICE: 'typo3-src' packaging is maintained in the 'Git' version control system at:
git://github.com/sir-gawain/debian-typo3.git
Skipping already downloaded file 'typo3-src_4.5.19+dfsg1-1.dsc'
Skipping already downloaded file 'typo3-src_4.5.19+dfsg1.orig.tar.gz'
Skipping already downloaded file 'typo3-src_4.5.19+dfsg1-1.debian.tar.gz'
Need to get 0 B of source archives.
Skipping unpack of already unpacked source in typo3-src-4.5.19+dfsg1

# preserve it
~/t$ cp typo3-src_4.5.19+dfsg1.orig.tar.gz typo3-src_4.5.19+dfsg1.orig.tar.gz.orig

# now show that busybox unxz chokes:
~/t$ gunzip typo3-src_4.5.19+dfsg1.orig.tar.gz
~/t$ pxz -z typo3-src_4.5.19+dfsg1.orig.tar
~/t$ busybox unxz typo3-src_4.5.19+dfsg1.orig.tar.xz
~/t$ tar tf typo3-src_4.5.19+dfsg1.orig.tar 1>/dev/null
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

# cleanup
~/t$ rm typo3-src_4.5.19+dfsg1.orig.tar
~/t$ cp typo3-src_4.5.19+dfsg1.orig.tar.gz.orig typo3-src_4.5.19+dfsg1.orig.tar.gz

# show that unxz has no problems
~/t$ gunzip typo3-src_4.5.19+dfsg1.orig.tar.gz
~/t$ pxz -z typo3-src_4.5.19+dfsg1.orig.tar
~/t$ unxz typo3-src_4.5.19+dfsg1.orig.tar.xz
~/t$ tar tf typo3-src_4.5.19+dfsg1.orig.tar 1>/dev/null
~/t$ echo $?
0

Any help with this is appreciated.


cheers,
	Holger

Message #10 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Bastian Blank <waldi@debian.org>

To: Holger Levsen <holger@layer-acht.org>, 686502@bugs.debian.org

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Sun, 2 Sep 2012 15:32:03 +0200

On Sun, Sep 02, 2012 at 03:18:40PM +0200, Holger Levsen wrote:
> ~/t$ busybox unxz typo3-src_4.5.19+dfsg1.orig.tar.xz
> ~/t$ tar tf typo3-src_4.5.19+dfsg1.orig.tar 1>/dev/null
> tar: Unexpected EOF in archive
> tar: Error is not recoverable: exiting now

Where does busybox choke here? I only see tar choking on the result of
the decompression. What do you find in the file?

Bastian

-- 
Beam me up, Scotty, there's no intelligent life down here!

Message #15 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: 686502@bugs.debian.org

Cc: debian-boot@lists.debian.org

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Sun, 2 Sep 2012 16:34:42 +0200

Hi Bastian,

On Sonntag, 2. September 2012, Bastian Blank wrote:
> On Sun, Sep 02, 2012 at 03:18:40PM +0200, Holger Levsen wrote:
> > ~/t$ busybox unxz typo3-src_4.5.19+dfsg1.orig.tar.xz
> > ~/t$ tar tf typo3-src_4.5.19+dfsg1.orig.tar 1>/dev/null
> > tar: Unexpected EOF in archive
> > tar: Error is not recoverable: exiting now
> 
> Where does busybox choke here? I only see tar choking on the result of
> the decompression. What do you find in the file?

understandable, my instructions to reproduce it hide the problem a bit ;)

tar fails, because "busybox unxz" fails to decompress correctly (as can be 
seen in the filesize) despite exiting with exit code 0:

~/t$ gunzip typo3-src_4.5.19+dfsg1.orig.tar.gz
~/t$ ls -l typo3-src_4.5.19+dfsg1.orig.tar   
-rw-r--r-- 1 me me 51845120 Aug 18 05:36 typo3-src_4.5.19+dfsg1.orig.tar
~/t$ pxz -z typo3-src_4.5.19+dfsg1.orig.tar
~/t$ busybox unxz typo3-src_4.5.19+dfsg1.orig.tar.xz 
~/t$ echo $?
0
~/t$ ls -l typo3-src_4.5.19+dfsg1.orig.tar   
-rw-r--r-- 1 me me 25169920 Sep  2 14:32 typo3-src_4.5.19+dfsg1.orig.tar


cheers,
	Holger

Message #20 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Jonathan Nieder <jrnieder@gmail.com>

To: Holger Levsen <holger@layer-acht.org>

Cc: 686502@bugs.debian.org, busybox@packages.debian.org

Subject: Re: pxz produces archives broken for busybox's unxz

Date: Sun, 2 Sep 2012 12:54:29 -0700

Hi Holger,

Holger Levsen wrote:

> pxz (somtimes) produces archives broken for busybox's unxz, while they 
> decompress fine with unxz from xz-utils packges. I noticed when trying
> to uncompress pxz compressed initramfs files, while this is an universal
> way to reproduce it:

Thanks!  Could you please attach such an example archive, or upload it
somewhere and send a link if it is too large?

Lazily,
Jonathan

Message #25 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: Jonathan Nieder <jrnieder@gmail.com>

Cc: 686502@bugs.debian.org

Subject: Re: pxz produces archives broken for busybox's unxz

Date: Mon, 3 Sep 2012 09:20:27 +0200

Hi Jonathan,

On Sonntag, 2. September 2012, Jonathan Nieder wrote:
> Thanks!  Could you please attach such an example archive, or upload it
> somewhere and send a link if it is too large?

the typo3 orig.tar.gz as it is in squeeze or sid is sufficient: gunzip it and 
compress it with pxz and "voila"...


cheers,
	Holger

Message #30 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Jonathan Nieder <jrnieder@gmail.com>

To: Holger Levsen <holger@layer-acht.org>

Cc: 686502@bugs.debian.org, busybox@packages.debian.org

Subject: Re: pxz produces archives broken for busybox's unxz

Date: Wed, 5 Sep 2012 01:14:45 -0700

[Message part 1 (text/plain, inline)]

Hi,

Holger Levsen wrote:

> the typo3 orig.tar.gz as it is in squeeze or sid is sufficient: gunzip it and 
> compress it with pxz and "voila"...

*builds pxz, tests*

Luckily the cause seems to be simple.  pxz works by concatenating
whole XZ files, but for simplicity busybox unxz ignores everything
past the end of the first stream it reads.  Meanwhile standard xz
handles concatenated streams by concatenating their decompressed form.

 $ xz --robot -v -l broken.tar.xz | grep stream
 stream	1	1	0	0	11083380	25169920	0.440	CRC64	0
 stream	2	1	11083380	25169920	5431156	25169920	0.216	CRC64	0
 stream	3	1	16514536	50339840	267724	1505280	0.178	CRC64	0
 $ busybox unxz -c <broken.tar.xz | wc -c
 25169920
 $ xz -dc <broken.tar.xz | wc -c
 51845120

This could be avoided by putting the compressed chunks into Blocks
instead of Streams, but that requires deeper knowledge of the file
format (see /usr/share/doc/liblzma-dev/xz-file-format.txt).  Or
perhaps busybox unxz should learn about concatenated streams.

Simplified example compressed file to play with attached.

	$ printf he | xz >test.xz
	$ echo llo | xz >>test.xz
	$ unxz <test.xz
	hello
	$ busybox unxz <test.xz
	he$

Have fun,
Jonathan

[concatenated.xz (application/octet-stream, attachment)]

Message #35 received at 686502@bugs.debian.org (full text, mbox, reply):

From: David Bremner <bremner@debian.org>

To: 686502@bugs.debian.org

Subject: picz

Date: Wed, 21 Nov 2012 07:35:05 -0400

FYI, I just filed an ITP for picz (#693859) mainly because it avoids
this bug.  At least, it seemed to in my testing (and the design
should). More testing would be very welcome.

d

Message #40 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: David Bremner <bremner@debian.org>, 686502@bugs.debian.org

Subject: Re: Bug#686502: picz

Date: Wed, 21 Nov 2012 17:57:48 +0100

Hi David,

On Mittwoch, 21. November 2012, David Bremner wrote:
> FYI, I just filed an ITP for picz (#693859) mainly because it avoids
> this bug.  At least, it seemed to in my testing (and the design
> should). More testing would be very welcome.

so once picz is in the archive, let's remove pxz?


cheers,
	Holger

Message #45 received at 686502@bugs.debian.org (full text, mbox, reply):

From: David Bremner <bremner@debian.org>

To: Holger Levsen <holger@layer-acht.org>, 686502@bugs.debian.org

Subject: Re: Bug#686502: picz

Date: Wed, 21 Nov 2012 17:13:47 -0400

Holger Levsen <holger@layer-acht.org> writes:

>
> so once picz is in the archive, let's remove pxz?
>

my bad spelling, it's actually pixz.

I'm not sure (yet). It seems pixz is not able to compress quite as much
(by around 15% in one experiment). So people might like to have both?

At a shallow read, concatenation is defined in the xz format
(http://tukaani.org/xz/format.html), so I wonder if this bug is not
really a problem with busybox?

All the best, 

David

Message #50 received at 686502@bugs.debian.org (full text, mbox, reply):

From: David Bremner <bremner@debian.org>

To: 686502@bugs.debian.org

Subject: multi-stream XZ

Date: Sun, 25 Nov 2012 09:20:31 -0400

I had another look at the XZ standard (http://tukaani.org/xz/xz-file-format-1.0.4.txt)

  While a typical file contains only one Stream and no Stream Padding, a
  decoder handling standalone .xz files SHOULD support files that have
  more than one Stream or Stream Padding.

My interpretation is that this bug is a wishlist bug in busybox, that
affects pxz. I leave it to the maintainer or reporter to reassign if
that makes sense.

Message #55 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Michael Tokarev <mjt@tls.msk.ru>

To: 686502@bugs.debian.org

Cc: David Bremner <bremner@debian.org>, Holger Levsen <holger@layer-acht.org>, Jonathan Nieder <jrnieder@gmail.com>, Bastian Blank <waldi@debian.org>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Thu, 20 Dec 2012 12:22:12 +0400

Control: forwarded -1 https://bugs.busybox.net/show_bug.cgi?id=5804
Control: reassign -1 src:busybox 1:1.17.1-8
Control: affects -1 pxz
Control: retitle -1 busybox unxz silently fails to decompress multiple compressed streams on input
Control: severity -1 grave

This is a grave bug in busybox.  Grave because it causes silent
data loss - valid (according to the format specs) input is
decompressed only partially.

Oh well.

/mjt

Message #72 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Bastian Blank <waldi@debian.org>

To: Michael Tokarev <mjt@tls.msk.ru>

Cc: 686502@bugs.debian.org, David Bremner <bremner@debian.org>, Holger Levsen <holger@layer-acht.org>, Jonathan Nieder <jrnieder@gmail.com>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Thu, 20 Dec 2012 12:30:37 +0100

On Thu, Dec 20, 2012 at 12:22:12PM +0400, Michael Tokarev wrote:
> This is a grave bug in busybox.  Grave because it causes silent
> data loss - valid (according to the format specs) input is
> decompressed only partially.

The documentation say: "SHOULD support files that have more than one
Stream or Stream padding". Please note the should. So missing support is
no bug on its own, but it should at least check that there is no
trailing garbage.

Bastian

-- 
	"What terrible way to die."
	"There are no good ways."
		-- Sulu and Kirk, "That Which Survives", stardate unknown

Message #77 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Abou Al Montacir <abou.almontacir@sfr.fr>

To: 686502@bugs.debian.org

Cc: Philipp Kern <pkern@debian.org>, Holger Levsen <holger@layer-acht.org>, Bastian Blank <waldi@debian.org>, Jonathan Nieder <jrnieder@gmail.com>, David Bremner <bremner@debian.org>, MichaelTokarev <mjt@tls.msk.ru>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Thu, 20 Dec 2012 22:42:41 +0100

[Message part 1 (text/plain, inline)]

Hi,

Can you please test the attached patch

Cheers,

[Message part 2 (text/html, inline)]

[busybox_1.20.0-7.1.debdiff (text/x-patch, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #82 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Bastian Blank <waldi@debian.org>

To: Abou Al Montacir <abou.almontacir@sfr.fr>

Cc: 686502@bugs.debian.org, Philipp Kern <pkern@debian.org>, Holger Levsen <holger@layer-acht.org>, Jonathan Nieder <jrnieder@gmail.com>, David Bremner <bremner@debian.org>, MichaelTokarev <mjt@tls.msk.ru>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Thu, 20 Dec 2012 23:08:35 +0100

On Thu, Dec 20, 2012 at 10:42:41PM +0100, Abou Al Montacir wrote:
> Can you please test the attached patch

How does it implement stream padding?

Bastian

-- 
What kind of love is that?  Not to be loved; never to have shown love.
		-- Commissioner Nancy Hedford, "Metamorphosis",
		   stardate 3219.8

Message #87 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Abou Al Montacir <abou.almontacir@sfr.fr>

To: Bastian Blank <waldi@debian.org>

Cc: 686502@bugs.debian.org, Philipp Kern <pkern@debian.org>, Holger Levsen <holger@layer-acht.org>, Jonathan Nieder <jrnieder@gmail.com>, David Bremner <bremner@debian.org>, MichaelTokarev <mjt@tls.msk.ru>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Fri, 21 Dec 2012 14:06:31 +0100

[Message part 1 (text/plain, inline)]

On Thu, 2012-12-20 at 23:08 +0100, Bastian Blank wrote:
> On Thu, Dec 20, 2012 at 10:42:41PM +0100, Abou Al Montacir wrote:
> > Can you please test the attached patch
> 
> How does it implement stream padding?

Hi Bastian,

As it is implemented, it will iterate until end of stream, but I did not
test this particular case.

If you can provide an example of files on which it will fail, I can try
to fix it.

Cheers,

[signature.asc (application/pgp-signature, inline)]

Message #92 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Michael Tokarev <mjt@tls.msk.ru>

To: Abou Al Montacir <abou.almontacir@sfr.fr>

Cc: Bastian Blank <waldi@debian.org>, 686502@bugs.debian.org, Philipp Kern <pkern@debian.org>, Holger Levsen <holger@layer-acht.org>, Jonathan Nieder <jrnieder@gmail.com>, David Bremner <bremner@debian.org>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Fri, 21 Dec 2012 17:13:27 +0400

21.12.2012 17:06, Abou Al Montacir wrote:
> On Thu, 2012-12-20 at 23:08 +0100, Bastian Blank wrote:
>> On Thu, Dec 20, 2012 at 10:42:41PM +0100, Abou Al Montacir wrote:
>>> Can you please test the attached patch
>>
>> How does it implement stream padding?
>
> Hi Bastian,
>
> As it is implemented, it will iterate until end of stream, but I did not
> test this particular case.

Actually it is not:

+			if (iobuf.in_pos == iobuf.in_size) {
+				break;
+			} else {

iobuf is what we've in memory.  We may've read some data which
ends in buffer exactly at the end of the stream.  There might
be next stream coming, but for it we may need to read a few
more bytes first...  At least if I read the code correctly.

It is sorta like testing if we reached end of file by testing
whenever we're at the end of stdio buffer.

Thanks,

/mjt

Message #97 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Bastian Blank <waldi@debian.org>

To: Abou Al Montacir <abou.almontacir@sfr.fr>, 686502@bugs.debian.org

Cc: Philipp Kern <pkern@debian.org>, Holger Levsen <holger@layer-acht.org>, Jonathan Nieder <jrnieder@gmail.com>, David Bremner <bremner@debian.org>, MichaelTokarev <mjt@tls.msk.ru>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Fri, 21 Dec 2012 14:24:05 +0100

On Fri, Dec 21, 2012 at 02:06:31PM +0100, Abou Al Montacir wrote:
> On Thu, 2012-12-20 at 23:08 +0100, Bastian Blank wrote:
> > On Thu, Dec 20, 2012 at 10:42:41PM +0100, Abou Al Montacir wrote:
> > > Can you please test the attached patch
> > How does it implement stream padding?
> As it is implemented, it will iterate until end of stream, but I did not
> test this particular case.

I ask you how it implements a mandantory feature and you are not able to
tell me?

> If you can provide an example of files on which it will fail, I can try
> to fix it.

Add stream padding as specified in the spec.

Bastian

-- 
There's a way out of any cage.
		-- Captain Christopher Pike, "The Menagerie" ("The Cage"),
		   stardate unknown.

Message #102 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Abou Al Montacir <abou.almontacir@sfr.fr>

To: Michael Tokarev <mjt@tls.msk.ru>

Cc: Bastian Blank <waldi@debian.org>, 686502@bugs.debian.org, Philipp Kern <pkern@debian.org>, Holger Levsen <holger@layer-acht.org>, Jonathan Nieder <jrnieder@gmail.com>, David Bremner <bremner@debian.org>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Fri, 21 Dec 2012 14:29:00 +0100

[Message part 1 (text/plain, inline)]

On Fri, 2012-12-21 at 17:13 +0400, Michael Tokarev wrote:
> 21.12.2012 17:06, Abou Al Montacir wrote:
> > On Thu, 2012-12-20 at 23:08 +0100, Bastian Blank wrote:
> >> On Thu, Dec 20, 2012 at 10:42:41PM +0100, Abou Al Montacir wrote:
> >>> Can you please test the attached patch
> >>
> >> How does it implement stream padding?
> >
> > Hi Bastian,
> >
> > As it is implemented, it will iterate until end of stream, but I did not
> > test this particular case.
> 
> Actually it is not:
> 
> +			if (iobuf.in_pos == iobuf.in_size) {
> +				break;
> +			} else {
> 
> iobuf is what we've in memory.  We may've read some data which
> ends in buffer exactly at the end of the stream.  There might
> be next stream coming, but for it we may need to read a few
> more bytes first...  At least if I read the code correctly.
> 
> It is sorta like testing if we reached end of file by testing
> whenever we're at the end of stdio buffer.

Hi Michael,

Good catch! I'll fix that by replacing that break by continue, so it
will read on next iteration and will break for end of file.

I'll submit a new patch soon.

Cheers,

[signature.asc (application/pgp-signature, inline)]

Message #107 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Abou Al Montacir <abou.almontacir@sfr.fr>

To: Bastian Blank <waldi@debian.org>

Cc: 686502@bugs.debian.org, Philipp Kern <pkern@debian.org>, Holger Levsen <holger@layer-acht.org>, Jonathan Nieder <jrnieder@gmail.com>, David Bremner <bremner@debian.org>, MichaelTokarev <mjt@tls.msk.ru>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Fri, 21 Dec 2012 14:34:53 +0100

[Message part 1 (text/plain, inline)]

On Fri, 2012-12-21 at 14:24 +0100, Bastian Blank wrote:
> On Fri, Dec 21, 2012 at 02:06:31PM +0100, Abou Al Montacir wrote:
> > On Thu, 2012-12-20 at 23:08 +0100, Bastian Blank wrote:
> > > On Thu, Dec 20, 2012 at 10:42:41PM +0100, Abou Al Montacir wrote:
> > > > Can you please test the attached patch
> > > How does it implement stream padding?
> > As it is implemented, it will iterate until end of stream, but I did not
> > test this particular case.
> 
> I ask you how it implements a mandantory feature and you are not able to
> tell me?
As I told you it will iterate on it and consume it until it finds a new valid header.
A smarter way is to eat zeros until next non zero data. Please c.f §2.2
on spec R1.0.4

> > If you can provide an example of files on which it will fail, I can try
> > to fix it.
> 
> Add stream padding as specified in the spec.
I'll provide a new patch for eating zeros and fixing issue pointed by Michael

Cheers,

[signature.asc (application/pgp-signature, inline)]

Message #112 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Abou Al Montacir <abou.almontacir@sfr.fr>

To: Bastian Blank <waldi@debian.org>

Cc: 686502@bugs.debian.org, Philipp Kern <pkern@debian.org>, Holger Levsen <holger@layer-acht.org>, Jonathan Nieder <jrnieder@gmail.com>, David Bremner <bremner@debian.org>, MichaelTokarev <mjt@tls.msk.ru>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Sat, 22 Dec 2012 00:03:31 +0100

[Message part 1 (text/plain, inline)]

On Fri, 2012-12-21 at 14:34 +0100, Abou Al Montacir wrote:
> > Add stream padding as specified in the spec.
> I'll provide a new patch for eating zeros and fixing issue pointed by Michael

Please find attached new patch handling padding and fixing issue
highlighted by Michael,

Cheers,

[busybox_1.20.0-7.1.debdiff (text/x-patch, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #117 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Jonathan Nieder <jrnieder@gmail.com>

To: Abou Al Montacir <abou.almontacir@sfr.fr>

Cc: 686502@bugs.debian.org, pxz@packages.debian.org

Subject: Re: pxz produces archives broken for busybox's unxz

Date: Sat, 22 Dec 2012 10:21:52 -0800

Abou Al Montacir wrote:

> +--- busybox-1.20.0~/archival/libarchive/decompress_unxz.c	2012-12-20 21:51:04.000000000 +0100
> ++++ busybox-1.20.0/archival/libarchive/decompress_unxz.c	2012-12-20 21:49:11.000000000 +0100
> +@@ -87,7 +87,17 @@ unpack_xz_stream(transformer_aux_data_t *aux, int src_fd, int dst_fd)
> + 			iobuf.out_pos = 0;
> + 		}
> + 		if (r == XZ_STREAM_END) {
> +-			break;
> ++			if (iobuf.in_pos != iobuf.in_size) {
[...]
> ++			}
> ++			// Look for other streams
> ++			continue;

What happens if a stream ends at a buffer boundary, followed by
padding?  Or if padding doesn't fit in the buffer, for that
matter?

Hope that helps,
Jonathan

Message #122 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Abou Al Montacir <abou.almontacir@sfr.fr>

To: Jonathan Nieder <jrnieder@gmail.com>

Cc: 686502@bugs.debian.org, pxz@packages.debian.org

Subject: Re: pxz produces archives broken for busybox's unxz

Date: Mon, 24 Dec 2012 11:01:41 +0100

[Message part 1 (text/plain, inline)]

On Sat, 2012-12-22 at 10:21 -0800, Jonathan Nieder wrote:
> What happens if a stream ends at a buffer boundary, followed by
> padding?  Or if padding doesn't fit in the buffer, for that
> matter?
That make very low probability but could happe indeed. I will upload a
new patch which fixes this case too.

Thank you for your review.

Cheers,

[signature.asc (application/pgp-signature, inline)]

Message #127 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Abou Al Montacir <abou.almontacir@sfr.fr>

To: Jonathan Nieder <jrnieder@gmail.com>

Cc: 686502@bugs.debian.org, pxz@packages.debian.org

Subject: Re: pxz produces archives broken for busybox's unxz

Date: Mon, 24 Dec 2012 23:42:55 +0100

[Message part 1 (text/plain, inline)]

On Sat, 2012-12-22 at 10:21 -0800, Jonathan Nieder wrote:
> 
> What happens if a stream ends at a buffer boundary, followed by
> padding?  Or if padding doesn't fit in the buffer, for that
> matter?
> 
> Hope that helps,

Please find attached new debdiff with fix of above mentioned issues.

Cheers,

[busybox_1.20.0-7.1.debdiff (text/x-patch, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #132 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Jonathan Nieder <jrnieder@gmail.com>

To: Abou Al Montacir <abou.almontacir@sfr.fr>

Cc: 686502@bugs.debian.org, pxz@packages.debian.org

Subject: Re: pxz produces archives broken for busybox's unxz

Date: Mon, 24 Dec 2012 17:11:43 -0800

Abou Al Montacir wrote:
> On Sat, 2012-12-22 at 10:21 -0800, Jonathan Nieder wrote:

>> What happens if a stream ends at a buffer boundary, followed by
>> padding?  Or if padding doesn't fit in the buffer, for that
>> matter?
[...]
> Please find attached new debdiff with fix of above mentioned issues.

Getting closer.  Does this correctly handle the case of a file with
zero streams?  (It should error out.)  How about a file with leading
NUL bytes, which is also invalid?

Does this implementation meet the following requirement (from the
spec)?

| Stream Padding MUST contain only null bytes. To preserve the
| four-byte alignment of consecutive Streams, the size of Stream
| Padding MUST be a multiple of four bytes. Empty Stream Padding
| is allowed. If these requirements are not met, the decoder MUST
| indicate an error.

Thanks for your patient work.
Jonathan

Message #137 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Abou Al Montacir <abou.almontacir@sfr.fr>

To: Jonathan Nieder <jrnieder@gmail.com>

Cc: 686502@bugs.debian.org, pxz@packages.debian.org

Subject: Re: pxz produces archives broken for busybox's unxz

Date: Thu, 27 Dec 2012 12:24:55 +0100

[Message part 1 (text/plain, inline)]

On Mon, 2012-12-24 at 17:11 -0800, Jonathan Nieder wrote:
> Abou Al Montacir wrote:
> > On Sat, 2012-12-22 at 10:21 -0800, Jonathan Nieder wrote:
> 
> >> What happens if a stream ends at a buffer boundary, followed by
> >> padding?  Or if padding doesn't fit in the buffer, for that
> >> matter?
> [...]
> > Please find attached new debdiff with fix of above mentioned issues.
> 
> Getting closer.  Does this correctly handle the case of a file with
> zero streams?  (It should error out.)  How about a file with leading
Let's see:
We will read buffer, do no find zeros and go into decoder and then issue
an error.
> NUL bytes, which is also invalid?
It will remove the zeros and then start decoding. I agree the file is
invalid, but I don't think it could harm to decode any valid stream
inside.

> Does this implementation meet the following requirement (from the
> spec)?
> 
> | Stream Padding MUST contain only null bytes. To preserve the
> | four-byte alignment of consecutive Streams, the size of Stream
> | Padding MUST be a multiple of four bytes. Empty Stream Padding
> | is allowed. If these requirements are not met, the decoder MUST
> | indicate an error.
Clearly it does not met this but could be done assuming few extra ifs.
Hover, I assume we can save this extra code as soon as we don't loose
data. My goal was more to avoid data loss for user rather than providing
a specification conformant decoder. I just want to ensure that a user
decoding a valid .xz file does not loose any data. If the decoder is
more tolerant than the standard, my goal is met.

Now if RT requires to have a full standard conformant decoder inside
busybox, I can do it.

> Thanks for your patient work.
Thank you for your careful review.

Cheers,

[signature.asc (application/pgp-signature, inline)]

Message #142 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Jonathan Nieder <jrnieder@gmail.com>

To: Abou Al Montacir <abou.almontacir@sfr.fr>

Cc: 686502@bugs.debian.org, pxz@packages.debian.org

Subject: Re: pxz produces archives broken for busybox's unxz

Date: Thu, 27 Dec 2012 08:38:53 -0800

Abou Al Montacir wrote:

> Hover, I assume we can save this extra code as soon as we don't loose
> data.

That's fine with me.  All you'd need to do is error out if there is
anything after the first stream.  That would make it a conformant
decoder and prevent silent data loss, though it would mean busybox
couldn't read the XZ files pxz produces.

(Context: the spec permits single-stream decoders because there are
decoders in the wild that need to be very simple, like the one built
into the Linux kernel that unpacks the kernel and initramfs.)

On the other hand, if busybox is to start decoding concatenated
streams (imitating the standard "xz" command), then the spec requires
also correctly implementing padding.  This might sound rigid, but it
is important for interoperability --- without such requirements,
whenever you share XZ files there would be a lot of confusion about
whether it is valid and which implementations can and can't decode it.

I think busybox upstream would agree that the spec shouldn't just be
ignored.

Hoping that clarifies,
Jonathan

Message #147 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Abou Al Montacir <abou.almontacir@sfr.fr>

To: Jonathan Nieder <jrnieder@gmail.com>

Cc: 686502@bugs.debian.org, pxz@packages.debian.org

Subject: Re: pxz produces archives broken for busybox's unxz

Date: Thu, 27 Dec 2012 22:08:07 +0100

[Message part 1 (text/plain, inline)]

On Thu, 2012-12-27 at 08:38 -0800, Jonathan Nieder wrote:
> Abou Al Montacir wrote:
> 
> > Hover, I assume we can save this extra code as soon as we don't loose
> > data.
> 
> That's fine with me.  All you'd need to do is error out if there is
> anything after the first stream.  That would make it a conformant
> decoder and prevent silent data loss, though it would mean busybox
> couldn't read the XZ files pxz produces.
Sure,

> (Context: the spec permits single-stream decoders because there are
> decoders in the wild that need to be very simple, like the one built
> into the Linux kernel that unpacks the kernel and initramfs.)
> 
> On the other hand, if busybox is to start decoding concatenated
> streams (imitating the standard "xz" command), then the spec requires
> also correctly implementing padding.  This might sound rigid, but it
> is important for interoperability --- without such requirements,
> whenever you share XZ files there would be a lot of confusion about
> whether it is valid and which implementations can and can't decode it.
Agree

> I think busybox upstream would agree that the spec shouldn't just be
> ignored.
You convinced me here. I admit you're right the we should either conform
to spec or conform to spec, no choice there.

I've fixed my patch and think that know it should really be conformant.
I also attached some short samples to be tested. One of them only should
fail to decode.

I really thank you for your support and review as well as for your sense
of details. I admit I've learned from you many things.

Thanks,

[fix-unxz-with-multiple-streams.patch (text/x-patch, attachment)]

[hel0000.xz (application/x-xz, attachment)]

[hel000lo.xz (application/x-xz, attachment)]

[hel0000lo.xz (application/x-xz, attachment)]

[hello.xz (application/x-xz, attachment)]

[hello1.xz (application/x-xz, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #150 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Philipp Kern <pkern@debian.org>

To: 686502@bugs.debian.org

Cc: Jonathan Nieder <jrnieder@gmail.com>, pxz@packages.debian.org, Abou Al Montacir <abou.almontacir@sfr.fr>, Bastian Blank <waldi@debian.org>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Sun, 6 Jan 2013 20:50:01 +0100

[Message part 1 (text/plain, inline)]

Hi,

On Thu, Dec 27, 2012 at 10:08:07PM +0100, Abou Al Montacir wrote:
> I've fixed my patch and think that know it should really be conformant.
> I also attached some short samples to be tested. One of them only should
> fail to decode.

could somebody please review that patch and if suitable upload it?

Thanks :-)
Philipp Kern

[signature.asc (application/pgp-signature, inline)]

Message #155 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Bastian Blank <waldi@debian.org>

To: Abou Al Montacir <abou.almontacir@sfr.fr>

Cc: 686502@bugs.debian.org, Philipp Kern <pkern@debian.org>, Holger Levsen <holger@layer-acht.org>, Jonathan Nieder <jrnieder@gmail.com>, David Bremner <bremner@debian.org>, MichaelTokarev <mjt@tls.msk.ru>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Sun, 6 Jan 2013 21:40:00 +0100

On Sat, Dec 22, 2012 at 12:03:31AM +0100, Abou Al Montacir wrote:
> --- busybox-1.20.0/debian/patches/fix-unxz-with-multiple-streams.patch	1970-01-01 01:00:00.000000000 +0100
> +++ busybox-1.20.0/debian/patches/fix-unxz-with-multiple-streams.patch	2012-12-21 19:23:12.000000000 +0100
> @@ -0,0 +1,25 @@
> +Author: Abou Al Montacir <abou.almontacir@sfr.fr>
> +Purpose: Fix decompression of multi stream XZ compressed files
> + (Closes: bug#686502)
> +
> +--- busybox-1.20.0~/archival/libarchive/decompress_unxz.c	2012-12-20 21:51:04.000000000 +0100
> ++++ busybox-1.20.0/archival/libarchive/decompress_unxz.c	2012-12-20 21:49:11.000000000 +0100
> +@@ -87,7 +87,17 @@ unpack_xz_stream(transformer_aux_data_t *aux, int src_fd, int dst_fd)
> + 			iobuf.out_pos = 0;
> + 		}
> + 		if (r == XZ_STREAM_END) {
> +-			break;
> ++			if (iobuf.in_pos != iobuf.in_size) {
> ++				// Initialize decoder for new stream
> ++				xz_dec_end(state);
> ++				state = xz_dec_init(XZ_DYNALLOC, 64*1024*1024);

Why can't you use the existing call somewhere at the beginning? If I
remember correctly, you need 128*1024*1024 to decompress all valid
files.

> ++				// Eat padding
> ++				while (iobuf.in[iobuf.in_pos] == 0){
> ++					iobuf.in_pos += 1;
> ++				}

Padding is a multiple of _four_ bytes. Did you read the spec?

> ++			}
> ++			// Look for other streams
> ++			continue;

Does it bail out if there is no new stream?

Bastian

-- 
Men will always be men -- no matter where they are.
		-- Harry Mudd, "Mudd's Women", stardate 1329.8

Message #160 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Bastian Blank <waldi@debian.org>

To: Abou Al Montacir <abou.almontacir@sfr.fr>

Cc: 686502@bugs.debian.org, Philipp Kern <pkern@debian.org>, Holger Levsen <holger@layer-acht.org>, Jonathan Nieder <jrnieder@gmail.com>, David Bremner <bremner@debian.org>, MichaelTokarev <mjt@tls.msk.ru>

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Sun, 6 Jan 2013 23:26:19 +0100

On Sun, Jan 06, 2013 at 09:40:00PM +0100, Bastian Blank wrote:

This was the wrong mail.

Bastian

-- 
Oblivion together does not frighten me, beloved.
		-- Thalassa (in Anne Mulhall's body), "Return to Tomorrow",
		   stardate 4770.3.

Message #165 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Bastian Blank <waldi@debian.org>

To: Abou Al Montacir <abou.almontacir@sfr.fr>, 686502@bugs.debian.org

Cc: Jonathan Nieder <jrnieder@gmail.com>, pxz@packages.debian.org

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Sun, 6 Jan 2013 23:32:14 +0100

On Thu, Dec 27, 2012 at 10:08:07PM +0100, Abou Al Montacir wrote:
> +		if (r == XZ_STREAM_END) {
> +			/* Eat padding. Stream never starts with zeros, and padding is 32 aligned */
> +			while ((iobuf.in_pos < iobuf.in_size) && (iobuf.in[iobuf.in_pos] == 0)) {
> +					iobuf.in_pos += 1;
> +			}
> +			/* Reached end of buffer. Fill it again from stream */
> +			if (iobuf.in_pos == iobuf.in_size) {
> +				continue;
> +			}
> +			if(iobuf.in_pos % 4){

Are you sure this is correct? in_pos is the position in tht buffer, not
the file. Also look out for coding style.

> +		if (r == XZ_STREAM_END) {

Again the same check?

>  		if (r == XZ_STREAM_END) {
> -			break;
> +			xz_dec_end(state);
> +			/* Look for any other streams */
> +			continue;

Why do you have three XZ_STREAM_END checks in this state machine?

Bastian

-- 
There are always alternatives.
		-- Spock, "The Galileo Seven", stardate 2822.3

Message #170 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Abou Al Montacir <abou.almontacir@sfr.fr>

To: Bastian Blank <waldi@debian.org>

Cc: 686502@bugs.debian.org, Jonathan Nieder <jrnieder@gmail.com>, pxz@packages.debian.org

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Mon, 07 Jan 2013 10:23:32 +0100

[Message part 1 (text/plain, inline)]

On Sun, 2013-01-06 at 23:32 +0100, Bastian Blank wrote:
> On Thu, Dec 27, 2012 at 10:08:07PM +0100, Abou Al Montacir wrote:
> > +		if (r == XZ_STREAM_END) {
> > +			/* Eat padding. Stream never starts with zeros, and padding is 32 aligned */
> > +			while ((iobuf.in_pos < iobuf.in_size) && (iobuf.in[iobuf.in_pos] == 0)) {
> > +					iobuf.in_pos += 1;
> > +			}
> > +			/* Reached end of buffer. Fill it again from stream */
> > +			if (iobuf.in_pos == iobuf.in_size) {
> > +				continue;
> > +			}
> > +			if(iobuf.in_pos % 4){
> 
> Are you sure this is correct? in_pos is the position in tht buffer, not
> the file. Also look out for coding style.

Provided the buffer size is multiple of 4 bytes, which is the case for
BUFSIZ. Of course one can decide to use a mis aligned buffer, but this
is not common and I consider it bad coding practice.

> > +		if (r == XZ_STREAM_END) {
> 
> Again the same check?
Not really, r could have been changed since the above check (r = XZ_DATA_ERROR; when %4 check fails)

> >  		if (r == XZ_STREAM_END) {
> > -			break;
> > +			xz_dec_end(state);
> > +			/* Look for any other streams */
> > +			continue;
> 
> Why do you have three XZ_STREAM_END checks in this state machine?
I use XZ_STREAM_END to check end of stream and eat padding, to check
there is still valid data (no paddding error) before initializing
decoder, and finally to free the decoder at en of current stream.

Cheers,

[signature.asc (application/pgp-signature, inline)]

Message #175 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Philipp Kern <phil@philkern.de>

To: Abou Al Montacir <abou.almontacir@sfr.fr>, 686502@bugs.debian.org

Cc: Bastian Blank <waldi@debian.org>, Jonathan Nieder <jrnieder@gmail.com>, pxz@packages.debian.org

Subject: Re: Bug#686502: pxz produces archives broken for busybox's unxz

Date: Sat, 23 Feb 2013 17:07:02 +0100

Hi,

so is the latest patch by Abou acceptable? If the logic's ok I guess the
committer could also fix up the last bunch of coding style issues.

Kind regards
Philipp Kern

Message #180 received at 686502@bugs.debian.org (full text, mbox, reply):

From: Michael Tokarev <mjt@tls.msk.ru>

To: John Spencer <maillist-busybox@barfooze.de>

Cc: Lasse Collin <lasse.collin@tukaani.org>, busybox@busybox.net, 686502@bugs.debian.org

Subject: Re: XZ embedded bug unpacking linux-3.8.tar.xz

Date: Tue, 26 Feb 2013 10:43:41 +0400

26.02.2013 03:21, John Spencer wrote:
> [ quoting the full mail of lasse since it didnt make its way into the bb maillist yet ]

Additionally there has been a discussion and attempts to cook up a
patch in Debian, see http://bugs.debian.org/686502 , which I submitted
as a bug to busybox bugzilla -- https://bugs.busybox.net/show_bug.cgi?id=5804 .
Cc'ing the Debian bugreport.  I like the below patch better :)

/mjt

> On 02/25/2013 12:05 PM, Lasse Collin wrote:
>> On 2013-02-25 Denys Vlasenko wrote:
>>> On Sunday 24 February 2013 22:37, John Spencer wrote:
>>>>> http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.8.tar.xz
>>>>>
>>>>> using busybox 1.20.2 and xz 5.0.3 or xz 5.0.4:
>>>>>
>>>>> $ tar xf linux-3.8.tar.xz
>>>>>
>>>>> i get: "short read" and exit status 1.
>>>>> however the data seems to be there (at least partial).
>>>>
>>>> the culprit is the file linux-3.8/drivers/media/tuners/mt2063.c
>>>>
>>>> after doing xzcat linux-3.8.tar.xz>  linux-3.8.tar , that file is
>>>> truncated after 4096*2+512 bytes.
>>>>
>>>> xzcat is from busybox (not from xz, as i assumed earlier)
>>>>
>>>> the .tar file is truncated at this point as well, it is only 200
>>>> MB, but with xzcat from xz package, it is>  500 MB.
>>>
>>> Apparently XZ embedded has a bug :(
>>> Not only our in-tree one, but the latest git of it is buggy too:
>>>
>>> $ git clone http://git.tukaani.org/xz-embedded.git
>>> $ cd xz-embedded/userspace
>>> $ make
>>> $ ./xzminidec</tmp/linux-3.8.tar.xz | wc -c
>>> ./xzminidec: Unsupported check; not verifying file integrity
>>> <......working for some time.......>
>>> 201330688
>>>
>>> (xzminidec doesn't crash: exit code is zero).
>>>
>>> The peculiar thing is that 201330688 is exactly 0x0c001000.
>>
>> linux-3.8.tar.xz from kernel.org has three concatenated .xz streams.
>> You can see this with e.g. "xz -l" or "xz -lvv". At least pxz creates
>> such .xz files. Such files are valid and fine.
>>
>> xzminidec is a limited example program. It doesn't support concatenated
>> streams. This is mentioned in the comment in the beginning of
>> xzminidec.c. One may argue if it is a bug or a feature, but at least
>> the limitation has been documented.
>>
>> Busybox' xzcat lacks support for concatenated .xz streams. For
>> comparison, Busybox' zcat and bzcat do support concatenated streams.
>>
>>      $ echo foo | gzip>  test.gz
>>      $ echo bar | gzip>>  test.gz
>>      $ busybox zcat test.gz
>>      foo
>>      bar
>>
>>      $ echo foo | xz>  test.xz
>>      $ echo bar | xz>>  test.xz
>>      $ busybox xzcat test.xz
>>      foo
>>
>> liblzma in XZ Utils has a flag to decode concatenated streams to make
>> it a bit easier to handle such files. I would prefer to not include
>> such a flag in XZ Embedded, since I think in most embedded situations
>> (boot loaders, kernels etc.) such a flag is useless. Busybox is an
>> exception to this.
>>
>> Below is a patch to add support for concatenated .xz streams. It also
>> handles possible padding (sequence of zero-bytes) between the streams.
>> It probably has room for improvement, but it should be a useful starting
>> point.
>>
>> diff --git a/archival/libarchive/decompress_unxz.c b/archival/libarchive/decompress_unxz.c
>> index 79b48a1..5ebbd28 100644
>> --- a/archival/libarchive/decompress_unxz.c
>> +++ b/archival/libarchive/decompress_unxz.c
>> @@ -86,8 +86,40 @@ unpack_xz_stream(transformer_aux_data_t *aux, int src_fd, int dst_fd)
>>               IF_DESKTOP(total += iobuf.out_pos;)
>>               iobuf.out_pos = 0;
>>           }
>> -        if (r == XZ_STREAM_END) {
>> -            break;
>> +        while (r == XZ_STREAM_END) {
>> +            /* Handle concatenated .xz Streams including possible
>> +             * Stream Padding.
>> +             */
>> +            if (iobuf.in_pos == iobuf.in_size) {
>> +                int rd = safe_read(src_fd, membuf, BUFSIZ);
>> +                if (rd<  0) {
>> +                    bb_error_msg(bb_msg_read_error);
>> +                    total = -1;
>> +                    goto out;
>> +                }
>> +                if (rd == 0)
>> +                    goto out;
>> +
>> +                iobuf.in_size = rd;
>> +                iobuf.in_pos = 0;
>> +            }
>> +
>> +            /* Stream Padding must always be a multiple of four
>> +             * bytes to preserve four-byte alignment. To keep the
>> +             * code slightly smaller, we aren't as strict here as
>> +             * the .xz spec requires. We just skip all zero-bytes
>> +             * without checking the alignment and thus can accept
>> +             * files that aren't valid, e.g. the XZ Utils test
>> +             * files bad-0pad-empty.xz and bad-0catpad-empty.xz.
>> +             */
>> +            while (iobuf.in_pos<  iobuf.in_size) {
>> +                if (membuf[iobuf.in_pos] != 0) {
>> +                    xz_dec_reset(state);
>> +                    r = XZ_OK;
>> +                    break;
>> +                }
>> +                ++iobuf.in_pos;
>> +            }
>>           }
>>           if (r != XZ_OK&&  r != XZ_UNSUPPORTED_CHECK) {
>>               bb_error_msg("corrupted data");
>> @@ -95,6 +127,8 @@ unpack_xz_stream(transformer_aux_data_t *aux, int src_fd, int dst_fd)
>>               break;
>>           }
>>       }
>> +
>> +out:
>>       xz_dec_end(state);
>>       free(membuf);
>>
>>
>> By the way, since Busybox' copy of XZ Embedded hasn't been updated
>> since unxz was added, this bug fix is missing from Busybox:
>>
>>      http://git.tukaani.org/?p=xz-embedded.git;a=commitdiff;h=4cec51e1be4797a4bd8b266a1d34cabd7fdb79fd
>>
>> There is also the following bug fix but I think it doesn't affect
>> Busybox' unxz:
>>
>>      http://git.tukaani.org/?p=xz-embedded.git;a=commitdiff;h=9690fe69dc97eb2e7fe2804e4448a5278cde5411
>>
>> Another known issue with Busybox' unxz is that most .xz files use CRC64
>> for integrity checking, but XZ Embedded supports only CRC32. So with
>> most .xz files, Busybox cannot verify the integrity check. This should
>> be fixed at XZ Embedded side first, of course.
>>
> 
> thanks lasse, i integrated all 3 patches into my busybox setup and can confirm that xzcat/tar behave as expected now.
> 
> --JS
> _______________________________________________
> busybox mailing list
> busybox@busybox.net
> http://lists.busybox.net/mailman/listinfo/busybox

Message #185 received at 686502@bugs.debian.org (full text, mbox, reply):

From: John Spencer <maillist-busybox@barfooze.de>

To: Michael Tokarev <mjt@tls.msk.ru>

Cc: Lasse Collin <lasse.collin@tukaani.org>, busybox@busybox.net, 686502@bugs.debian.org

Subject: Re: XZ embedded bug unpacking linux-3.8.tar.xz

Date: Tue, 26 Feb 2013 12:23:26 +0100

On 02/26/2013 07:43 AM, Michael Tokarev wrote:
> 26.02.2013 03:21, John Spencer wrote:
>> [ quoting the full mail of lasse since it didnt make its way into the bb maillist yet ]
>
> Additionally there has been a discussion and attempts to cook up a
> patch in Debian, see http://bugs.debian.org/686502 , which I submitted
> as a bug to busybox bugzilla -- https://bugs.busybox.net/show_bug.cgi?id=5804 .
> Cc'ing the Debian bugreport.  I like the below patch better :)

the patches for busybox 1.20.2 are available in this commit
https://github.com/rofl0r/sabotage/commit/c03ddd39878473939bda6b574bc8854c533b4b00


(so that you dont have to backport them yourselves again)

i.e.
https://raw.github.com/rofl0r/sabotage/c03ddd39878473939bda6b574bc8854c533b4b00/KEEP/busybox-xz-bugfix1.patch
https://raw.github.com/rofl0r/sabotage/c03ddd39878473939bda6b574bc8854c533b4b00/KEEP/busybox-xz-bugfix2.patch
https://raw.github.com/rofl0r/sabotage/c03ddd39878473939bda6b574bc8854c533b4b00/KEEP/busybox-xz-bugfix3.patch

>
> /mjt
>

--JS

Message #194 received at 686502-close@bugs.debian.org (full text, mbox, reply):

From: Michael Tokarev <mjt@tls.msk.ru>

To: 686502-close@bugs.debian.org

Subject: Bug#686502: fixed in busybox 1:1.20.0-8

Date: Sat, 02 Mar 2013 13:02:37 +0000

Source: busybox
Source-Version: 1:1.20.0-8

We believe that the bug you reported is fixed in the latest version of
busybox, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 686502@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Michael Tokarev <mjt@tls.msk.ru> (supplier of updated busybox package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Sat, 02 Mar 2013 16:42:49 +0400
Source: busybox
Binary: busybox busybox-static busybox-udeb busybox-syslogd udhcpc udhcpd
Architecture: source all i386
Version: 1:1.20.0-8
Distribution: unstable
Urgency: low
Maintainer: Debian Install System Team <debian-boot@lists.debian.org>
Changed-By: Michael Tokarev <mjt@tls.msk.ru>
Description: 
 busybox    - Tiny utilities for small and embedded systems
 busybox-static - Standalone rescue shell with tons of builtin utilities
 busybox-syslogd - Provides syslogd and klogd using busybox
 busybox-udeb - Tiny utilities for the debian-installer (udeb)
 udhcpc     - Provides the busybox DHCP client implementation
 udhcpd     - Provides the busybox DHCP server implementation
Closes: 686502 695862 701959 701965 701968
Changes: 
 busybox (1:1.20.0-8) unstable; urgency=low
 .
   * grep-fix-grep--Fw-not-respecting-the--w-option.patch - implement
     fgrep -w correctly (Closes: #695862)
   * xz-support-concatenated-xz-streams.patch (Closes: #686502)
   * lineedit-initialize-delptr.patch - fix segfault in line editing
     facility (Closes: #701959)
   * mdev-fix-mode-of-dir1-in-=dir1-dir2-file-rule.patch - make intermediate
     dirs in /dev to be of mode 0755 not 0777 (Closes: #701965)
   * fix unaligned access macros (Closes: #701968)
     - fix-move_to_unaligned16.patch
     - xz-fix-put_unaligned_e32.patch
Checksums-Sha1: 
 0ce0fbc54964bf44985376b0be266876e09b86a9 1610 busybox_1.20.0-8.dsc
 9fbe30f7c6de03bdf5c9bcd16c8b297148d4d7bc 62951 busybox_1.20.0-8.debian.tar.gz
 564c8a5482eccb3b437b155327f59cd92368ac79 20872 busybox-syslogd_1.20.0-8_all.deb
 0607bf93b22f6db3d079feaa489d6041c4e8a485 881974 busybox-static_1.20.0-8_i386.deb
 2a332d254cab678211d5b5a50b25aca6a96a4f24 441558 busybox_1.20.0-8_i386.deb
 fb7aeb57240c702f8356e699333b1cf209d93c67 18332 udhcpc_1.20.0-8_i386.deb
 7404e547496a19b3d4d8e90585c2a4d9bee6494a 21776 udhcpd_1.20.0-8_i386.deb
 3557ff8ad2c958b144015fc3728c53b65678d610 178600 busybox-udeb_1.20.0-8_i386.udeb
Checksums-Sha256: 
 1ae77a1cda0552f9590929dde677de1e2ddf932c5edfbff3baff65750d9a832c 1610 busybox_1.20.0-8.dsc
 c22d58e9a877fbac974fb951ab8c1c752124df69bbb141c824e02d9aa47aa68c 62951 busybox_1.20.0-8.debian.tar.gz
 b19001f738d6c7ad2c4269e69bfdad65fc4c71decd1ce74675c05935193d9b20 20872 busybox-syslogd_1.20.0-8_all.deb
 15988d9f533aa6ecfbd06fcd049eecf7d174137b9dd4a802a545b505088a1343 881974 busybox-static_1.20.0-8_i386.deb
 02eb9ce87e2d5aaccfd628754fd2cb53abdf9354425892c05f3325d19ad689a9 441558 busybox_1.20.0-8_i386.deb
 7b3f0fa7d0d941acdb19f71303a2adfe50e003d41fbc2d3a62f859c1ac46d4bd 18332 udhcpc_1.20.0-8_i386.deb
 cadf87560047aad778842d008bff3f0d716827648830d7cb6cdf5e4779c0b161 21776 udhcpd_1.20.0-8_i386.deb
 9eb6820ce22f9350eac5551b74173dd03d23966f2bd8990148b3cc02a1b71fbf 178600 busybox-udeb_1.20.0-8_i386.udeb
Files: 
 b837aef93d3ff1736a5a8db10fa7dfeb 1610 utils optional busybox_1.20.0-8.dsc
 6d6b3587b8b3ef984ca16346391657b6 62951 utils optional busybox_1.20.0-8.debian.tar.gz
 b5aefa36f0e2eaba6bbc4d20a2514e6c 20872 utils optional busybox-syslogd_1.20.0-8_all.deb
 cdc2694ae30e7b76fc797558c6605889 881974 shells extra busybox-static_1.20.0-8_i386.deb
 a35b88c961b5348a0f4c8d42d9b5e5b8 441558 utils optional busybox_1.20.0-8_i386.deb
 0f544071f26ebee7fcc9012faadf1bda 18332 net optional udhcpc_1.20.0-8_i386.deb
 91fd467a8dee635ba7dc90d02169b92a 21776 net optional udhcpd_1.20.0-8_i386.deb
 a480fcc6d2b893556bbf494b7e358273 178600 debian-installer extra busybox-udeb_1.20.0-8_i386.udeb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iJwEAQECAAYFAlEx9fQACgkQUlPFrXTwyDg5SQQAsVhgh6fMcUHVUzAb4Czz6a0k
5WR/c0Y07DCNef0JIFvPPJTTv/WcSgDq0R6w0vmWCTmaW0qi3UGIQIwEziEK8hS4
gyaBm/PA3FZmJz63RlzRAY5Mvz+39ZERSedvevFdSjGukaTiNt/fHsfPYA8K+oMZ
BVoHIlIj4a+gYFpHnQ8=
=5YsY
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.