Debian Bug report logs - #669061
apt: http method dies on sparc

version graph

Package: apt; Maintainer for apt is APT Development Team <deity@lists.debian.org>; Source for apt is src:apt.

Reported by: Kurt Roeckx <kurt@roeckx.be>

Date: Mon, 16 Apr 2012 20:45:02 UTC

Severity: serious

Found in version apt/0.9.0

Fixed in version apt/0.9.2

Done: Michael Vogt <mvo@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#669061; Package apt. (Mon, 16 Apr 2012 20:45:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Kurt Roeckx <kurt@roeckx.be>:
New Bug report received and forwarded. Copy sent to APT Development Team <deity@lists.debian.org>. (Mon, 16 Apr 2012 20:45:05 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Kurt Roeckx <kurt@roeckx.be>
To: submit@bugs.debian.org
Subject: apt: http method dies on sparc
Date: Mon, 16 Apr 2012 22:41:17 +0200
Package: apt
Version: 0.9.0
Severity: serious

Hi,

apt-get update now results in this on sparc:
Get:1 http://debian.carnet.hr unstable InRelease [208 kB]
E: Method http has died unexpectedly!
E: Sub-process http received signal 10.



Kurt





Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#669061; Package apt. (Mon, 16 Apr 2012 22:15:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to David Kalnischkies <kalnischkies+debian@gmail.com>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Mon, 16 Apr 2012 22:15:03 GMT) Full text and rfc822 format available.

Message #10 received at 669061@bugs.debian.org (full text, mbox):

From: David Kalnischkies <kalnischkies+debian@gmail.com>
To: Kurt Roeckx <kurt@roeckx.be>, 669061@bugs.debian.org
Subject: Re: Bug#669061: apt: http method dies on sparc
Date: Tue, 17 Apr 2012 00:12:22 +0200
On Mon, Apr 16, 2012 at 22:41, Kurt Roeckx <kurt@roeckx.be> wrote:
> apt-get update now results in this on sparc:
> Get:1 http://debian.carnet.hr unstable InRelease [208 kB]
> E: Method http has died unexpectedly!
> E: Sub-process http received signal 10.

Any trace?
Which version was it which worked last?
Could you maybe try intermediate versions (poor-mans bisect)?

I have no access to sparc - just having i386, amd64 and armel here on
which it seems to work so any (correctly aligned) pointers are welcome.


Best regards

David Kalnischkies




Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#669061; Package apt. (Mon, 16 Apr 2012 22:27:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Kurt Roeckx <kurt@roeckx.be>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Mon, 16 Apr 2012 22:27:03 GMT) Full text and rfc822 format available.

Message #15 received at 669061@bugs.debian.org (full text, mbox):

From: Kurt Roeckx <kurt@roeckx.be>
To: David Kalnischkies <kalnischkies+debian@gmail.com>
Cc: 669061@bugs.debian.org
Subject: Re: Bug#669061: apt: http method dies on sparc
Date: Tue, 17 Apr 2012 00:22:25 +0200
On Tue, Apr 17, 2012 at 12:12:22AM +0200, David Kalnischkies wrote:
> On Mon, Apr 16, 2012 at 22:41, Kurt Roeckx <kurt@roeckx.be> wrote:
> > apt-get update now results in this on sparc:
> > Get:1 http://debian.carnet.hr unstable InRelease [208 kB]
> > E: Method http has died unexpectedly!
> > E: Sub-process http received signal 10.
> 
> Any trace?
> Which version was it which worked last?
> Could you maybe try intermediate versions (poor-mans bisect)?
> 
> I have no access to sparc - just having i386, amd64 and armel here on
> which it seems to work so any (correctly aligned) pointers are welcome.

I guess you're not a DD, and so don't have access to the porter
machines?

Anyway, the backtrace looks like:
Reading symbols from /usr/lib/apt/methods/http...(no debugging symbols found)...done.
[New LWP 9241]
Core was generated by `/usr/lib/apt/methods/http'.
Program terminated with signal 10, Bus error.
#0  0xf7e4e24c in ?? () from /usr/lib/sparc-linux-gnu/libapt-pkg.so.4.12
(gdb) bt
#0  0xf7e4e24c in ?? () from /usr/lib/sparc-linux-gnu/libapt-pkg.so.4.12
#1  0x5f1d36f9 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Which doesn't look very helpful.


Kurt





Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#669061; Package apt. (Tue, 17 Apr 2012 20:36:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Kurt Roeckx <kurt@roeckx.be>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Tue, 17 Apr 2012 20:36:04 GMT) Full text and rfc822 format available.

Message #20 received at 669061@bugs.debian.org (full text, mbox):

From: Kurt Roeckx <kurt@roeckx.be>
To: Michael Vogt <mvo@debian.org>, 665969@bugs.debian.org
Cc: "Adam D. Barratt" <adam@adam-barratt.org.uk>, 669061@bugs.debian.org
Subject: Re: Bug#665969: nmu: apt_0.9.0
Date: Tue, 17 Apr 2012 22:33:50 +0200
On Tue, Apr 17, 2012 at 09:59:09AM +0200, Michael Vogt wrote:
> > > Problems reported so far include:
> > > 
> > > E: Method http has died unexpectedly!
> > > E: Sub-process http received signal 10.
> > 
> > This is #669061
> 
> This appears to be Sparc only, right? We do had trouble with
> alignments in the hashsum calculations in the past, maybe that is
> re-occuring with the addition of sha512. 

I've only seen it on sparc, but I didn't try this on many arches.

I've been trying to reproduce this, and it seems to happen after
the first write() call that writes the file in
/var/cache/apt/archives/.



Kurt





Added indication that bug 669061 blocks 665969 Request was from Adam D. Barratt <adam@adam-barratt.org.uk> to control@bugs.debian.org. (Tue, 17 Apr 2012 20:51:12 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#669061; Package apt. (Wed, 18 Apr 2012 16:09:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to David Kalnischkies <kalnischkies+debian@gmail.com>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Wed, 18 Apr 2012 16:09:04 GMT) Full text and rfc822 format available.

Message #27 received at 669061@bugs.debian.org (full text, mbox):

From: David Kalnischkies <kalnischkies+debian@gmail.com>
To: 669061@bugs.debian.org
Cc: Kurt Roeckx <kurt@roeckx.be>, debian-sparc@lists.debian.org
Subject: Re: Bug#669061: apt: http method dies on sparc
Date: Wed, 18 Apr 2012 18:07:15 +0200
On Tue, Apr 17, 2012 at 00:22, Kurt Roeckx <kurt@roeckx.be> wrote:
> I guess you're not a DD, and so don't have access to the porter
> machines?

Correct guess, but I am adventures enough to work around it:
I had a lot of fun today trying to compile apt=0.9.1 in a qemu-sparc
debian etch image from aurel32 [0]. Took ages and (surprisingly) few
changes to compile with g++-4.1 and the environment in general. :)

It seems to be indeed the way the sha256 and sha512 checksums are
calculated. To make it a bit obscure: we have a testcase checking them
and they run successfully producing the correct output.
(for reference: test/libapt/hashsums_test.cc)

The bug seems only to be triggered if a checksum is "slowly" build
as it is the case in the http transport while downloading.

To reproduce you can use:
>>>>>
$ cat request
601 Configuration
Config-Item: Acquire::http::DependOnSTDIN=0

600 Acquire URI
URI: http://www.debian.org/index.html
Filename: debian.index.html

$ cat request | /usr/lib/apt/methods/http
<<<<<
(the empty line at the end of 'request' is important!)

What you should see is a '201 URI Done' message at the
end which includes a few checksums. Instead sparc instantly
crashes after starting the download ('200 URI Start') with a
bus error.

etch's gdb is crashing on me with internal errors[2] while trying to
debug this and warns me, that it might be completely wrong about
the following, but it seems to be the case that the very first call
to SHA256_Transform in apt-pkg/contrib/sha2_internal.cc [1] fails
on line 467 (which is intended to be big endian code)
called from SHA256_Final on line 595 (=first occurrence).
[apt uses the not unrolled transform method, but the unrolled seems
to have a similar issue as it crashes, too]

SHA512 is very similar, so i presume it has the very same problem.


As i run out of time for today i am leaving these dangling pointers as
they are for now and hope someone can connect the dots. CC'ing sparc
mailinglist as they have properly a good chance to find the problem
before i manage to warp my head around it…


Best regards

David Kalnischkies

[0] http://people.debian.org/~aurel32/qemu/sparc/
[1] http://anonscm.debian.org/loggerhead/apt/debian-sid/annotate/head:/apt-pkg/contrib/sha2_internal.cc#L441
[2] internal-error: register_size: Asseration 'regnum >= 0 && regnum <
(NUM_REGS + NUM_PSEUDO_REGS)' failed.




Information forwarded to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>:
Bug#669061; Package apt. (Fri, 20 Apr 2012 11:28:53 GMT) Full text and rfc822 format available.

Acknowledgement sent to David Kalnischkies <kalnischkies+debian@gmail.com>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>. (Fri, 20 Apr 2012 11:28:55 GMT) Full text and rfc822 format available.

Message #32 received at 669061@bugs.debian.org (full text, mbox):

From: David Kalnischkies <kalnischkies+debian@gmail.com>
To: 669061@bugs.debian.org
Cc: debian-sparc@lists.debian.org
Subject: Re: Bug#669061: apt: http method dies on sparc
Date: Fri, 20 Apr 2012 12:52:32 +0200
[Message part 1 (text/plain, inline)]
On Wed, Apr 18, 2012 at 18:07, David Kalnischkies
<kalnischkies+debian@gmail.com> wrote:
> It seems to be indeed the way the sha256 and sha512 checksums are
> calculated. To make it a bit obscure: we have a testcase checking them
> and they run successfully producing the correct output.
> (for reference: test/libapt/hashsums_test.cc)
>
> The bug seems only to be triggered if a checksum is "slowly" build
> as it is the case in the http transport while downloading.

/me can't see the forest for the trees…

While our testcase as well as other places in the library use well-aligned
buffers which can be casted from byte to word (or double-word) the
http transport is with its circle buffer implementation an exception to
this as it usually reads full words, but writes them to "random" places
which do not care for word boundaries…

More as a workaround than a completely proper fix i changed the only
place in which the input buffer is directly passed into Transform
(which was somehow not in the path gdb claimed it is used)
to memcpy it instead into a local buffer to ensure that the cast later
will work. The alternative would have been to change the http method,
but given the high debugging cost in case someone uses these
methods on unaligned buffers and that the calculations aren't that
time-critical we can life with that for now.

A more interesting fix would it be to make the Transform
methods work with bytes instead of words, but i guess
we can better think about that without blocking a transition…

(Michael will upload that soon together with a few more fixes.
 As of writing this should make apt new-serious-bug-free… yeah!
 We will see how long this will last… :) )

Best regards

David Kalnischkies
[apt-669061-unaligned-http-buffer-sigbus-on-sparc.diff (application/octet-stream, attachment)]

Reply sent to Michael Vogt <mvo@debian.org>:
You have taken responsibility. (Fri, 20 Apr 2012 12:51:16 GMT) Full text and rfc822 format available.

Notification sent to Kurt Roeckx <kurt@roeckx.be>:
Bug acknowledged by developer. (Fri, 20 Apr 2012 12:51:16 GMT) Full text and rfc822 format available.

Message #37 received at 669061-close@bugs.debian.org (full text, mbox):

From: Michael Vogt <mvo@debian.org>
To: 669061-close@bugs.debian.org
Subject: Bug#669061: fixed in apt 0.9.2
Date: Fri, 20 Apr 2012 12:47:21 +0000
Source: apt
Source-Version: 0.9.2

We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive:

apt-doc_0.9.2_all.deb
  to main/a/apt/apt-doc_0.9.2_all.deb
apt-transport-https_0.9.2_amd64.deb
  to main/a/apt/apt-transport-https_0.9.2_amd64.deb
apt-utils_0.9.2_amd64.deb
  to main/a/apt/apt-utils_0.9.2_amd64.deb
apt_0.9.2.dsc
  to main/a/apt/apt_0.9.2.dsc
apt_0.9.2.tar.gz
  to main/a/apt/apt_0.9.2.tar.gz
apt_0.9.2_amd64.deb
  to main/a/apt/apt_0.9.2_amd64.deb
libapt-inst1.5_0.9.2_amd64.deb
  to main/a/apt/libapt-inst1.5_0.9.2_amd64.deb
libapt-pkg-dev_0.9.2_amd64.deb
  to main/a/apt/libapt-pkg-dev_0.9.2_amd64.deb
libapt-pkg-doc_0.9.2_all.deb
  to main/a/apt/libapt-pkg-doc_0.9.2_all.deb
libapt-pkg4.12_0.9.2_amd64.deb
  to main/a/apt/libapt-pkg4.12_0.9.2_amd64.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 669061@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Michael Vogt <mvo@debian.org> (supplier of updated apt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Fri, 20 Apr 2012 11:26:16 +0200
Source: apt
Binary: apt libapt-pkg4.12 libapt-inst1.5 apt-doc libapt-pkg-dev libapt-pkg-doc apt-utils apt-transport-https
Architecture: source all amd64
Version: 0.9.2
Distribution: unstable
Urgency: low
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Michael Vogt <mvo@debian.org>
Description: 
 apt        - commandline package manager
 apt-doc    - documentation for APT
 apt-transport-https - https download transport for APT
 apt-utils  - package managment related utility programs
 libapt-inst1.5 - deb package format runtime library
 libapt-pkg-dev - development files for APT's libapt-pkg and libapt-inst
 libapt-pkg-doc - documentation for APT development
 libapt-pkg4.12 - package managment runtime library
Closes: 610069 669061 669328 669377 669591
Changes: 
 apt (0.9.2) unstable; urgency=low
 .
   [ Michael Vogt ]
   * apt-inst/contrib/extracttar.cc:
     - ensure that in StartGzip the InFd is set to "AutoClose" to ensure
       that the pipe is closed when InFd is closed. This fixes a Fd leak
       (LP: #985452)
 .
   [ David Kalnischkies ]
    * apt-pkg/deb/deblistparser.cc:
     - only treat the native apt as essential by default (Closes: #669377)
   * apt-pkg/contrib/fileutl.cc:
     - redirect stderr from compressors to /dev/null
   * apt-pkg/aptconfiguration.cc:
     - if the compressor is not installed, but we link against it's
       library accept it as a CompressionType (Closes: #669328)
   * apt-pkg/contrib/sha2_internal.cc:
     - do not use the input data directly but memcpy it instead as
       it could be unaligned as in the http-transport which causes
       a sigbus error on sparc (Closes: #669061)
   * apt-pkg/cacheset.cc:
     - actually return to the fallback modifier if we have detected we
       should for packagenames which look like modifiers (Closes: #669591)
 .
   [ Adam Conrad ]
   * Set FD_CLOEXEC on history.log's FD (Closes: #610069, LP: #636010)
 .
   [ Thorsten Spindler ]
   * apt-pkg/deb/dpkgpm.cc:
     - do not crash if (*I).Pkg is NULL (LP: #939867)
 .
   [ Malcolm Scott ]
   * apt-pkg/packagemanager.cc:
     - iterate over all pre-depends or-group member instead of looping
       endlessly over the first member in SmartUnpack (LP: #985852)
Checksums-Sha1: 
 843f6bc61f431b9d396da6e69ea3789267a35b3f 1663 apt_0.9.2.dsc
 a864bfeffc26236d611a7042feee3336612f6e93 3413679 apt_0.9.2.tar.gz
 4d1bd752672a57a9c3290baec7c0ad0f01a50a27 254554 apt-doc_0.9.2_all.deb
 c29f0a377be318d87201d705a2366466d6d8ab53 831988 libapt-pkg-doc_0.9.2_all.deb
 adc7a3b63388b848b6e2fe47c9ee69bb785dafca 874564 libapt-pkg4.12_0.9.2_amd64.deb
 6a9050da437dc6395f4e0f306f50e49b47f02955 157436 libapt-inst1.5_0.9.2_amd64.deb
 ae5487ed3f8f7de5b38af5a24549e2eac47dc893 1157930 apt_0.9.2_amd64.deb
 943ec8b1f77d589f1089335f8b0877d63ea6011f 177874 libapt-pkg-dev_0.9.2_amd64.deb
 06bfd0e82b8ef411d512e879fcecad0ff69249d3 369892 apt-utils_0.9.2_amd64.deb
 0aed3f425a18631ca2a19eedaf4439e871cbeeef 100822 apt-transport-https_0.9.2_amd64.deb
Checksums-Sha256: 
 44ae26fe1116b707688d09f4b0d51c8681100460bbe4e26e827a533872136883 1663 apt_0.9.2.dsc
 8ef1e68165dc7c670838e84d4d230aaf5d93b42eeb1865114e7ef7fb703dfb6c 3413679 apt_0.9.2.tar.gz
 6b0ba668da2ee38b6e9dfbda7c3edfccc6d85fb24e71d4527e0abe39ec55a7ea 254554 apt-doc_0.9.2_all.deb
 ad7f63a98fb7ac5ace3483662ef275743e456894d0ed8686ff8a2bdfe94cf5df 831988 libapt-pkg-doc_0.9.2_all.deb
 d19e5f700c8fd01a7fb0b88cb31e1682bedbf7547c7d065688b474542b8e337f 874564 libapt-pkg4.12_0.9.2_amd64.deb
 8dd74ded4e667d859cc153a253eb9efcaa675ce0a54bb88b672a703289def11f 157436 libapt-inst1.5_0.9.2_amd64.deb
 52104750dc71c45037e32d0d2001aff1155b21e402983171cc5f96e988651aa4 1157930 apt_0.9.2_amd64.deb
 74fa7a03ef05d9d0607ca75997b8fe295221e532202503bda6905e77b53cd7a1 177874 libapt-pkg-dev_0.9.2_amd64.deb
 b719cc5c706490e6f486a967369bd70d56444007442ce785a5820479b27abbce 369892 apt-utils_0.9.2_amd64.deb
 8fea8183f3276d55f77f62220d711990c996a8d8eddc3c8e5d50ea09716c85e4 100822 apt-transport-https_0.9.2_amd64.deb
Files: 
 4e52387c12326c1973a6095a81ac7758 1663 admin important apt_0.9.2.dsc
 8a7bbab661fdde926d96a3833c1c6399 3413679 admin important apt_0.9.2.tar.gz
 bd6738e083aa25db295704690de06c20 254554 doc optional apt-doc_0.9.2_all.deb
 40f2fcf6ceda83eaf76fc99f3442358d 831988 doc optional libapt-pkg-doc_0.9.2_all.deb
 70152a67b5156a0769c1569ad98fcb65 874564 admin important libapt-pkg4.12_0.9.2_amd64.deb
 00b7f707960e4ce76f110db6803509f8 157436 admin important libapt-inst1.5_0.9.2_amd64.deb
 17972a0c09a7d93f42a52e908ccf7596 1157930 admin important apt_0.9.2_amd64.deb
 412381acac4a6242218584bdc2f61ba8 177874 libdevel optional libapt-pkg-dev_0.9.2_amd64.deb
 61d7459135c9c103f300ca0b106cf91e 369892 admin important apt-utils_0.9.2_amd64.deb
 b995e03ee89e77ef1c2188dad2b26a66 100822 admin optional apt-transport-https_0.9.2_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAk+RVaEACgkQliSD4VZixzQecQCffknjnGymH7Gc+i1z6y03Wix6
pyUAoI2eGPMnjrtSlbGrniaTHP5oCw8u
=ROXM
-----END PGP SIGNATURE-----





Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Tue, 22 May 2012 07:38:06 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Wed Apr 16 11:12:08 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.