Debian Bug report logs - #685192
apt: redirection handling changes in 0.9.4 may break aptitude

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Raphael Geissert <geissert@debian.org>

To: submit@bugs.debian.org

Subject: apt: redirection handling changes in 0.9.4 may break aptitude

Date: Fri, 17 Aug 2012 19:53:08 -0500

Package: apt
Version: 0.9.4
Severity: grave
Control: affects -1 aptitude
X-Debbugs-CC: mika@debian.org, debian-release@lists.debian.org

Hi,

Michael Prokop noticed that in some cases an aptitude update would fail with 
a "E: Method gave invalid 200 URI Start message" when using http.debian.net.

After a lot of confusion and attempts to debug the problem, in a chroot 
where the issue could be reproduced, I eventually noticed that the versions 
of apt and libapt-pkg4.12 didn't match:

apt: 0.8.15.9 (old version)
libapt-pkg4.12: 0.9.7.2 (wheezy)
aptitude: 0.6.8-1 (wheezy)

The problem:
When aptitude uses libapt-pkg to download a file, it starts an instance of 
the http method that doesn't include the changes made in #668111. This means 
that it sends the 103 redirection message as usual, but it handles it 
internally. Given that aptitude uses libapt-pkg4.12, it handles the 103 
message in the new fashion and starts a new http process to handle the 
redirection.
Since the first http process is handling the request internally, it 
eventually sends a 200 "URI Start" message with the original URI (the one of 
http.debian.net). By then, libapt-pkg4.12 has already marked such URI as 
done, removed it from the http:http.debian.net queue, and more importantly: 
changed the URI of the Itm (pkgAcquire::Queue::QItem). So, when it receives 
the 200 message it can't even match the URI of the message to a QItem, 
therefore aborting with the "E: Method gave invalid 200 URI Start message" 
error.

Why APT still works:
The old apt version works just fine because it uses libapt-pkg4.10, which 
means it handles the redirection internally. According to dpkg -S the 
libapt-pkg4.10 used by apt is provided by the apt package itself. I.e. it is 
not a separate package.

Note: if it is not obvious enough, this isn't restricted to http.debian.net. 
Any mirror that sends a redirection could trigger this bug.


Now, the easiest way to prevent this kind of conflict would be by adding a 
Depends: apt >= 0.9.4 to libapt-pkg4.12. Not sure how much trouble it would 
cause to a squeeze->wheezy upgrade, as it would force apt to also be 
upgraded when upgrading aptitude (upgrading apt already requires upgrading 
aptitude.) It also introduces a soft dependency loop, but it seems harmless.
The alternative way to express it would be by adding a Breaks: apt (<< 
0.9.4) to libapt-pkg4.12. I think this last form would cause more noise 
during the upgrade.

Introducing a new redirection code (104?) would probably cause more trouble 
at this point than handling the problem via the dependencies system.

Toughts?

Sorry for not noticing it before. Somehow I knew I should have bumped the 
redirection code :-/

Cheers,
-- 
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net

Message #12 received at 685192@bugs.debian.org (full text, mbox, reply):

From: David Kalnischkies <kalnischkies+debian@gmail.com>

To: Raphael Geissert <geissert@debian.org>, 685192@bugs.debian.org

Cc: mika@debian.org, debian-release@lists.debian.org

Subject: Re: Bug#685192: apt: redirection handling changes in 0.9.4 may break aptitude

Date: Tue, 21 Aug 2012 15:50:34 +0200

For clarity: This partial upgrade thing effects not only aptitude, but APT
itself and "just" by extension all front-ends even if the message just talks
about how aptitude is unable to handle the internal change in libapt and
how it talks to his own http-method shipped in 'apt'.

And I doubt that a bug containing the words "partial upgrade" and
"unofficial sources" (which http.debian.net still is, even as a well-recieved
"mirror" of official content) fits very well in the severity "grave" bucket,
but I let it slight for the moment.


On Sat, Aug 18, 2012 at 2:53 AM, Raphael Geissert <geissert@debian.org> wrote:
> Now, the easiest way to prevent this kind of conflict would be by adding a
> Depends: apt >= 0.9.4 to libapt-pkg4.12. Not sure how much trouble it would
> cause to a squeeze->wheezy upgrade, as it would force apt to also be
> upgraded when upgrading aptitude (upgrading apt already requires upgrading
> aptitude.) It also introduces a soft dependency loop, but it seems harmless.

I think Depends are a bit hard in that case. It's not only a loop, but
libapt-pkg can be used without the method-binaries in a lot of cases, so a
Recommends: apt (>= ${binary:Version})
feels more appropriated and should trigger an upgrade of 'apt' in this
partial upgrade situation as well (as long as the installation of Recommends
are not disabled) without negative consequences on the installation order.


The only thing not covered by this Recommends is that you can still remove
apt from your system and possibly break aptitude (and other packages using
the acquire-system from libapt) - for any libapt user this will be equal to
the removal of an essential package through, however the specific front-end
handles this (apt-get is e.g. very vocal about that).

The net-result would be that front-ends should depend on 'apt' if they use
the acquire system (some do, even if they don't, packagesearch for example
 seems to be such a candidate).

Yet, this might be wrong in the (less likely case) that a user uses only
debtorrent or https which is provided by other packages and therefore the
acquire system could be used without needing the "standard" methods in 'apt'.
So again, a Recommends would be more in order maybe.

On the other hand: A depends could be added automatically with our symbol
file if an acquire symbol is used, recommends can't be added in this way.
Maybe we should add such a feature to dpkg-dev as it could come in handy for
(big) libraries using other tools internally in certain paths.
Might be better than requiring the library user to declare such a relation.


In the end we are talking about an "priority: important" package, so a user
who wants to remove it should be able to handle the pain s/he has to suffer.
'apt' doesn't depend on a network-manager, even through it is likely that
you need some sort of network access to get packages from somewhere else…

Same case if s/he prefers to disable installation of recommends.
And with this back to the initial topic: Adding a recommends, okay?


Best regards

David Kalnischkies

Message #17 received at 685192@bugs.debian.org (full text, mbox, reply):

From: Raphael Geissert <geissert@debian.org>

To: 685192@bugs.debian.org

Cc: mika@debian.org, debian-release@lists.debian.org

Subject: Re: Bug#685192: apt: redirection handling changes in 0.9.4 may break aptitude

Date: Tue, 21 Aug 2012 10:56:06 -0500

H David,

On Tuesday 21 August 2012 08:50:34 David Kalnischkies wrote:
> For clarity: This partial upgrade thing effects not only aptitude, but
> APT itself and "just" by extension all front-ends even if the message
> just talks about how aptitude is unable to handle the internal change in
> libapt and how it talks to his own http-method shipped in 'apt'.

As far as I tested, it doesn't affect APT as long as it isn't a partial 
upgrade from the experimental version that had a separate libapt-pk4.10.
Upgrading apt will also pull in libapt-pkg4.12, and at the time the new 
packages are unpacked no new http method is started. The next call to APT 
would already use the new versions of apt and the http method.

Am I missing something?

> And I doubt that a bug containing the words "partial upgrade" and
> "unofficial sources" (which http.debian.net still is, even as a
> well-recieved "mirror" of official content) fits very well in the
> severity "grave" bucket, but I let it slight for the moment.

Just one fact:
I have seen more than one mirror, part of the Debian mirrors network, 
redirect from /debian/ to /pub/linux/debian/ and stuff like that.
At the moment there should be none of those in the mirrors list, but users 
who had picked one of those mirrors before the path was changed would be 
affected.

That said, if you disagree with the severity, feel free to change it.

Not sure how common Michael Prokop's scenario is with FAI. He was using a 
minimal debootstrapped chroot and then upgrading it.

> I think Depends are a bit hard in that case. It's not only a loop, but
> libapt-pkg can be used without the method-binaries in a lot of cases, so
> a Recommends: apt (>= ${binary:Version})
> feels more appropriated and should trigger an upgrade of 'apt' in this
> partial upgrade situation as well (as long as the installation of
> Recommends are not disabled) without negative consequences on the
> installation order.
> 
> 
> The only thing not covered by this Recommends is that you can still
> remove apt from your system and possibly break aptitude (and other
> packages using the acquire-system from libapt) - for any libapt user
> this will be equal to the removal of an essential package through,
> however the specific front-end handles this (apt-get is e.g. very vocal
> about that).

If you do consider those cases, then Breaks should probably be used instead.
Recommends is not enough even for the scenario where this bug was 
reproduced: grml - recommends are disabled by default.

I haven't tested a squeeze->wheezy upgrade with Breaks, though. Will try to 
get around it today so that I can report back...

> Same case if s/he prefers to disable installation of recommends.
> And with this back to the initial topic: Adding a recommends, okay?

... because I don't think Recommends is appropriate.

Cheers,
-- 
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net

Message #22 received at 685192@bugs.debian.org (full text, mbox, reply):

From: Raphael Geissert <geissert@debian.org>

To: 685192@bugs.debian.org

Cc: mika@debian.org, debian-release@lists.debian.org

Subject: Re: Bug#685192: apt: redirection handling changes in 0.9.4 may break aptitude

Date: Thu, 23 Aug 2012 11:47:43 -0500

One day later than expected...

On Tuesday 21 August 2012 10:56:06 Raphael Geissert wrote:
> If you do consider those cases, then Breaks should probably be used
> instead. Recommends is not enough even for the scenario where this bug
> was reproduced: grml - recommends are disabled by default.
> 
> I haven't tested a squeeze->wheezy upgrade with Breaks, though. Will try
> to get around it today so that I can report back...

It went fine. APT of course had to be deconfigured due to the Breaks, but it 
was handled just fine.

I used a Breaks: apt (<< 0.9.4~).

Cheers,
-- 
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net

Message #27 received at 685192@bugs.debian.org (full text, mbox, reply):

From: David Kalnischkies <kalnischkies+debian@gmail.com>

To: Raphael Geissert <geissert@debian.org>, 685192@bugs.debian.org

Cc: mika@debian.org, debian-release@lists.debian.org

Subject: Re: Bug#685192: apt: redirection handling changes in 0.9.4 may break aptitude

Date: Tue, 28 Aug 2012 09:39:10 +0200

On Thu, Aug 23, 2012 at 6:47 PM, Raphael Geissert <geissert@debian.org> wrote:
> One day later than expected...

(Several days later than expected …)

> On Tuesday 21 August 2012 10:56:06 Raphael Geissert wrote:
>> If you do consider those cases, then Breaks should probably be used
>> instead. Recommends is not enough even for the scenario where this bug
>> was reproduced: grml - recommends are disabled by default.
>>
>> I haven't tested a squeeze->wheezy upgrade with Breaks, though. Will try
>> to get around it today so that I can report back...
>
> It went fine. APT of course had to be deconfigured due to the Breaks, but it
> was handled just fine.
>
> I used a Breaks: apt (<< 0.9.4~).

Which is after a bit of thinking not that surprising:
libapt-pkg is already unpacked and configured before apt is unpacked anyway
(as APT handles itself as essential), so the solution we arrive at is more or
less the same - good to know that at least sometimes theory isn't disproved by
the implementation. :)

Scheduled for 0.9.7.5
ETA: After we know what will happen with 0.9.7.4 (#685155)


Best regards

David Kalnischkies

Message #32 received at 685192-close@bugs.debian.org (full text, mbox, reply):

From: Michael Vogt <mvo@debian.org>

To: 685192-close@bugs.debian.org

Subject: Bug#685192: fixed in apt 0.9.7.5

Date: Tue, 11 Sep 2012 15:32:49 +0000

Source: apt
Source-Version: 0.9.7.5

We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 685192@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Michael Vogt <mvo@debian.org> (supplier of updated apt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Tue, 11 Sep 2012 15:56:44 +0200
Source: apt
Binary: apt libapt-pkg4.12 libapt-inst1.5 apt-doc libapt-pkg-dev libapt-pkg-doc apt-utils apt-transport-https
Architecture: source all amd64
Version: 0.9.7.5
Distribution: unstable
Urgency: low
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Michael Vogt <mvo@debian.org>
Description: 
 apt        - commandline package manager
 apt-doc    - documentation for APT
 apt-transport-https - https download transport for APT
 apt-utils  - package managment related utility programs
 libapt-inst1.5 - deb package format runtime library
 libapt-pkg-dev - development files for APT's libapt-pkg and libapt-inst
 libapt-pkg-doc - documentation for APT development
 libapt-pkg4.12 - package managment runtime library
Closes: 670900 678227 684435 685192 685989 686346 686975
Changes: 
 apt (0.9.7.5) unstable; urgency=low
 .
   [ Manpages translation updates ]
   * Japanese (KURASAWA Nozomu) (Closes: #684435)
   * Portuguese (Américo Monteiro) (Closes: #686975)
 .
   [ David Kalnischkies ]
   * handle packages without a mandatory architecture (debian-policy §5.3)
     by introducing a pseudo-architecture 'none' so that the small group of
     users with these packages can get right of them without introducing too
     much hassle for other users (Closes: #686346)
   * apt-pkg/cdrom.cc:
     - copy only configured translation files from a CD-ROM and not all
       available translation files preventing new installs with d-i from
       being initialized with all translations (Closes: #678227)
     - handle Components in the reduction for the source.list as multi-arch CDs
       otherwise create duplicated source entries (e.g. "wheezy main main")
   * apt-pkg/packagemanager.cc:
     - unpack versions only in case a different version from the package
       is currently in unpack state to recover from broken system states
       (like different file in M-A:same package and other dpkg errors)
       and avoid re-unpack otherwise (Closes: #670900)
   * debian/control:
     - let libapt-pkg break apt < 0.9.4 to ensure that the installed http-
       method supports the new redirection-style, thanks to Raphael Geissert
       for reporting & testing (Closes: #685192)
   * doc/apt_preferences.5.xml:
     - use the correct interval (x <= P < y) for pin value documentation as
       these are the intervals used by the code (Closes: #685989)
   * apt-pkg/indexcopy.cc:
     - do not create duplicated flat-archive CD-ROM sources for foreign
       architectures on multi-arch CD-ROMs
     - do not warn about files which have a record in the Release file, but
       are not present on the CD to mirror the behavior of the other methods
       and to allow uncompressed indexes to be dropped without scaring users
   * apt-pkg/pkgcachegen.cc:
     - do not create 'native' (or now 'none') package structures as a side
       effect of description translation parsing as it pollutes the cache
Checksums-Sha1: 
 7e666c8c7e7c7ab27c858d54a19485232229ac98 1689 apt_0.9.7.5.dsc
 f87aebbc6b9c413821bc5f19c91c87224f016357 3387864 apt_0.9.7.5.tar.gz
 6e54a50f3f23ca461e2525df41a856f98beab286 260874 apt-doc_0.9.7.5_all.deb
 f13303f91b27168ae497e1de8ec2af407a104a43 960408 libapt-pkg-doc_0.9.7.5_all.deb
 d74697ed798b7e089bf40a3e05fd80532590ae33 882860 libapt-pkg4.12_0.9.7.5_amd64.deb
 1afa928bbd988ce3a910bc6ecd93c27fb673d84d 164498 libapt-inst1.5_0.9.7.5_amd64.deb
 73f443563ef51198d12678c0611d9af678a1a8f7 1234400 apt_0.9.7.5_amd64.deb
 f35d3c5999a8fbe336b98ff7bb439a8f45b54c51 185406 libapt-pkg-dev_0.9.7.5_amd64.deb
 195ed4819d3fddf1f3cf7b9770d47ab4e8c4b592 373924 apt-utils_0.9.7.5_amd64.deb
 e4fbf830b93d5cd83b1202005b544de2b3741c08 107078 apt-transport-https_0.9.7.5_amd64.deb
Checksums-Sha256: 
 0c04c31d986810d2b52de501a45fad989b0ec55d8a96f73544d89126c58de45e 1689 apt_0.9.7.5.dsc
 82ff902cc33dab89e875c5827f625a64b4b872c34f1a16a8f04e07a7a1c30ced 3387864 apt_0.9.7.5.tar.gz
 ca64252135bae07fa54a534488e8a234f282113b1353ddfc0e39072834ee7c5a 260874 apt-doc_0.9.7.5_all.deb
 6dbe1b97ad10fe96a90f58d7a378346545ca58f6c2629331c802aae53ba490b6 960408 libapt-pkg-doc_0.9.7.5_all.deb
 913b5a7f26ed9de17f4e4257189f2136f2d00af783da80654a1b7f57e514beb6 882860 libapt-pkg4.12_0.9.7.5_amd64.deb
 cb40f8f8788e71b034de387c78c61e0be9b2b09cacfe8c7b995394f4b2c5cf06 164498 libapt-inst1.5_0.9.7.5_amd64.deb
 3c0d3d4b72d5de132560a26e3c86b9d8d759354e6a349b46b1b203b18b0c5563 1234400 apt_0.9.7.5_amd64.deb
 11ca12b5f43266108df9845bf5040f71d0ddddccd9760dad7a1e4efb8e4f92da 185406 libapt-pkg-dev_0.9.7.5_amd64.deb
 78e2f8b98489dde3d297ca855c3da72e6b583b97352993de7e3c697190fa0590 373924 apt-utils_0.9.7.5_amd64.deb
 442c083c20401c30dbbfcd9f71af85451bd81dec4dc847885ef222dede4cbf54 107078 apt-transport-https_0.9.7.5_amd64.deb
Files: 
 83737a282f81be3d24be43a60ef5aa11 1689 admin important apt_0.9.7.5.dsc
 51bed58178d73c21e240cb4e137afa93 3387864 admin important apt_0.9.7.5.tar.gz
 2ef4da2a1cece4afc1688c5d3b9e3aeb 260874 doc optional apt-doc_0.9.7.5_all.deb
 a983ad6026b1fd187988a21220298740 960408 doc optional libapt-pkg-doc_0.9.7.5_all.deb
 aa21106040013490cfe64f26020e0340 882860 libs important libapt-pkg4.12_0.9.7.5_amd64.deb
 6851100d5db7997c2f4e58b179908a15 164498 libs important libapt-inst1.5_0.9.7.5_amd64.deb
 073cfd3f4781fe1978ca00ea6853e78f 1234400 admin important apt_0.9.7.5_amd64.deb
 acaaa33ff73372b684ffa75e652c1cec 185406 libdevel optional libapt-pkg-dev_0.9.7.5_amd64.deb
 7da608a838d4a58ce26c62498c2d70c2 373924 admin important apt-utils_0.9.7.5_amd64.deb
 06997202bccda2f2cb72fb162b17b03a 107078 admin optional apt-transport-https_0.9.7.5_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlBPV4AACgkQliSD4VZixzTcvwCePyBv2X3kGh0Ba/ATKszd2zxr
/H4AoIK/zrO+im4bX7XaRgX/KqvAMmh4
=Oqfs
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.