Debian Bug report logs - #705648
if-modfied-since undhandled case causes apt lists corruption

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Ludovico Cavedon <cavedon@debian.org>

To: submit@bugs.debian.org

Subject: if-modfied-since undhandled case causes apt lists corruption

Date: Wed, 17 Apr 2013 18:11:46 -0700

[Message part 1 (text/plain, inline)]

Package: apt-transport-https
Version: 0.8.16~exp12
Tags: patch

When using an https repository, apt will use libcurl to download
files, setting the If-Modified-Since herder in the http request.

If the server replies with a 200 OK but a Last-Modified header with a
date that is not newer that the requested If-Modified-Since, libcurl
drop the file, reporting an errorcode 200, but the actual downloaded
file will be empty.

Causing messages like
W: Size of file
/var/lib/apt/lists/partial/repo.server.com_dists_precise_main_binary-amd64_Packages
is not what the server reported 0 25118

See the attached patch that fixes the issue.

Apt needs to check the CURLINFO_CONDITION_UNMET to know if libcurl
discarded the payload.
Also TotalSize will get updated to the actual size of the file as
progress_callback will be called with the download size even in the
case libcurl decides to discard the payload. This will cause the check
for the size to fail. In my patch I just disable the check if the
actual size of the file is 0. Probably a better way would be adding a
new header is the message from the worker stating that the file was
discarded.

The cases why the http server would give this kind of response are, e.g.
1) if-modified-since used like a etag header
http://trac.nginx.org/nginx/ticket/93

(under this light, it might makes sense to avoid using the
CURLOPT_TIMECONDITION feature and handle If-Modified-SInce as an Etag)

2) a bug in apt-cacher-ng which sometimes returns 200 OK with a
Last-Modified equal to Last-Modified (about to be reported)

This bug is present at least as far back as version 0.8.16~exp12 and
still affects the latest in experimental (0.9.7.9~exp3). The attached
patch is for 0.9.7.9~exp3.

Cheers,
Ludovico

[check-time-condition.patch (application/octet-stream, attachment)]

Message #10 received at 705648@bugs.debian.org (full text, mbox, reply):

From: Ludovico Cavedon <cavedon@debian.org>

To: 705648@bugs.debian.org

Subject: Re: Bug#705648: if-modfied-since undhandled case causes apt lists corruption

Date: Fri, 19 Apr 2013 16:42:10 -0700

FYI,
my fix to this bug triggers a bug in libcurl:
http://bugs.debian.org/705783

Cheers,
Ludovico

Message #15 received at 705648@bugs.debian.org (full text, mbox, reply):

From: Ludovico Cavedon <cavedon@debian.org>

To: 705648@bugs.debian.org

Subject: Re: Bug#705648: if-modfied-since undhandled case causes apt lists corruption

Date: Mon, 22 Apr 2013 19:25:51 -0700

[Message part 1 (text/plain, inline)]

Attached is an improved version of the patch.

Cheers,
Ludovico

[check-time-condition-v02.patch (application/octet-stream, attachment)]

Message #20 received at 705648@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: Ludovico Cavedon <cavedon@debian.org>

Cc: 705648@bugs.debian.org

Subject: Re: Bug#705648: if-modfied-since undhandled case causes apt lists corruption

Date: Thu, 02 May 2013 13:58:59 +0800

[Message part 1 (text/plain, inline)]

We are experiencing this issue on our Ubuntu LTS (precise) systems at
work. I've tested the patch provided by Ludovico and it fixed the issue.
It would be great to get this fix included in the first wheezy point
release, because we are thinking about switching to Debian.

-- 
bye,
pabs

http://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]

Message #25 received at 705648-close@bugs.debian.org (full text, mbox, reply):

From: Michael Vogt <mvo@debian.org>

To: 705648-close@bugs.debian.org

Subject: Bug#705648: fixed in apt 0.9.8

Date: Wed, 08 May 2013 17:03:09 +0000

Source: apt
Source-Version: 0.9.8

We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 705648@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Michael Vogt <mvo@debian.org> (supplier of updated apt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Wed, 08 May 2013 18:43:28 +0200
Source: apt
Binary: apt libapt-pkg4.12 libapt-inst1.5 apt-doc libapt-pkg-dev libapt-pkg-doc apt-utils apt-transport-https
Architecture: source all amd64
Version: 0.9.8
Distribution: unstable
Urgency: low
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Michael Vogt <mvo@debian.org>
Description: 
 apt        - commandline package manager
 apt-doc    - documentation for APT
 apt-transport-https - https download transport for APT
 apt-utils  - package managment related utility programs
 libapt-inst1.5 - deb package format runtime library
 libapt-pkg-dev - development files for APT's libapt-pkg and libapt-inst
 libapt-pkg-doc - documentation for APT development
 libapt-pkg4.12 - package managment runtime library
Closes: 154868 322074 620344 651640 699759 704653 704723 705087 705648
Changes: 
 apt (0.9.8) unstable; urgency=low
 .
   [ Ludovico Cavedon ]
   * properly handle if-modfied-since with libcurl/https
     (closes: #705648)
 .
   [ Andreas Beckman ]
   * apt-pkg/algorithms.cc:
     - Do not propagate negative scores from rdepends. Propagating the absolute
       value of a negative score may boost obsolete packages and keep them
       installed instead of installing their successors.  (Closes: #699759)
 .
   [ Michael Vogt ]
   * apt-pkg/sourcelist.cc:
     - fix segfault when a hostname contains a [, thanks to
       Tzafrir Cohen (closes: #704653)
   * debian/control:
     - replace manpages-it (closes: #704723)
 .
   [ David Kalnischkies ]
   * various simple changes to fix cppcheck warnings
   * apt-pkg/pkgcachegen.cc:
     - do not store the MD5Sum for every description language variant as
       it will be the same for all so it can be shared to save cache space
     - handle language tags for descriptions are unique strings to be shared
     - factor version string creation out of NewDepends, so we can easily reuse
       version strings e.g. for implicit multi-arch dependencies
     - equal comparisions are used mostly in same-source relations,
       so use this to try to reuse some version strings
     - sort group and package names in the hashtable on insert
     - share version strings between same versions (of different architectures)
       to save some space and allow quick comparisions later on
   * apt-pkg/pkgcache.cc:
     - assume sorted hashtable entries for groups/packages
   * apt-pkg/cacheiterators.h:
     - provide DepIterator::IsSatisfied as a nicer shorthand for DepCheck
   * apt-pkg/deb/debversion.cc:
     - add a string-equal shortcut for equal version comparisions
 .
   [ Marc Deslauriers ]
   * make apt-ftparchive generate missing deb-src hashes (LP: #1078697)
 .
   [ Yaroslav Halchenko ]
   * Fix English spelling error in a message ('A error'). Unfuzzy
     translations. Closes: #705087
 .
   [ Programs translations ]
   * French translation completed (Christian Perrier)
 .
   [ Manpages translations ]
   * French translation completed (Christian Perrier)
 .
   [ Daniel Hartwig ]
   * apt-pkg/contrib/strutl.cc:
     - include port in shortened URIs (e.g. with apt-cache policy, progress
       display) thanks to James McCoy (Closes: #154868, #322074)
     - percent-encode username and password when writing URIs
   * methods/http.cc:
     - properly escape IP-literals (e.g. IPv6 address) when building
       Host headers and URIs (Closes: #620344)
   * methods/https.cc:
     - use https_proxy environment variable if present, falling back to
       http_proxy otherwise
     - use authentication credentials from proxy URI
       (Closes: #651640, LP: #1087512)
     - environment variables do not override an explicit no proxy
       directive ("DIRECT") in apt.conf
     - disregard all_proxy environment variable, like other methods
Checksums-Sha1: 
 116360c77f1f8c54891b327d39acf363a41261bc 1682 apt_0.9.8.dsc
 5b76a6358b11ab485dbef2bc16ba5328cdea7948 3500796 apt_0.9.8.tar.gz
 89461f552f12ee6f0274855736a8a588bc47948a 290444 apt-doc_0.9.8_all.deb
 3cb37e9e8382c3abb1525b9e3e891f15475b17c6 964930 libapt-pkg-doc_0.9.8_all.deb
 4883ad8769334cec21be7b811ee4825860cd7ed7 896220 libapt-pkg4.12_0.9.8_amd64.deb
 7f5f6ab632bb27d7ca22051c95588136c49ff9bd 168314 libapt-inst1.5_0.9.8_amd64.deb
 43bc95f409d0e949506b759b78c638b746ad7d5e 1313098 apt_0.9.8_amd64.deb
 0796669c1b3a13e0d78397e2351b35e2bcd6f146 189842 libapt-pkg-dev_0.9.8_amd64.deb
 9d7dabe9ea311a2b7ae69d0cb63c5860e33755e8 389146 apt-utils_0.9.8_amd64.deb
 f1a5ca9a3a3e69786789da741549a24408a885a1 110600 apt-transport-https_0.9.8_amd64.deb
Checksums-Sha256: 
 e27e8507f97cc1a3e8f2cfd9a3a488fec1af4559a87c19b27eb950275c4db475 1682 apt_0.9.8.dsc
 91937aff743892892949e54d0329496ddbfb6181d126406a05f6762cdbbab594 3500796 apt_0.9.8.tar.gz
 1b266da0a5ddcd9015e4475507399371fee4b4b282a18f2c6ba004d4a5ce0af3 290444 apt-doc_0.9.8_all.deb
 0864bba06d93ae00b65f469ed8a25b5d0562a50477515a1af0d640e7e562d521 964930 libapt-pkg-doc_0.9.8_all.deb
 adedd7b43c0eb542b7137071ecaf8a27c363618d483a6a9364ba73c5fa05baca 896220 libapt-pkg4.12_0.9.8_amd64.deb
 899892c2bf8daac6da30da3be265c15a5e3959d94ec3be3a8352f047238d8562 168314 libapt-inst1.5_0.9.8_amd64.deb
 dbccf49f58e0d9de654566dd913a25db2bc5989cb356ed98be6450f7c0b9fcec 1313098 apt_0.9.8_amd64.deb
 a8e67828dd4426ea11943970e63908e5c435a9aac126ef8f4c076364e9933b44 189842 libapt-pkg-dev_0.9.8_amd64.deb
 fe2c2b0eb57e8cf0721f5c1075e01a81833a668331206628787ca12439b334af 389146 apt-utils_0.9.8_amd64.deb
 9dc84186723b66b7b335b3e149fefe61b1c5b7c8e759633c09a262b784965fd3 110600 apt-transport-https_0.9.8_amd64.deb
Files: 
 ca35aee225910dec2e3cb19df376cd71 1682 admin important apt_0.9.8.dsc
 90aa29992d8b8a43885c6a28ccf507d4 3500796 admin important apt_0.9.8.tar.gz
 bc44fced1b97e5ef24d807fcf777646a 290444 doc optional apt-doc_0.9.8_all.deb
 71b09b35661f93f5c8e9f87e71fb10fd 964930 doc optional libapt-pkg-doc_0.9.8_all.deb
 6a813bfe8a601fa320b936b0ee716efc 896220 libs important libapt-pkg4.12_0.9.8_amd64.deb
 e79d8e6799a69d8a9939629780109c3c 168314 libs important libapt-inst1.5_0.9.8_amd64.deb
 b627de5b3fe06182475120d008ad3773 1313098 admin important apt_0.9.8_amd64.deb
 829606dd9ec7e0cdad7080232eabfc9d 189842 libdevel optional libapt-pkg-dev_0.9.8_amd64.deb
 623d0730b1d36443a3341e79371ab67b 389146 admin important apt-utils_0.9.8_amd64.deb
 09d97f139dba326bc8ba0412b3a0648b 110600 admin optional apt-transport-https_0.9.8_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlGKgy4ACgkQliSD4VZixzSsWwCeILYp85ky0KJWPqr+W67M7uXq
Hw0An2jzGjzbOQtOkiQA7+OGa5wQoIWC
=tem2
-----END PGP SIGNATURE-----

Message #30 received at 705648-close@bugs.debian.org (full text, mbox, reply):

From: Michael Vogt <mvo@debian.org>

To: 705648-close@bugs.debian.org

Subject: Bug#705648: fixed in apt 0.9.7.9

Date: Wed, 05 Jun 2013 23:17:05 +0000

Source: apt
Source-Version: 0.9.7.9

We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 705648@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Michael Vogt <mvo@debian.org> (supplier of updated apt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Tue, 04 Jun 2013 11:24:15 +0200
Source: apt
Binary: apt libapt-pkg4.12 libapt-inst1.5 apt-doc libapt-pkg-dev libapt-pkg-doc apt-utils apt-transport-https
Architecture: source all amd64
Version: 0.9.7.9
Distribution: stable
Urgency: low
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Michael Vogt <mvo@debian.org>
Description: 
 apt        - commandline package manager
 apt-doc    - documentation for APT
 apt-transport-https - https download transport for APT
 apt-utils  - package managment related utility programs
 libapt-inst1.5 - deb package format runtime library
 libapt-pkg-dev - development files for APT's libapt-pkg and libapt-inst
 libapt-pkg-doc - documentation for APT development
 libapt-pkg4.12 - package managment runtime library
Closes: 699759 705648
Changes: 
 apt (0.9.7.9) stable; urgency=low
 .
   [ Ludovico Cavedon ]
   * properly handle if-modfied-since with libcurl/https
     (closes: #705648)
 .
   [ Andreas Beckman ]
   * apt-pkg/algorithms.cc:
     - Do not propagate negative scores from rdepends. Propagating the absolute
       value of a negative score may boost obsolete packages and keep them
       installed instead of installing their successors.  (Closes: #699759)
Checksums-Sha1: 
 5ce1d659b835b45a0e1b6952124287a46420a8a2 1696 apt_0.9.7.9.dsc
 bcc38d7dd4e93f21f5e95bdc63b057f85b62cda1 3397270 apt_0.9.7.9.tar.gz
 f16195abfea61e088984af74a9216ff99825286c 262488 apt-doc_0.9.7.9_all.deb
 35230a59a197d2633d20ef42c09d99e102976608 960044 libapt-pkg-doc_0.9.7.9_all.deb
 6d9436914485986f7a8f21a66421ab07314fd91d 897002 libapt-pkg4.12_0.9.7.9_amd64.deb
 b8f44ae7c300d4d11895a0b0f14ff1c16c628965 166610 libapt-inst1.5_0.9.7.9_amd64.deb
 6a271487ceee6f6d7bc4c47a8a16f49c26e4ca04 1253524 apt_0.9.7.9_amd64.deb
 a25c18a083f23193ca069cb14974ed3b5b68fb09 186754 libapt-pkg-dev_0.9.7.9_amd64.deb
 57950f083a74849cb380bc6f1da4870a02702704 377312 apt-utils_0.9.7.9_amd64.deb
 b498a3e620f648b2d3b9901ee6a33565e2aeedb2 108550 apt-transport-https_0.9.7.9_amd64.deb
Checksums-Sha256: 
 f669d1e7a019446bda1aa370f667d0f23ada7c5f2b408d54d47d180426b235f1 1696 apt_0.9.7.9.dsc
 062084d8bde5e1ac7219e1de21342f1bc3c0a028e87ce0a4fa6efe845462e6b6 3397270 apt_0.9.7.9.tar.gz
 fffdf20273b98aa0c9f40bed8b1e560120d24be6289c4d36161b6f2800157b86 262488 apt-doc_0.9.7.9_all.deb
 36e8a1508cd74098cfeb7e41052bd1b3f46c7999ac965cec3d0893262941929a 960044 libapt-pkg-doc_0.9.7.9_all.deb
 32ac1e812247d716287b476513c475be7457d58ad85c77732c30ed001de48916 897002 libapt-pkg4.12_0.9.7.9_amd64.deb
 8f3bfa84d47fbb2983d5801b526102eb262742ca6a78c452c26e55025e605edc 166610 libapt-inst1.5_0.9.7.9_amd64.deb
 3bba3b15fb5ace96df052935d7069e0d21ff1f5b496510ec9d2dc939eefad104 1253524 apt_0.9.7.9_amd64.deb
 8eba032573f9a949e9e24715975af5839cadddcd11d872b07f4292114bf5e132 186754 libapt-pkg-dev_0.9.7.9_amd64.deb
 b7b12eb0e23c13da27a736718741d940d4e80c140e23c312e327d33c5561de5b 377312 apt-utils_0.9.7.9_amd64.deb
 b4de1480e324ce1dbbc695f8472882a1b4d9f86783ed3d619e6820e58a232fe6 108550 apt-transport-https_0.9.7.9_amd64.deb
Files: 
 ba017b6f709ef689efb8830ba8e3ae09 1696 admin important apt_0.9.7.9.dsc
 4b992b556ad179d51ab524befc09d182 3397270 admin important apt_0.9.7.9.tar.gz
 446954b76e7c45d9faddf9537a59accc 262488 doc optional apt-doc_0.9.7.9_all.deb
 c64cbd436eaf9194a68bc4621c9ceb6a 960044 doc optional libapt-pkg-doc_0.9.7.9_all.deb
 533c2811532dca6a643c061136096daf 897002 libs important libapt-pkg4.12_0.9.7.9_amd64.deb
 f33e9b2c342a376a973c763e8cd89075 166610 libs important libapt-inst1.5_0.9.7.9_amd64.deb
 00a128b2eb2b08f4ecee7fe0d7e3c1c4 1253524 admin important apt_0.9.7.9_amd64.deb
 e46ff3a3365ff751b7619046bf5718e9 186754 libdevel optional libapt-pkg-dev_0.9.7.9_amd64.deb
 dcdc35df8ee7b21564ca8bc30955cb01 377312 admin important apt-utils_0.9.7.9_amd64.deb
 d6fe788d79df8d147c61ef46db4c1ed9 108550 admin optional apt-transport-https_0.9.7.9_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEUEARECAAYFAlGtwOsACgkQliSD4VZixzTxWgCYsm4tGD/MWTJxRQ0CUJ5zBSqb
LQCeOeKz+05LKShfXJyk7gdpVUjpYBk=
=GTFG
-----END PGP SIGNATURE-----

Message #35 received at 705648@bugs.debian.org (full text, mbox, reply):

From: Manfred Stock <manfred.stock+debian@gmail.com>

To: 705648@bugs.debian.org

Subject: Re: Bug #705648: fixed in apt 0.9.7.9

Date: Mon, 17 Jun 2013 16:40:21 +0200

Hi,

after upgrading a system to the latest point release of Wheezy this
morning, I started to notice that 'apt-get upgrade' did not see new
packages that were added to an internal repository (that is accessed
via https), even after 'apt-get update'. After verifying that the
repository looked fine and checking the output of running 'apt-get
update' with '-oDebug::Acquire::https=1' (which looked fine,
If-Modified-Since and Last-Modified had the expected values), I
noticed this bug report that was fixed in apt in the latest point
release. I was able to temporarily fix my problem by removing the
lists in /var/lib/apt/lists, however, the next time I wanted to update
packages, the problem was there again. It also seemed to help when I
commented two of the three internal repositories in the sources list
(those which had no changes).

After looking at the patch and the bug report again, I noticed the
reference in [1] to a bug in curl [2] - unfortunately, however, it
seems this bug is not yet fixed for Wheezy, but the description
sounded just like the problem I had (we have 3 https-based
repositories on the same server, two of which had no changes for some
days, one of them had very recent changes). So I tried again with a
freshly patched and built curl 7.26.0-1+wheezy2 (+ the patch from
[2]), and it seemed to work again.

So the fix for #705648 together with the currently missing fix for
#705783 breaks apt for software updates and the installation of new
packages (that were not yet in the repo at all the last time the
sources list was correctly updated) in our setup.

Best regards
Manfred

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=705648#10
[2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=705783

Send a report that this bug log contains spam.