Debian Bug report logs - #413324
apt: http pipelining breaks with squid proxy

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Lucas Nussbaum <lucas@lucas-nussbaum.net>

To: submit@bugs.debian.org

Subject: apt: http pipelining breaks with squid proxy

Date: Sun, 4 Mar 2007 10:19:59 +0100

Package: apt
Version: 0.6.46.4-0.1
Severity: important

Hi,

I am using a local squid proxy to cache packages files (so I modified
squid's configuration to also cache large files).

When I download a large number of packages, which are cached by squid, I
run into errors:
Failed to fetch
http://idpot.grenoble.grid5000.fr/debian/pool/main/libx/libx11/libx11-data_1.0.3-5_all.deb
Error reading from server - read (104 Connection reset by peer)
Failed to fetch
http://idpot.grenoble.grid5000.fr/debian/pool/main/f/fontconfig/fontconfig-config_2.4.2-1_all.deb
Error reading from server - read (104 Connection reset by peer)
Failed to fetch
http://idpot.grenoble.grid5000.fr/debian/pool/main/libx/libxt/libxt6_1.0.2-2_amd64.deb
Error reading from server - read (104 Connection reset by peer)
Failed to fetch
http://idpot.grenoble.grid5000.fr/debian/pool/main/m/mesa/libgl1-mesa-glx_6.5.1-0.6_amd64.deb
Error reading from server - read (104 Connection reset by peer)
Failed to fetch
http://idpot.grenoble.grid5000.fr/debian/pool/main/libp/libpng/libpng12-0_1.2.15~beta5-1_amd64.deb
Error reading from server - read (104 Connection reset by peer)
Failed to fetch
http://idpot.grenoble.grid5000.fr/debian/pool/main/libx/libxaw/libxaw7_1.0.2-4_amd64.deb
Error reading from server - read (104 Connection reset by peer)
Failed to fetch
http://idpot.grenoble.grid5000.fr/debian/pool/main/b/bind9/libisc11_9.3.4-2_amd64.deb
Error reading from server - read (104 Connection reset by peer)
Failed to fetch
http://idpot.grenoble.grid5000.fr/debian/pool/main/b/bind9/dnsutils_9.3.4-2_amd64.deb
Error reading from server - read (104 Connection reset by peer)

Changing the value of Acquire::http::Pipeline-Depth fixes the problem:
with 10 (the default) or 9, it fails nearly during every time.
with 8, I could do 43 runs without any failure.

What I use to test:
nb=0
while apt-get clean ; apt-get -o Acquire::http::Pipeline-Depth=5 \
install -y -d gnome kde; do
	nb=$(expr $nb + 1)
	echo "Runs OK: $nb"
	sleep 2
done
echo "Runs OK: $nb"

I tried with both the "squid" package and the "squid3" package. Both
failed in the same way.

I'm not sure if it's a bug in squid or in apt. This code is quite old
(1998 it seems), but the default squid configuration doesn't allow to
cache big files, so it's really trivial to discover this.

On a side note, apt.conf(5) says that Acquire::http::Pipeline-Depth can
be a value from 0 to 5. The default value is 10 in the code.

Thank you,
-- 
| Lucas Nussbaum
| lucas@lucas-nussbaum.net   http://www.lucas-nussbaum.net/ |
| jabber: lucas@nussbaum.fr             GPG: 1024D/023B3F4F |

Message #10 received at 413324@bugs.debian.org (full text, mbox, reply):

From: Joey Hess <joeyh@debian.org>

To: 413324@bugs.debian.org

Subject: also seeing this

Date: Fri, 25 May 2007 19:25:00 -0400

[Message part 1 (text/plain, inline)]

I'm also seeing what I think is this bug, with a similar squid
configuration. Since it occurs when I'm automcatically testing full
debian installs with d-i, I can't really tweak the pipeline depth
parameter.

-- 
see shy jo

[signature.asc (application/pgp-signature, inline)]

Message #15 received at 413324@bugs.debian.org (full text, mbox, reply):

From: Otavio Salvador <otavio@debian.org>

To: 413324@bugs.debian.org

Subject: Fails with lighttpd too

Date: Sat, 02 Jun 2007 16:43:43 -0300

I'm also having this problem but when I use lighttpd as server of the
mirror.

Using lighttpd is very easy to reproduce the problem. The easier way
is change lighttpd to use a small pipe size and ask APT to download
packages.

More info at http://trac.lighttpd.net/trac/ticket/657

To reproduce set server.max-keep-alive-requests to a small number, as
5, for example.

-- 
        O T A V I O    S A L V A D O R
---------------------------------------------
 E-mail: otavio@debian.org      UIN: 5906116
 GNU/Linux User: 239058     GPG ID: 49A5F855
 Home Page: http://otavio.ossystems.com.br
---------------------------------------------
"Microsoft sells you Windows ... Linux gives
 you the whole house."

Message #20 received at 413324-close@bugs.debian.org (full text, mbox, reply):

From: Otavio Salvador <otavio@debian.org>

To: 413324-close@bugs.debian.org

Subject: Bug#413324: fixed in apt 0.7.4

Date: Tue, 24 Jul 2007 15:32:02 +0000

Source: apt
Source-Version: 0.7.4

We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive:

apt-doc_0.7.4_all.deb
  to pool/main/a/apt/apt-doc_0.7.4_all.deb
apt-transport-https_0.7.4_i386.deb
  to pool/main/a/apt/apt-transport-https_0.7.4_i386.deb
apt-utils_0.7.4_i386.deb
  to pool/main/a/apt/apt-utils_0.7.4_i386.deb
apt_0.7.4.dsc
  to pool/main/a/apt/apt_0.7.4.dsc
apt_0.7.4.tar.gz
  to pool/main/a/apt/apt_0.7.4.tar.gz
apt_0.7.4_i386.deb
  to pool/main/a/apt/apt_0.7.4_i386.deb
libapt-pkg-dev_0.7.4_i386.deb
  to pool/main/a/apt/libapt-pkg-dev_0.7.4_i386.deb
libapt-pkg-doc_0.7.4_all.deb
  to pool/main/a/apt/libapt-pkg-doc_0.7.4_all.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 413324@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Otavio Salvador <otavio@debian.org> (supplier of updated apt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.7
Date: Tue, 24 Jul 2007 09:55:50 -0300
Source: apt
Binary: apt-utils libapt-pkg-doc libapt-pkg-dev apt-transport-https apt-doc apt
Architecture: source all i386
Version: 0.7.4
Distribution: unstable
Urgency: low
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Otavio Salvador <otavio@debian.org>
Description: 
 apt        - Advanced front-end for dpkg
 apt-doc    - Documentation for APT
 apt-transport-https - APT https transport
 apt-utils  - APT utility programs
 libapt-pkg-dev - Development files for APT's libapt-pkg and libapt-inst
 libapt-pkg-doc - Documentation for APT development
Closes: 359634 413324 432223
Changes: 
 apt (0.7.4) unstable; urgency=low
 .
   [ Michael Vogt ]
   * cmdline/apt-get.cc:
     - fix in the task-install code regexp (thanks to Adam Conrad and
       Colin Watson)
     - support task removal too: apt-get remove taskname^
       (thanks to Matt Zimmerman reporting this problem)
 .
   [ Otavio Salvador ]
   * Fix a typo on 0.7.3 changelog entry about g++ (7.3 to 4.3)
   * Fix compilation warnings:
     - apt-pkg/contrib/configuration.cc: wrong argument type;
     - apt-pkg/deb/dpkgpm.cc: wrong signess;
     - apt-pkg-acquire-item.cc: wrong signess and orderned initializers;
     - methods/https.cc:
       - type conversion;
       - unused variable;
       - changed SetupProxy() method to void;
   * Simplified HttpMethod::Fetch on http.cc removing Tail variable;
   * Fix pipeline handling on http.cc (closes: #413324)
   * Fix building to properly support binNMUs. Thanks to Daniel Schepler
     <schepler@math.unipd.it> by the patch (closes: #359634)
   * Fix example for Install-{Recommends,Suggests} options on
     configure-index example file. Thanks to Peter Eisentraut
     <peter_e@gmx.net> by the patch (closes: #432223)
 .
   [ Christian Perrier ]
   * Basque translation update. Closes: ##423766
   * Unfuzzy formerly complete translations
   * French translation update
   * Re-generate PO(T) files
   * Spanish translation update
   * Swedish translation update
Files: 
 80ed70af8d90096b9b8cb814fb264c42 905 admin important apt_0.7.4.dsc
 5cfd7887c6f2d7052ef21ae2ac3b5a44 1822084 admin important apt_0.7.4.tar.gz
 55e2aea58c9ef583e644a3e30222036b 91758 doc optional apt-doc_0.7.4_all.deb
 27b32b190ef2e901edd7572dcbb481b3 115668 doc optional libapt-pkg-doc_0.7.4_all.deb
 c1198114d082870cffec1e9336a3d0a4 50588 admin optional apt-transport-https_0.7.4_i386.deb
 7daa35c0d788b226940ede819455e867 1456062 admin important apt_0.7.4_i386.deb
 3c5f2b9d182fde593d07b7dca9cfd0c4 99448 libdevel optional libapt-pkg-dev_0.7.4_i386.deb
 8be6fef32568264395fb57615e5598d1 190262 admin important apt-utils_0.7.4_i386.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFGphlwLqiZQEml+FURAi6sAJ47KnfT6I7zxEge0FegDZbsjVPBKwCcDBTD
ZKCOOjYB2vDTzci3gVal75s=
=q2Of
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.