Debian Bug report logs - #810796
HTTP pipelining is broken and causes download failures

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Robie Basak <robie.basak@ubuntu.com>

To: submit@bugs.debian.org

Subject: HTTP pipelining is broken and causes download failures

Date: Tue, 12 Jan 2016 11:57:49 +0000

[Message part 1 (text/plain, inline)]

Package: apt
Version: 1.10
User: ubuntu-devel@lists.ubuntu.com
Usertags: origin-ubuntu xenial

apt enables pipelining by default. But it does not check to see if the
server supports HTTP/1.1, so where the server doesn't apt is breaking
the HTTP protocol by attempting to use pipelining. This causes download
failures.

Workaround: "Acquire::http::Pipeline-Depth 0;" makes it work. It might
be an idea to drop the pipelining default until this is fixed.

In my case, I have http://ftp.debian.org/debian in my sources.list, but
go through a squid 2.7.STABLE7 proxy as the machine running apt doesn't
have direct access to the Internet. This is done explicitly using a
Acquire::HTTP::Proxy setting (so not using transparent interception).

I'm getting "Hash Sum mismatch" errors on some downloaded debs. Right
now I get this failure if I have a sid chroot created by "mk-sbuild"
(from ubuntu-dev-tools) and attempt to "apt-get build-dep mysql-5.6"
through my proxy. I'm not sure how reproducible this might be for
others, but I can reproduce it reliably right now. Unfortunately
behaviour might also change as sid changes.

I captured apt's traffic to my proxy. and in the failure case I see it
requesting pipelined files immediately, which I think is wrong of apt,
followed by HTTP/1.0 "Connection: close" responses from the proxy (which
is why I think it is wrong of apt, since it should only request
pipelining with HTTP/1.1 servers).

In most cases the server returns the first file only and then apt
(perhaps in response to the Connection: close?) closes the connection
gracefully. Then apt makes other pipelined requests for the files it did
not successfully retrieve because they were not the first in its
invalid pipelined requests.

In some cases I see the server hasn't finished sending the first file
before I see an RST. In this case apt doesn't get the whole file,
doesn't retry it and this leads to a Hash Sum mismatch.

Speculation: the server always tries to send an RST after processing the
first request if it sees further incoming data at this time, since that
is in violation of HTTP/1.0. apt closes the connection gracefully after
it sees either an HTTP/1.0 response or a "Connection: close" (not sure
which, or maybe both). So this is a race. If the RST arrives and is
processed first, then apt gets partial data and fails. But the race only
exists because apt invalidly attempts to pipeline when it shouldn't.

Thanks,

Robie

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 810796@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <jak@debian.org>

To: Robie Basak <robie.basak@ubuntu.com>, 810796@bugs.debian.org

Subject: Re: Bug#810796: HTTP pipelining is broken and causes download failures

Date: Tue, 12 Jan 2016 14:36:28 +0100

On 12 January 2016 at 12:57, Robie Basak <robie.basak@ubuntu.com> wrote:
> Package: apt
> Version: 1.10
> User: ubuntu-devel@lists.ubuntu.com
> Usertags: origin-ubuntu xenial
>
> apt enables pipelining by default. But it does not check to see if the
> server supports HTTP/1.1, so where the server doesn't apt is breaking
> the HTTP protocol by attempting to use pipelining. This causes download
> failures.
>
> Workaround: "Acquire::http::Pipeline-Depth 0;" makes it work. It might
> be an idea to drop the pipelining default until this is fixed.
>
> In my case, I have http://ftp.debian.org/debian in my sources.list, but
> go through a squid 2.7.STABLE7 proxy as the machine running apt doesn't
> have direct access to the Internet. This is done explicitly using a
> Acquire::HTTP::Proxy setting (so not using transparent interception).
>
> I'm getting "Hash Sum mismatch" errors on some downloaded debs. Right
> now I get this failure if I have a sid chroot created by "mk-sbuild"
> (from ubuntu-dev-tools) and attempt to "apt-get build-dep mysql-5.6"
> through my proxy. I'm not sure how reproducible this might be for
> others, but I can reproduce it reliably right now. Unfortunately
> behaviour might also change as sid changes.
>
> I captured apt's traffic to my proxy. and in the failure case I see it
> requesting pipelined files immediately, which I think is wrong of apt,
> followed by HTTP/1.0 "Connection: close" responses from the proxy (which
> is why I think it is wrong of apt, since it should only request
> pipelining with HTTP/1.1 servers).

Well, yes, nobody really uses HTTP/1.0 servers, so it's not really tested much.

If we have hashes, we will try to do pipelining and then fall back if
the server messes up the response.

Maybe it helps to also disable pipelining if the server responds with
HTTP/1.0, like this:

diff --git a/methods/server.cc b/methods/server.cc
index d5188d4..a64bf0f 100644
--- a/methods/server.cc
+++ b/methods/server.cc
@@ -150,9 +150,14 @@ bool ServerState::HeaderLine(string Line)
       else
       {
         if (Major == 1 && Minor == 0)
+        {
            Persistent = false;
+           Pipeline = false;
+        }
         else
+        {
            Persistent = true;
+        }
       }

       return true;


Not sure though

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.

Message #15 received at 810796@bugs.debian.org (full text, mbox, reply):

From: Robie Basak <robie.basak@ubuntu.com>

To: Julian Andres Klode <jak@debian.org>

Subject: Re: Bug#810796: HTTP pipelining is broken and causes download failures

Date: Tue, 12 Jan 2016 13:56:10 +0000

[Message part 1 (text/plain, inline)]

On Tue, Jan 12, 2016 at 02:36:28PM +0100, Julian Andres Klode wrote:
> Well, yes, nobody really uses HTTP/1.0 servers, so it's not really tested much.

I just checked and a newer squid (3.1.19) that I have handy also
responds with HTTP/1.0. I don't have a squid 3.4 deployment to check,
but 3.1 shipped in wheezy (Debian LTS project EOL is 2018) and in Ubuntu
12.04, which is not EOL until 2019. I think apt needs to be able to work
by default through squid proxies that are still commonly deployed.

> If we have hashes, we will try to do pipelining and then fall back if
> the server messes up the response.
> 
> Maybe it helps to also disable pipelining if the server responds with
> HTTP/1.0, like this:

I think this would help. You will reduce the number of pipelined
requests, and thus the number of times the race is lost. But you won't
eliminate the race completely. Really you need to only enable pipelining
after you see an HTTP/1.1 or higher response, rather than the other way
round. That is trickier to code correctly but is required by the
standards, AIUI.

How about disabling pipelining by default if Acquire::http:Proxy is
defined? This still wouldn't be technically correct but might be a good
interim workaround to avoid users having to research the workaround
themselves.

And in many cases a locally defined proxy will also mean that the proxy
is local, so you have less to lose by not having pipelining in those
cases anyway, even if the proxy does support it.

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 810796@bugs.debian.org (full text, mbox, reply):

From: Robie Basak <robie.basak@ubuntu.com>

To: Julian Andres Klode <jak@debian.org>, 810796@bugs.debian.org

Subject: Re: Bug#810796: HTTP pipelining is broken and causes download failures

Date: Tue, 12 Jan 2016 13:57:20 +0000

[Message part 1 (text/plain, inline)]

Sorry, I failed to Cc: the bug. Done now, and bounced my original
message to it also for the archive.

[signature.asc (application/pgp-signature, inline)]

Message #25 received at 810796@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <juliank@ubuntu.com>

To: Robie Basak <robie.basak@ubuntu.com>

Cc: 810796@bugs.debian.org

Subject: Re: Bug#810796: HTTP pipelining is broken and causes download failures

Date: Tue, 12 Jan 2016 16:31:04 +0100

On Tue, Jan 12, 2016 at 01:56:10PM +0000, Robie Basak wrote:
> On Tue, Jan 12, 2016 at 02:36:28PM +0100, Julian Andres Klode wrote:
> > Well, yes, nobody really uses HTTP/1.0 servers, so it's not really tested much.
> 
> I just checked and a newer squid (3.1.19) that I have handy also
> responds with HTTP/1.0. I don't have a squid 3.4 deployment to check,
> but 3.1 shipped in wheezy (Debian LTS project EOL is 2018) and in Ubuntu
> 12.04, which is not EOL until 2019. I think apt needs to be able to work
> by default through squid proxies that are still commonly deployed.
> 
> > If we have hashes, we will try to do pipelining and then fall back if
> > the server messes up the response.
> > 
> > Maybe it helps to also disable pipelining if the server responds with
> > HTTP/1.0, like this:
> 
> I think this would help. You will reduce the number of pipelined
> requests, and thus the number of times the race is lost. But you won't
> eliminate the race completely. Really you need to only enable pipelining
> after you see an HTTP/1.1 or higher response, rather than the other way
> round. That is trickier to code correctly but is required by the
> standards, AIUI.

Yes. We are taking a look at this.

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.

When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to (`inline'). Thank you.

Message #30 received at 810796@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <jak@debian.org>

To: Robie Basak <robie.basak@ubuntu.com>, 810796@bugs.debian.org

Subject: Re: Bug#810796: HTTP pipelining is broken and causes download failures

Date: Tue, 12 Jan 2016 23:27:12 +0100

On Tue, Jan 12, 2016 at 01:56:10PM +0000, Robie Basak wrote:
> On Tue, Jan 12, 2016 at 02:36:28PM +0100, Julian Andres Klode wrote:
> > Well, yes, nobody really uses HTTP/1.0 servers, so it's not really tested much.
> 
> I just checked and a newer squid (3.1.19) that I have handy also
> responds with HTTP/1.0. I don't have a squid 3.4 deployment to check,
> but 3.1 shipped in wheezy (Debian LTS project EOL is 2018) and in Ubuntu
> 12.04, which is not EOL until 2019. I think apt needs to be able to work
> by default through squid proxies that are still commonly deployed.
> 
> > If we have hashes, we will try to do pipelining and then fall back if
> > the server messes up the response.
> > 
> > Maybe it helps to also disable pipelining if the server responds with
> > HTTP/1.0, like this:
> 
> I think this would help. You will reduce the number of pipelined
> requests, and thus the number of times the race is lost. But you won't
> eliminate the race completely. Really you need to only enable pipelining
> after you see an HTTP/1.1 or higher response, rather than the other way
> round. That is trickier to code correctly but is required by the
> standards, AIUI.
> 

Fixed in my branch in this commit:
  https://github.com/Debian/apt/commit/2caf777bf7badbc75e6503c4dcee71e51b0a43d8
just need to fix a failing test case that checks if pipelining is messed up,
as that now does no pipelining for the first package anymore.

And change the variable name to something nicer.

diff --git a/methods/server.cc b/methods/server.cc
index d5188d4..9f21eff 100644
--- a/methods/server.cc
+++ b/methods/server.cc
@@ -150,9 +150,15 @@ bool ServerState::HeaderLine(string Line)
       else
       {
 	 if (Major == 1 && Minor == 0)
+	 {
 	    Persistent = false;
+	 }
 	 else
+	 {
 	    Persistent = true;
+	    if (ServerIsDoingPipeliningWrong == false)
+	       Pipeline = true;
+	 }
       }
 
       return true;
@@ -240,7 +246,7 @@ bool ServerState::HeaderLine(string Line)
 }
 									/*}}}*/
 // ServerState::ServerState - Constructor				/*{{{*/
-ServerState::ServerState(URI Srv, ServerMethod *Owner) : ServerName(Srv), TimeOut(120), Owner(Owner)
+ServerState::ServerState(URI Srv, ServerMethod *Owner) : ServerName(Srv), TimeOut(120), Owner(Owner), ServerIsDoingPipeliningWrong(false)
 {
    Reset();
 }
@@ -532,6 +538,7 @@ int ServerMethod::Loop()
 	    _error->Discard();
 	    Server->Close();
 	    Server->Pipeline = false;
+	    Server->ServerIsDoingPipeliningWrong = true;
 	    
 	    if (FailCounter >= 2)
 	    {
@@ -604,6 +611,7 @@ int ServerMethod::Loop()
 			   strprintf(out, _("Automatically disabled %s due to incorrect response from server/proxy. (man 5 apt.conf)"), "Acquire::http::PipelineDepth");
 			   std::cerr << "W: " << out << std::endl;
 			   Server->Pipeline = false;
+			   Server->ServerIsDoingPipeliningWrong = true;
 			   // we keep the PipelineDepth value so that the rest of the queue can be fixed up as well
 			}
 			Rename(Res.Filename, I->DestFile);
diff --git a/methods/server.h b/methods/server.h
index 365bc08..8f81090 100644
--- a/methods/server.h
+++ b/methods/server.h
@@ -51,6 +51,7 @@ struct ServerState
    enum {Chunked,Stream,Closes} Encoding;
    enum {Header, Data} State;
    bool Persistent;
+   bool ServerIsDoingPipeliningWrong;
    std::string Location;
 
    // This is a Persistent attribute of the server itself.
@@ -86,7 +87,7 @@ struct ServerState
    bool Comp(URI Other) const {return Other.Host == ServerName.Host && Other.Port == ServerName.Port;};
    virtual void Reset() {Major = 0; Minor = 0; Result = 0; Code[0] = '\0'; TotalFileSize = 0; JunkSize = 0;
 		 StartPos = 0; Encoding = Closes; time(&Date); HaveContent = false;
-		 State = Header; Persistent = false; Pipeline = true; MaximumSize = 0;};
+		 State = Header; Persistent = false; Pipeline = false; MaximumSize = 0;};
    virtual bool WriteResponse(std::string const &Data) = 0;
 
    /** \brief Transfer the data from the socket */


-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.

When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to (`inline'). Thank you.

Message #33 received at 810796-submitter@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <jak@debian.org>

To: 810796-submitter@bugs.debian.org

Subject: Bug#810796 in apt marked as pending

Date: Tue, 12 Jan 2016 22:41:21 +0000

Control: tag 810796 pending

Hello,

Bug #810796 in apt reported by you has been fixed in the Git repository. You can
see the commit message below, and you can check the diff of the fix at:

    https://anonscm.debian.org/cgit/apt/apt.git/diff/?id=b6d88f3

(this message was generated automatically based on the git commit message)
---
commit b6d88f39aceda2e093e1bf8751f07236b7d9e483
Author: Julian Andres Klode <jak@debian.org>
Date:   Tue Jan 12 15:18:12 2016 +0100

    Only enable pipelining if server is HTTP/1.1
    
    Just enabling it for anyone breaks with HTTP/1.0 servers and
    proxies sometimes.
    
    Closes: #810796

Message #38 received at 810796-submitter@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <jak@debian.org>

To: 810796-submitter@bugs.debian.org

Subject: Re: Bug#810796: in apt marked as pending

Date: Wed, 13 Jan 2016 00:21:46 +0100

On 12 January 2016 at 23:41, Julian Andres Klode <jak@debian.org> wrote:
> Control: tag 810796 pending
>
> Hello,
>
> Bug #810796 in apt reported by you has been fixed in the Git repository. You can
> see the commit message below, and you can check the diff of the fix at:
>
>     https://anonscm.debian.org/cgit/apt/apt.git/diff/?id=b6d88f3
>
> (this message was generated automatically based on the git commit message)
> ---
> commit b6d88f39aceda2e093e1bf8751f07236b7d9e483
> Author: Julian Andres Klode <jak@debian.org>
> Date:   Tue Jan 12 15:18:12 2016 +0100
>
>     Only enable pipelining if server is HTTP/1.1
>
>     Just enabling it for anyone breaks with HTTP/1.0 servers and
>     proxies sometimes.
>
>     Closes: #810796
>

JFYI: I expect this to be released this week in 1.2 - probably Friday,
meaning it would be synced xenial on Saturday.


-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.

Message #43 received at 810796-quiet@bugs.debian.org (full text, mbox, reply):

From: Robie Basak <robie.basak@ubuntu.com>

To: Julian Andres Klode <jak@debian.org>, 810796-quiet@bugs.debian.org

Subject: Re: Bug#810796: in apt marked as pending

Date: Wed, 13 Jan 2016 08:59:44 +0000

[Message part 1 (text/plain, inline)]

On Wed, Jan 13, 2016 at 12:21:46AM +0100, Julian Andres Klode wrote:
> On 12 January 2016 at 23:41, Julian Andres Klode <jak@debian.org> wrote:
> JFYI: I expect this to be released this week in 1.2 - probably Friday,
> meaning it would be synced xenial on Saturday.

Thanks!

[signature.asc (application/pgp-signature, inline)]

Message #48 received at 810796-close@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <jak@debian.org>

To: 810796-close@bugs.debian.org

Subject: Bug#810796: fixed in apt 1.2

Date: Fri, 15 Jan 2016 19:18:47 +0000

Source: apt
Source-Version: 1.2

We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 810796@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Julian Andres Klode <jak@debian.org> (supplier of updated apt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Fri, 15 Jan 2016 19:52:19 +0100
Source: apt
Binary: apt libapt-pkg5.0 libapt-inst2.0 apt-doc libapt-pkg-dev libapt-pkg-doc apt-utils apt-transport-https
Architecture: source
Version: 1.2
Distribution: unstable
Urgency: medium
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Julian Andres Klode <jak@debian.org>
Description:
 apt        - commandline package manager
 apt-doc    - documentation for APT
 apt-transport-https - https download transport for APT
 apt-utils  - package management related utility programs
 libapt-inst2.0 - deb package format runtime library
 libapt-pkg-dev - development files for APT's libapt-pkg and libapt-inst
 libapt-pkg-doc - documentation for APT development
 libapt-pkg5.0 - package management runtime library
Closes: 734922 810046 810509 810622 810796
Changes:
 apt (1.2) unstable; urgency=medium
 .
   [ Julian Andres Klode ]
   * Do not break apt-file (<< 3.0~exp1~), 2.X works fine
   * Version the build-depends on liblz4-dev to >= 0.0~r126
   * search: Handle packages without description (Closes: #810622)
   * Only enable pipelining if server is HTTP/1.1 (Closes: #810796)
 .
   [ Mattia Rizzolo ]
   * Sort the list of sources to be built and linked (Closes: #810509)
 .
   [ Michael Vogt ]
   * Do not show multiple identical apt-cache showsrc entries (Closes: #734922)
 .
   [ David Kalnischkies ]
   * evaluate sourceslist-list-format entity in vendors sources.list.
     Thanks to Niels Thykier for reporting on IRC
   * update symbols file
   * fix M-A:foreign provides creation for unknown archs
   * revert file-hash based action-merging in acquire (Closes: 810046)
   * return correct position in APT::StringView::(r)find
 .
   [ Zhou Mo ]
   * po: update simplified chinese programs translation
Checksums-Sha1:
 66b20a6b69fa612a4e8cdd33168cf7009d6f480f 2331 apt_1.2.dsc
 491c976997a267bb94178ffba50c6063054941ab 2014968 apt_1.2.tar.xz
Checksums-Sha256:
 cb9c95772ae1d0e62a0eeea427c49dc3826b443bf5e52ee789befb2a570dd37a 2331 apt_1.2.dsc
 8e330c9ceee328075d527108997fc0395dc10603d30eedc0d322783058674ed0 2014968 apt_1.2.tar.xz
Files:
 4b617d2e1cb2eb0d666cbfa11879ac96 2331 admin important apt_1.2.dsc
 9104ce061470a43addf30ab976181adc 2014968 admin important apt_1.2.tar.xz

-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJWmUBpAAoJENc8OeVlgLOGxjsQAJCePOHpUKcH94yxoKfHKeLt
dQ3Jn4FrSjNPquW3Mu+O5eWM6eVFxCA8eY+4ZxsNMcupxBlvjDg+86gMBgqlOMFd
EyqTw4XndzfC4gV6QBpBqxdW0kecSi3G2Xc64vjtmMYY+OmWgyxP7wblMcSxDHRS
U+vgxfFHG6padgLr2nj3XpG+2HDqULxFa9C4rC8oXx1bmBmByfKWv4ztF5uymjAe
JT99kbkXLqkOEkvOsi3/+nA7uwCmUJjtiEKus97/t4scvz3/SvhiKf7DLDGMoKqE
dru5F8SQDFqOxNZ8Kv9DaWhqlGWq4O6IVTa2ip9TqKCSzmjKeRRrIEjJw+z9Y3w5
laENnuRXOLvi0mtWAnVxTHdwhGVP37fsYGhBdCKzq53XfrWkWsI5FYhbkvlKxBxI
kmSqM+pFEJas5LSJZ3aFeQJJub4A3T6tBJTAbklp+ZEAI5uCr2HjWcEheZMa9Jzq
IgsQuZ1czj/Ki6jsS2TJqn7U/xl+/sGPgy2LsEeU20gUasnq5nxL7tyln9ptYDqB
UmSJXXxcPOPIZWEl+hbuO7TF1LueMUSw//Z0Pg6BzoqeJnOAcr/PV+y2ZCe/DGji
5fY5mmMV8g78YTCDYPwgf8HTJpNYb6BiJwLvUaHmEt1Up+q8bQWH7eRRljTyf65M
i9Fr4dBEygFLFW/BLx2D
=xrCH
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.