Debian Bug report logs - #572276
curl fails if Content-Length is present and Transfer-Encoding is chunked

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Akos PASZTORY <akos.pasztory@gmail.com>

To: submit@bugs.debian.org

Subject: curl fails if Content-Length is present and Transfer-Encoding is chunked

Date: Tue, 2 Mar 2010 23:07:12 +0200

[Message part 1 (text/plain, inline)]

Package: curl
Version: 7.19.7-1
Tags: patch


If a server returns a Content-Length header and then it uses chunked transfer,
curl often fails with with the following error:

  curl: (18) transfer closed with outstanding read data remaining

I think the cause is that when it finds a Content-Length header while parsing
the response, it makes k->maxdownload = contentlength; (transfer.c:1216).
Although it sets k->size = -1; (transfer.c:1014), k->maxdownload is still the
contentlength:

          /* According to RFC2616 section 4.4, we MUST ignore
             Content-Length: headers if we are now receiving data
             using chunked Transfer-Encoding.
          */
          if(k->chunk)
            k->size=-1;

Later, when the reading from the server (readwrite_data), the following can
happen: if the server wants to send the final "0\r\n\r\n" chunk in a separate
packet, libcurl will hit the condition in transfer.c:635:

    if((-1 != k->maxdownload) &&
       (k->bytecount + nread >= k->maxdownload)) { ... }

where it sets

    k->keepon &= ~KEEP_RECV; /* we're done reading */

and considers reading finished.  So it will never read the final chunk, hence
the error.

On the other hand, if the server sends the final chunk in a previous packet
with other data, Curl_httpchunk_read() will happily loop through all of it and
correctly register the final state (CHUNK_STOP).

Setting k->maxdownload to -1 when the transfer encoding is chunked seems to
solve the problem (for me at least):

--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -1010,8 +1010,10 @@ static CURLcode readwrite_http_headers(struct
SessionHandle *data,
              Content-Length: headers if we are now receiving data
              using chunked Transfer-Encoding.
           */
-          if(k->chunk)
+          if(k->chunk) {
             k->size=-1;
+           k->maxdownload=-1;
+         }

         }
         if(-1 != k->size) {


The curl upstream diverged somewhat already from the version used in Debian,
but the problem persists there also (I checked today's cvs snapshot).

[reset-maxdownload-if-chunked.patch (text/x-diff, attachment)]

Message #10 received at 572276@bugs.debian.org (full text, mbox, reply):

From: Daniel Stenberg <daniel@haxx.se>

To: 572276@bugs.debian.org

Subject: Can't repeat

Date: Thu, 4 Mar 2010 22:27:03 +0100 (CET)

Hello!

Thanks for the report, but somehow I fail to repeat this issue. Can you edit 
test 34 accordingly so that it fails the way you describe and then attach that 
new test file?

I've tried to insert "Content-Length: 12" both before and after the 
"Transfer-Encoding: chunked" header but they worked fine still...?

-- 

 / daniel.haxx.se

Message #15 received at 572276@bugs.debian.org (full text, mbox, reply):

From: Akos PASZTORY <akos.pasztory@gmail.com>

To: 572276@bugs.debian.org

Subject: instead of a test...

Date: Mon, 08 Mar 2010 23:52:31 +0200

[Message part 1 (text/plain, inline)]

I'm not sure it's possible to modify that test case to reproduce it.
The point is that the behavior depends on whether the server sends the
final chunk in the same TCP packet as the real payload, or it wants to
send it in a separate one.  IOW whether the read() syscall returns the
final chunk together with the real data or not.

I'm attaching a small proxy in perl that I was trying to use when the
problem occurred.  Run the following command a few times, and I guess
you will get some "curl: (18) transfer closed with outstanding read data
remaining" errors:

curl http://127.0.0.1:1080/debian/dists/README

The commented push_filter is a workaround to disable sending the
content-length header in the proxy, which seems to fix the problem for
me.

Hope it helps.

[t.pl (text/x-perl, inline)]

#!/usr/bin/perl -w

use HTTP::Proxy;
use HTTP::Proxy::HeaderFilter::simple;
use strict;

my $port = 1080;
my $destination = 'ftp.fi.debian.org';

{
	package ForceNoContentLength;
	use base qw(HTTP::Proxy::BodyFilter);
        sub filter { 1; }
}

my $proxy = HTTP::Proxy->new(port => $port,
			     x_forwarded_for => 0,
			     via => '');
$proxy->logmask(HTTP::Proxy::ALL);
$proxy->push_filter(
	request => HTTP::Proxy::HeaderFilter::simple->new(
		sub {
			my ($self, $headers, $request) = @_;
			$request->uri->host($destination);
			$request->uri->port(80);
                        $headers->header(Host => $destination);
		}
	)
);
#$proxy->push_filter(response => ForceNoContentLength->new());
$proxy->start;

Message #20 received at 572276@bugs.debian.org (full text, mbox, reply):

From: Daniel Stenberg <daniel@haxx.se>

To: 572276@bugs.debian.org, libcurl hacking <curl-library@cool.haxx.se>

Subject: Re: Bug#572276: instead of a test...

Date: Tue, 23 Mar 2010 15:13:27 +0100 (CET)

Hey

Concerning this bug report 
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=572276, please subscribe to 
and continue this talk on the curl-library list instead (this mail is sent 
that way as well).

The Debian bug tracker is a horrible place (to try to debug things in).

Your provided script fails to run due to missing dependencies and I don't feel 
like chasing down CPAN modules. Can you you please see why lib/http.c:3340 
isn't enough to cover for this problem?

-- 

 / daniel.haxx.se

Message #25 received at 572276@bugs.debian.org (full text, mbox, reply):

From: Daniel Stenberg <daniel@haxx.se>

To: libcurl development <curl-library@cool.haxx.se>

Cc: 572276@bugs.debian.org

Subject: Re: Bug#572276: instead of a test...

Date: Tue, 23 Mar 2010 15:23:57 +0100 (CET)

On Tue, 23 Mar 2010, Daniel Stenberg wrote:

> Can you you please see why lib/http.c:3340 isn't enough to cover for this 
> problem?

Hm, it's because of line lib/http.c:3561 which sets maxdownload so it is 
already set later on when it spots that both chunked encoded and 
content-length is specified, but only ->size is set to -1 and maxdownload is 
left untouch then.

Akos Pasztory's original suggestion seems like the accurate fix. Thanks.

-- 

 / daniel.haxx.se

Message #30 received at 572276-close@bugs.debian.org (full text, mbox, reply):

From: Ramakrishnan Muthukrishnan <rkrishnan@debian.org>

To: 572276-close@bugs.debian.org

Subject: Bug#572276: fixed in curl 7.20.1-1

Date: Tue, 20 Apr 2010 04:17:21 +0000

Source: curl
Source-Version: 7.20.1-1

We believe that the bug you reported is fixed in the latest version of
curl, which is due to be installed in the Debian FTP archive:

curl_7.20.1-1.debian.tar.gz
  to main/c/curl/curl_7.20.1-1.debian.tar.gz
curl_7.20.1-1.dsc
  to main/c/curl/curl_7.20.1-1.dsc
curl_7.20.1-1_amd64.deb
  to main/c/curl/curl_7.20.1-1_amd64.deb
curl_7.20.1.orig.tar.gz
  to main/c/curl/curl_7.20.1.orig.tar.gz
libcurl3-dbg_7.20.1-1_amd64.deb
  to main/c/curl/libcurl3-dbg_7.20.1-1_amd64.deb
libcurl3-gnutls_7.20.1-1_amd64.deb
  to main/c/curl/libcurl3-gnutls_7.20.1-1_amd64.deb
libcurl3_7.20.1-1_amd64.deb
  to main/c/curl/libcurl3_7.20.1-1_amd64.deb
libcurl4-gnutls-dev_7.20.1-1_amd64.deb
  to main/c/curl/libcurl4-gnutls-dev_7.20.1-1_amd64.deb
libcurl4-openssl-dev_7.20.1-1_amd64.deb
  to main/c/curl/libcurl4-openssl-dev_7.20.1-1_amd64.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 572276@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Ramakrishnan Muthukrishnan <rkrishnan@debian.org> (supplier of updated curl package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Mon, 19 Apr 2010 09:21:35 +0530
Source: curl
Binary: curl libcurl3 libcurl3-gnutls libcurl4-openssl-dev libcurl4-gnutls-dev libcurl3-dbg
Architecture: source amd64
Version: 7.20.1-1
Distribution: unstable
Urgency: low
Maintainer: Ramakrishnan Muthukrishnan <rkrishnan@debian.org>
Changed-By: Ramakrishnan Muthukrishnan <rkrishnan@debian.org>
Description: 
 curl       - Get a file from an HTTP, HTTPS or FTP server
 libcurl3   - Multi-protocol file transfer library (OpenSSL)
 libcurl3-dbg - libcurl compiled with debug symbols
 libcurl3-gnutls - Multi-protocol file transfer library (GnuTLS)
 libcurl4-gnutls-dev - Development files and documentation for libcurl (GnuTLS)
 libcurl4-openssl-dev - Development files and documentation for libcurl (OpenSSL)
Closes: 572276 576237
Changes: 
 curl (7.20.1-1) unstable; urgency=low
 .
   * New upstream release.
   * debian/patches/missing-double-quote: No longer needed as it has been
     fixed by the upstream.
   * debian/patches/no_com_err: Reworked the patches for the new release.
   * debian/patches/versioned: fix for build failure of 'make test'.
     (closes: #576237)
   * debian/rules: removed --enable-ldaps option from the configure as LDAP
     SSL (Novell extensions to openldap) is not available as Debian packages.
   * lib/http.c: chunked-encoding with Content-Length header problem has
     been fixed in the upstream. (closes: #572276)
Checksums-Sha1: 
 f738dab99b56cda33a2893c2af7702d17018ab35 1483 curl_7.20.1-1.dsc
 766764281435a070445004d1ebb4fb3237cb6dd3 2652616 curl_7.20.1.orig.tar.gz
 f100728e7f29fd08a7c1839001b45217c812aa69 87362 curl_7.20.1-1.debian.tar.gz
 5f0218205574443b0840af38f20e4f6aa958e1d6 222592 curl_7.20.1-1_amd64.deb
 5d8584933ccbb2281ae55cdd89bf9233b93f11de 271060 libcurl3_7.20.1-1_amd64.deb
 2cf3eed9b1187803d78bc147e836d083d68c68ef 251496 libcurl3-gnutls_7.20.1-1_amd64.deb
 73c5a083b45bede2c1f87efd28481bde6d50490e 1070996 libcurl4-openssl-dev_7.20.1-1_amd64.deb
 50b0af716db1915011b9052335254f2f9797f2a5 1046256 libcurl4-gnutls-dev_7.20.1-1_amd64.deb
 4164b51b91c6c74575ebfbc653ea13a0dff84af3 100800 libcurl3-dbg_7.20.1-1_amd64.deb
Checksums-Sha256: 
 1dc295b14e3e07fba119ad6d0c27df2a81f6809bae1d09152d2fd88eecb6b75a 1483 curl_7.20.1-1.dsc
 694d9062280b08b99ba483d04f91bbe63b01a8a116fd36c31ce513c220f6d640 2652616 curl_7.20.1.orig.tar.gz
 0c006db8be731e3d74c0789c268ed48bf4e041cce24c54e19c1a213aaf44ca2b 87362 curl_7.20.1-1.debian.tar.gz
 dd0bec34d83ad40181236e3cffbd33cd835129e0b657aa015c36385497a467ac 222592 curl_7.20.1-1_amd64.deb
 d3b3c151f522ed99e4f47e7974a51e5759f4851f4879d0c77934859ea51cb6df 271060 libcurl3_7.20.1-1_amd64.deb
 078021439de61a633ec90e0bee5e0c10f226957a8a1325693991843428723f00 251496 libcurl3-gnutls_7.20.1-1_amd64.deb
 b97bfd68b93a3ecdbad64299bfc24a671ee563e2abe96e6ab52b9c58d5b7eef5 1070996 libcurl4-openssl-dev_7.20.1-1_amd64.deb
 3a028dfaf1ecfdd742258ccda4eb0d58446b2fa6a90e4c121f194a4be7bb685a 1046256 libcurl4-gnutls-dev_7.20.1-1_amd64.deb
 986a018c32b878d63302af0255fd58b56311458a885c39ff6562af6f0ab4ef2f 100800 libcurl3-dbg_7.20.1-1_amd64.deb
Files: 
 f292f23124c0d9c44d6e291d66573459 1483 web optional curl_7.20.1-1.dsc
 d7df0507db48546661b155b612cac19c 2652616 web optional curl_7.20.1.orig.tar.gz
 fd460de203bf0fc9570b71b6e0f56249 87362 web optional curl_7.20.1-1.debian.tar.gz
 847d345a3ea28eaf74c9febd77e122e6 222592 web optional curl_7.20.1-1_amd64.deb
 666aea4dbfdcac2231c6043807338131 271060 libs optional libcurl3_7.20.1-1_amd64.deb
 c9946ea082a95ccdf23a8a7b15b3e1a1 251496 libs optional libcurl3-gnutls_7.20.1-1_amd64.deb
 11d129755b50cf12d04ee5da8a4413ac 1070996 libdevel optional libcurl4-openssl-dev_7.20.1-1_amd64.deb
 418ddb79b833d2957565eb87fcb193d8 1046256 libdevel optional libcurl4-gnutls-dev_7.20.1-1_amd64.deb
 69d6f0d991c4b07ef7ae1199fbf053f5 100800 debug extra libcurl3-dbg_7.20.1-1_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkvNKOYACgkQFyn1hmqfPDgk3QCeP9ZrJkHS/TjAH37fuxCgmsrB
FwUAoKO6Bh0VWO9xUHTGQ0D0wnokNZ6e
=DJCE
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.