Debian Bug report logs - #534982
squid - DoS in external auth header parser

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Bastian Blank <waldi@debian.org>
To: submit@bugs.debian.org
Subject: squid - DoS in external auth header parser
Date: Sun, 28 Jun 2009 20:19:34 +0200

Package: squid
Version: 2.7.STABLE3-4.1
Severity: normal

My main squid reverse proxy suddenly stopped working after some days.
The last time it happened, I managed to dig a bit around and also got a
core dump and analyzed it as far as this works without debugging
symbols. This happened on my own rebuild with SSL enabled, but the
affected code region does not even consider SSL support.

Config excerpt:

| http_port 80 accel vhost defaultsite=example.com
| https_port 443 accel vhost defaultsite=example.com cert=/etc/squid/ssl/all options=NO_SSLv2
| icp_port 3130
| 
| logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %Hs %<st %Ss %Sh/%<A "%{Referer}>h" "%{User-Agent}>h"
| cache_access_log /srv/squid/prod/log/access.log
| cache_access_log /srv/squid/prod/log/combined.log combined
| cache_log /srv/squid/prod/log/cache.log
| cache_store_log /srv/squid/prod/log/store.log
| 
| acl accelerated_domains dstdomain example.com
| acl accelerated_protocols proto http https
| 
| external_acl_type zope_auth ttl=0 %PATH %{Cookie:;__ac} /etc/squid/auth/auth /etc/squid/zope_auth.conf
| acl zope_auth external zope_auth
| 
| http_access allow accelerated_domains accelerated_protocols zope_auth
| http_access deny all

Available threads:

| (gdb) info threads
|   17 process 17096  0x00002b7100488bc8 in strcspn () from /lib/libc.so.6
|   16 process 17138  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   15 process 17137  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   14 process 17136  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   13 process 17135  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   12 process 17134  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   11 process 17133  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   10 process 17132  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   9 process 17131  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   8 process 17130  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   7 process 17129  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   6 process 17128  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   5 process 17127  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   4 process 17126  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   3 process 17125  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
|   2 process 17124  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
| * 1 process 17123  0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

So 16 threads suddenly waited for something shared and only the 17th did
something usefull.

Annotated backtrace of thread 17 (I had to reconstruct the function names from
a similar binary):

| (gdb) bt
| #0  0x00002b7100488bc8 in strcspn () from /lib/libc.so.6
| #1  0x0000000000456021 in ?? ()
0000000000455f80 g     F .text  0000000000000191              strListGetItem
| #2  0x000000000045395e in ?? ()
00000000004538b0 g     F .text  000000000000014a              httpHeaderGetListMember
| #3  0x000000000043923a in ?? ()
0000000000438e60 l     F .text  0000000000000648              makeExternalAclKey
| #4  0x0000000000439f6b in ?? ()
0000000000439e70 g     F .text  000000000000048c              aclMatchExternal
| #5  0x000000000040a24c in ?? ()
0000000000409f30 g     F .text  0000000000000eef              aclMatchAclList
| #6  0x000000000040ae61 in ?? ()
000000000040ae20 l     F .text  000000000000044d              aclCheck
| #7  0x000000000042652b in ?? ()
| #8  0x0000000000431105 in ?? ()
| #9  0x00000000004601a0 in ?? ()
| #10 0x00002b710042c1a6 in __libc_start_main () from /lib/libc.so.6

Register dump to show the parameters for strcspn:

| (gdb) info registers
| rax            0x0      0
| rbx            0x199d93d        26859837
| rcx            0x3      3
| rdx            0x199d93d        26859837
| rsi            0x700690 7341712
| rdi            0x199d93d        26859837
| rbp            0x0      0x0
| rsp            0x7fffab99df88   0x7fffab99df88
| r8             0x199d93d        26859837
| r9             0x1      1
| r10            0x353a39353a333220       3835440933431882272
| r11            0x2b71005153a0   47764336628640
| r12            0x7fffab99e130   140736072376624
| r13            0x7fffab99e128   140736072376616
| r14            0x3b     59
| r15            0x7fffab99e13c   140736072376636
| rip            0x2b7100488bc8   0x2b7100488bc8 <strcspn+24>

Parameters are in rsi and rdi:

| (gdb) print {char[5]}0x700690
| $14 = "\";,\000"
| (gdb) print {char[50]}0x199d93d
| $16 = ", 31-Dec-97 23:59:59 GMT; Max-Age=0", '\0' <repeats 14 times>

So it calls
| strcspn("\";,", ", 31-Dec-97 23:59:59 GMT; Max-Age=0")

But suddenly the value starts with ",", which is defined as a field
delimiter in the Cookie-header, but incorrectly used in this case.

More data:

| (gdb) print {char[100]}0x199d910
| $18 = "statusmessages=\"deleted\"; Path=/; Expires=Wed, 31-Dec-97 23:59:59 GMT; Max-Age=0", '\0' <repeats 19 times>

It loops around this strcspn call all the time. As far as I know always
with the same parameters.

Bastian

-- 
There is an order of things in this universe.
		-- Apollo, "Who Mourns for Adonais?" stardate 3468.1

Message #10 received at 534982@bugs.debian.org (full text, mbox):

From: Bastian Blank <waldi@debian.org>
To: 534982@bugs.debian.org
Subject: Re: Bug#534982: Acknowledgement (squid - DoS in external auth
	header parser)
Date: Sun, 28 Jun 2009 20:46:23 +0200

Digging into the code shows that this behaviour is clearly written
their[1]. It it only triggerable by either external auth or access log
formats which includes parts of headers with delimiters.

Bastian

[1]: src/HttpHeaderTools.c:239-283
-- 
Without facts, the decision cannot be made logically.  You must rely on
your human intuition.
		-- Spock, "Assignment: Earth", stardate unknown

Message #15 received at 534982@bugs.debian.org (full text, mbox):

From: Bastian Blank <waldi@debian.org>
To: 534982@bugs.debian.org
Subject: squid - DoS in external auth helper
Date: Mon, 6 Jul 2009 12:09:17 +0200

severity 534982 important
tags 534982 security
thanks

No response from maintainer. As the cause is clear, I'm setting it to
appropriate values.

Bastian

-- 
There's coffee in that nebula!
		-- Capt. Kathryn Janeway, Star Trek: Voyager, "The Cloud"

Message #24 received at 534982@bugs.debian.org (full text, mbox):

From: Luigi Gangitano <luigi@debian.org>
To: Bastian Blank <waldi@debian.org>,
 534982@bugs.debian.org
Cc: control@bugs.debian.org
Subject: Re: Bug#534982: squid - DoS in external auth helper
Date: Mon, 6 Jul 2009 19:09:05 +0200

forwarded 534982 http://www.squid-cache.org/bugs/show_bug.cgi?id=2704
thanks

Hi Bastian,
sorry for the late reply. I'm forwarding this bug upstream, since the  
issue is better handled there.

Regards,

L

--
Luigi Gangitano -- <luigi@debian.org> -- <gangitano@lugroma3.org>
GPG: 1024D/924C0C26: 12F8 9C03 89D3 DB4A 9972  C24A F19B A618 924C 0C26

Message #31 received at 534982@bugs.debian.org (full text, mbox):

From: Bastian Blank <waldi@debian.org>
To: 534982@bugs.debian.org
Subject: patch
Date: Mon, 10 Aug 2009 20:22:19 +0200

[Message part 1 (text/plain, inline)]

The attached patch fixes the problem by eliminating the additional ","
as seperator. I found no sign why this was added in the first place
after some code reconstruction. Also it makes the code reentrant.

Bastian

-- 
We have phasers, I vote we blast 'em!
		-- Bailey, "The Corbomite Maneuver", stardate 1514.2

[diff (text/plain, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #36 received at 534982@bugs.debian.org (full text, mbox):

From: Henrik Nordstrom <henrik@henriknordstrom.net>
To: 534982@bugs.debian.org
Subject: Re: Bug#534982: squid - DoS in external auth helper
Date: Wed, 19 Aug 2009 13:47:34 +0200

The comma is there because comma is always a valid list separator in
HTTP headers due to rules on how multiple values of the same header may
be combined.

Regards
Henrik

Message #41 received at 534982@bugs.debian.org (full text, mbox):

From: Nico Golde <nion@debian.org>
To: 534982@bugs.debian.org
Subject: CVE id assigned
Date: Thu, 20 Aug 2009 01:04:40 +0200

[Message part 1 (text/plain, inline)]

Hi,
CVE-2009-2855 was assigned to this issue, please make sure 
to reference it in the changelog if you fix this bug.

Cheers
Nico

-- 
Nico Golde - http://www.ngolde.de - nion@jabber.ccc.de - GPG: 0xA0A0AAAA
For security reasons, all text in this mail is double-rot13 encrypted.

[Message part 2 (application/pgp-signature, inline)]

Message #50 received at 534982-close@bugs.debian.org (full text, mbox):

From: Luigi Gangitano <luigi@debian.org>
To: 534982-close@bugs.debian.org
Subject: Bug#534982: fixed in squid 2.7.STABLE7-1
Date: Sun, 20 Sep 2009 12:48:47 +0000

Source: squid
Source-Version: 2.7.STABLE7-1

We believe that the bug you reported is fixed in the latest version of
squid, which is due to be installed in the Debian FTP archive:

squid-cgi_2.7.STABLE7-1_i386.deb
  to pool/main/s/squid/squid-cgi_2.7.STABLE7-1_i386.deb
squid-common_2.7.STABLE7-1_all.deb
  to pool/main/s/squid/squid-common_2.7.STABLE7-1_all.deb
squid_2.7.STABLE7-1.diff.gz
  to pool/main/s/squid/squid_2.7.STABLE7-1.diff.gz
squid_2.7.STABLE7-1.dsc
  to pool/main/s/squid/squid_2.7.STABLE7-1.dsc
squid_2.7.STABLE7-1_i386.deb
  to pool/main/s/squid/squid_2.7.STABLE7-1_i386.deb
squid_2.7.STABLE7.orig.tar.gz
  to pool/main/s/squid/squid_2.7.STABLE7.orig.tar.gz



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 534982@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Luigi Gangitano <luigi@debian.org> (supplier of updated squid package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Sun, 20 Sep 2009 01:25:52 +0200
Source: squid
Binary: squid squid-common squid-cgi
Architecture: source all i386
Version: 2.7.STABLE7-1
Distribution: unstable
Urgency: low
Maintainer: Luigi Gangitano <luigi@debian.org>
Changed-By: Luigi Gangitano <luigi@debian.org>
Description: 
 squid      - Internet object cache (WWW proxy cache)
 squid-cgi  - Squid cache manager CGI program
 squid-common - Internet object cache (WWW proxy cache) - common files
Closes: 534982 541460 542723
Changes: 
 squid (2.7.STABLE7-1) unstable; urgency=low
 .
   * New upstream release
     - Fixes DoS in exthernal auth header parser (Ref: CVE-2009-2855)
       (Closes: #534982)
 .
   * debian/squid.rc
     - Fixed init.d script dependencies, thanks to Petter Reinholdtsen
       (Closes: #541460)
 .
   * debian/{control,rules}
     - Added hardening build options (Closes: #542723)
 .
   * debian/control
     - Bumped Standard-Version to 3.8.3, no change needed
 .
   * debian/rules
     - Make link to squid-langpack directory relative
Checksums-Sha1: 
 d8f1acc9eb27c6abce8021e2590e37f5e9ffef33 1149 squid_2.7.STABLE7-1.dsc
 94a385a494f50cf3394e226ce743ae3a3ad51440 1784325 squid_2.7.STABLE7.orig.tar.gz
 224e01851db33de12922a23bac8db76376eb1540 300374 squid_2.7.STABLE7-1.diff.gz
 8c8e1086d548cd9404a5219cf4f6e7fa7f46ec9f 351002 squid-common_2.7.STABLE7-1_all.deb
 4ce3033cd50e4b41022a03fe263d87f894ddcb92 759998 squid_2.7.STABLE7-1_i386.deb
 fab88dc10c8e7d40ea96b06bfbbb319d9870b956 121302 squid-cgi_2.7.STABLE7-1_i386.deb
Checksums-Sha256: 
 d12f3409fabd082461ca9e3f91bc92d7666dfb41307915a7cc3c2ec82d712542 1149 squid_2.7.STABLE7-1.dsc
 2b926aece7d4beac0370d9b72899d3be7f48c29e973a1328666938a2e4c4ffa6 1784325 squid_2.7.STABLE7.orig.tar.gz
 9c84ede0dbb25df4d6f628b62d967f7b4c264b097290d31f5ff0215d5a836fb9 300374 squid_2.7.STABLE7-1.diff.gz
 e856b380dce72ae41bb6a8a05fe30fe3825b53d65c1263b7b5bad6b85670f331 351002 squid-common_2.7.STABLE7-1_all.deb
 3ab49ccb3a922e1fa7e15d8b6e7a5c0ba6ffa926409015300c06fe3087509d1d 759998 squid_2.7.STABLE7-1_i386.deb
 1516617bdf77a0b2f2cf401d74da6888d6d01a326591631acbe0fbbf436c5762 121302 squid-cgi_2.7.STABLE7-1_i386.deb
Files: 
 03b8ffc57fffb5afa81b891aa8888da1 1149 web optional squid_2.7.STABLE7-1.dsc
 c506207f921a6da1878b4085e202e190 1784325 web optional squid_2.7.STABLE7.orig.tar.gz
 8fcd6411b77bfc1c2d7061c4f9dfabcc 300374 web optional squid_2.7.STABLE7-1.diff.gz
 401495f9e634e7ebfed42363c19cf0f8 351002 web optional squid-common_2.7.STABLE7-1_all.deb
 139c822f0553cb03cd1c4cb3dc89f13b 759998 web optional squid_2.7.STABLE7-1_i386.deb
 d3cb67378334c83132279663d584f3e4 121302 web optional squid-cgi_2.7.STABLE7-1_i386.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iEYEARECAAYFAkq2Ix0ACgkQ8ZumGJJMDCbdtACfW+YqEE/BUJQ4ULW0B3BnOBes
8W0AoIDtAuw7D/6tp0IaN14jjCtDmJuL
=QcCl
-----END PGP SIGNATURE-----

Message #55 received at 534982@bugs.debian.org (full text, mbox):

From: Terry Burton <tez@terryburton.co.uk>
To: 534982@bugs.debian.org
Cc: team@security.debian.org
Subject: squid - DoS in external auth header parser
Date: Wed, 30 Sep 2009 17:08:42 +0100

reopen 534982
severity 534982 critical

This bug has recently hit us hard resulting in repeated DoS of a
production web service running on Debian Lenny.

What is the intended mitigation strategy for this DoS for users of
Debian Stable who rely on Squid support for external_acl_type? For the
time being I have had to rebuild appropriately patched squid packages
for Lenny to guard against this.

Since there will redoubtably be many production web servers running
Debian that are vulnerable to CVE-2009-2855 the patch should be
backported into the squid packages shipped with Lenny and released
through the security repository.


Thanks,

Terry

Message #60 received at 534982@bugs.debian.org (full text, mbox):

From: Florian Weimer <fw@deneb.enyo.de>
To: Terry Burton <tez@terryburton.co.uk>
Cc: 534982@bugs.debian.org,  team@security.debian.org
Subject: Re: squid - DoS in external auth header parser
Date: Wed, 30 Sep 2009 16:49:49 +0000

* Terry Burton:

> This bug has recently hit us hard resulting in repeated DoS of a
> production web service running on Debian Lenny.

Do you know if this was triggered accidentally or deliberately?

> Since there will redoubtably be many production web servers running
> Debian that are vulnerable to CVE-2009-2855 the patch should be
> backported into the squid packages shipped with Lenny and released
> through the security repository.

Luigi, do you plan to prepare an update for (old)stable?

Message #65 received at 534982@bugs.debian.org (full text, mbox):

From: Terry Burton <tez@terryburton.co.uk>
To: Florian Weimer <fw@deneb.enyo.de>
Cc: 534982@bugs.debian.org, team@security.debian.org
Subject: Re: squid - DoS in external auth header parser
Date: Wed, 30 Sep 2009 17:58:36 +0100

On Wed, Sep 30, 2009 at 5:49 PM, Florian Weimer <fw@deneb.enyo.de> wrote:
> * Terry Burton:
>
>> This bug has recently hit us hard resulting in repeated DoS of a
>> production web service running on Debian Lenny.
>
> Do you know if this was triggered accidentally or deliberately?

Florian,

Probably accidentally, though it may possibly have been deliberate.

(I will disclose further information privately as a follow up to this message.)


Many thanks,

Terry

Message #74 received at 534982@bugs.debian.org (full text, mbox):

From: Amos Jeffries <squid3@treenet.co.nz>
To: <534982@bugs.debian.org>
Cc: <team@security.debian.org>
Subject: Re: squid - DoS in external auth header parser
Date: Tue, 13 Oct 2009 11:23:47 +1300

I see "Bug No longer marked as fixed in versions squid/2.7.STABLE7-1 and
reopened." above.

Is this correct and 2.7.STABLE7 still showing the vulnerable behaviour?

Terry:  If possible I'd like to see some details of the event, for my
interest.

Amos
Squid Team

Send a report that this bug log contains spam.