Acknowledgement sent
to Brian Kroth <bpkroth@gmail.com>:
New Bug report received and forwarded. Copy sent to Debian Apache Maintainers <debian-apache@lists.debian.org>.
(Tue, 21 Mar 2017 17:00:04 GMT) (full text, mbox, link).
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: apache2: segfaults upon recieving bad request when using
worker/event mpm and cgid errordoc
Date: Tue, 21 Mar 2017 11:56:40 -0500
Package: apache2.2-common
Version: 2.2.22-13+deb7u8
Severity: normal
Tags: security
Dear Maintainer,
We have some websites running on Debian Wheezy, so still using Apache
2.2.22, that are configured either in Worker or Event MPM (so are using
mod_cgid in what follows), and have a custom "ErrorDocument 400"
directive that points at a perl script for providing custom ModSecurity
error.
I haven't dug up an older version of the package from snapshots to
confirm this, but I think that since the recently backported
HttpProtocolOptions directive to that version (BTW, where was that
announced - I had to run strings on the binary to find it), I've been
seeing a lot of segfault/coredumps registered in the Apache error logs.
After some analysis, I've found that I can reproduce the error with a
fairly trivial shell command:
# echo -ne "GET / HTTP/1.0\n" | nc $some_website 80
>From the coredump, I was able to find that this line (1371) in the
cgid_handler() code in the modules/generators/mod_cgid.c source file has
a null pointer issue on the r->protocol field:
is_included = !strcmp(r->protocol, "INCLUDED");
Seems like a bit of a security issue to me.
No combination of adjustments to the HttpProtocolOptions directive
seemed to help from what I could see.
I also haven't been able to reproduce this issue on a Prefork MPM
backend webserver.
Varying the details of the perl ErrorDocument script's implementation
don't appear to help either (eg: it still occurs even with a simple
hello world script).
The error still occurs if I disable ModSecurity, but leave the
ErrorDocument for 400 messages.
Let me know if you need any more details or have any questions.
Thanks,
Brian
-- Package-specific info:
List of /etc/apache2/mods-enabled/*.load:
alias auth_basic authn_file authz_default authz_groupfile
authz_host authz_user autoindex cgid deflate dir env expires
headers include info mime mod-security negotiation reqtimeout
rewrite rpaf setenvif shib2 ssl status unique_id userdir*
(A * means that the .conf file for that module is not enabled in
/etc/apache2/mods-enabled/)
-- System Information:
Debian Release: 7.11
APT prefers oldstable
APT policy: (500, 'oldstable')
Architecture: amd64 (x86_64)
Kernel: Linux 3.16.0-0.bpo.4-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages apache2 depends on:
ii apache2-mpm-worker 2.2.22-13+deb7u8
ii apache2.2-common 2.2.22-13+deb7u8
apache2 recommends no packages.
apache2 suggests no packages.
Versions of packages apache2.2-common depends on:
ii apache2-utils 2.2.22-13+deb7u8
ii apache2.2-bin 2.2.22-13+deb7u8
ii lsb-base 4.1+Debian8+deb7u1
ii mime-support 3.52-1+deb7u1
ii perl 5.14.2-21+deb7u4
ii procps 1:3.3.3-3
Versions of packages apache2.2-common recommends:
pn ssl-cert <none>
Versions of packages apache2.2-common suggests:
pn apache2-doc <none>
pn apache2-suexec | apache2-suexec-custom <none>
ii lynx-cur [www-browser] 2.8.8dev.12-2+deb7u1
-- no debconf information
Acknowledgement sent
to Doran Moppert <dmoppert@redhat.com>:
Extra info received and forwarded to list. Copy sent to Debian Apache Maintainers <debian-apache@lists.debian.org>.
(Tue, 11 Apr 2017 02:39:02 GMT) (full text, mbox, link).
Acknowledgement sent
to Antoine Beaupre <anarcat@orangeseeds.org>:
Extra info received and forwarded to list. Copy sent to Debian Apache Maintainers <debian-apache@lists.debian.org>.
(Wed, 19 Jul 2017 18:18:03 GMT) (full text, mbox, link).
Hi!
First, thank you very much for the detailed bug report, very useful!
Responses inline.
On Tue, Mar 21, 2017 at 11:56:40AM -0500, Brian Kroth wrote:
> Package: apache2.2-common
> Version: 2.2.22-13+deb7u8
> Severity: normal
> Tags: security
>
> Dear Maintainer,
>
> We have some websites running on Debian Wheezy, so still using Apache
> 2.2.22, that are configured either in Worker or Event MPM (so are using
> mod_cgid in what follows), and have a custom "ErrorDocument 400" directive
> that points at a perl script for providing custom ModSecurity error.
Interesting. It would have been useful to see actual configuration
examples, as I had to go through a little bit of digging to reproduce
the issue. Just using cgid or worker is not sufficient - you actually
*do* need a ErrorDocument directive as well. So thanks to providing
those details, but snippets would have helped as well! :)
> I haven't dug up an older version of the package from snapshots to confirm
> this, but I think that since the recently backported HttpProtocolOptions
> directive to that version (BTW, where was that announced - I had to run
> strings on the binary to find it), I've been seeing a lot of
> segfault/coredumps registered in the Apache error logs.
I'm sorry to hear that. The change was announced in DLA-841-1 back in
February:
https://lists.debian.org/debian-lts-announce/2017/02/msg00031.html
If you haven't seen that announcement, you may not be subscribed to the
debian-lts-announce mailing list, in which case I strongly suggest you
subscribe:
https://lists.debian.org/debian-lts-announce/
There was also a lengthy discussion regarding the patchset on the
debian-lts mailing list:
https://lists.debian.org/87fukh7hcq.fsf@curie.anarc.at
Participation to the debian-lts mailing list is not mandatory for LTS
users, however...
> After some analysis, I've found that I can reproduce the error with a fairly
> trivial shell command:
>
> # echo -ne "GET / HTTP/1.0\n" | nc $some_website 80
This definitely looks like a heading parsing regression caused by
DLA-841-1. I cannot reproduce with a vanilla Apache install.
To reproduce, on wheezy:
1. apt-get install apache2
2. create a dummy perl script in /usr/lib/cgi-bin/hello.pl
#!/usr/bin/perl
print "Content-type: text/html\n\n";
print "Hello, World.";
3. make it executable
4. add this directive, say in /etc/apache2/conf.d/hello
ErrorDocument 400 /cgi-bin/hello.pl
5. reload apache (apache2ctl graceful or whatever)
6. issue the killer request:
echo -ne "GET / HTTP/1.0\n" | nc localhost 80
> > From the coredump, I was able to find that this line (1371) in the
> cgid_handler() code in the modules/generators/mod_cgid.c source file has a
> null pointer issue on the r->protocol field:
>
> is_included = !strcmp(r->protocol, "INCLUDED");
>
> Seems like a bit of a security issue to me.
Definifely.
[...]
On Tue, Apr 11, 2017 at 12:08:11PM +0930, Doran Moppert wrote:
> This looks like a form of CVE-2015-0253, which affected upstream apache
> 2.4.11, was introduced by the backport. The fix is to ensure
> r->protocol is always populated:
>
> https://svn.apache.org/viewvc?view=revision&revision=1668879
Good findings. The fun part is that the above patch doesn't apply
because the protocol initialization was just missing. So this is
possibly broader than just 400 errors.
Note that I have tested this in jessie, and it doesn't seem affected, so
the backport for CVE-2016-8743 was done correctly there.
I'll prepare a package to fix this shortly.
A.
--
Cyberspace. A consensual hallucination experienced daily by billions
of legitimate operators, in every nation, by children being taught
mathematical concepts...
- William Gibson
Acknowledgement sent
to Antoine Beaupre <anarcat@orangeseeds.org>:
Extra info received and forwarded to list. Copy sent to Debian Apache Maintainers <debian-apache@lists.debian.org>.
(Wed, 19 Jul 2017 19:48:03 GMT) (full text, mbox, link).
Hi,
(Sorry for the large CC list, but I am hoping to get a broad approval of
the next changes for this in order to avoid previous mistakes. ;) In
particular, I'd be very grateful for some input by Stefan considering
his knowledge of the Apache codebase and how ... exotic this problems
is.)
As I mentioned in the #858373 bug report, I started looking at fixing
the regression introduced by the 2.2.22-13+deb7u8 upload, part of
DLA-841-1. The problem occurs when a CGI(d) ErrorDocument is configured
to handle 400 error messages that can be triggered with a simple "GET /
HTTP/1.0\n\n". Such a request segfaults Apache in Wheezy right now.
I have been able to confirm that there is an unitialized variable that
gets carried around. This issue was introduced as part of
CVE-2016-8743-aux.patch in the original upload, although I fail to
remember now why this hunk is there exactly. It seems to be related to a
patch I somewhat blindly and incorrectly merged (see
87r33tqvqs.fsf@curie.anarc.at for details).
Unfortunately, re-introducing the protocol initialization code isn't
sufficient: it does fix the segfaults, but the ErrorDocument handling is
not quite working yet. Instead of seeing the output of the
ErrorDocument, after 10 seconds, I get the raw 400 message, doubled with
a 500 error document warning:
$ echo -ne "GET /foo HTTP/1.0\n\n" | nc localhost 80
HTTP/1.1 400 Bad Request
Date: Wed, 19 Jul 2017 19:11:13 GMT
Server: Apache/2.2.22 (Debian)
Vary: Accept-Encoding
Content-Length: 433
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
<p>Additionally, a 500 Internal Server Error
error was encountered while trying to use an ErrorDocument to handle the request.</p>
<hr>
<address>Apache/2.2.22 (Debian) Server at wheezy.raw Port 80</address>
</body></html>
In the error log, I see:
[Wed Jul 19 19:11:23 2017] [error] [client 127.0.0.1] (70007)The timeout
specified has expired: Error reading request entity data
The first part of the error is mod_reqtimeout kicking in as the request
parser stalls on the CGI script. The second part is mod_cgi(d) failing
to read the request from the CGI script, obviously.
My theory is that there is *still* something wrong with the request
parser, even after fixing the r->protocol initialization flaw. I base
this theory on the fact that a 404 ErrorDocument works without problem.
$ echo -ne "GET /foo HTTP/1.0\r\n\r\n" | nc localhost 80
HTTP/1.1 404 Not Found
Date: Wed, 19 Jul 2017 19:13:44 GMT
Server: Apache/2.2.22 (Debian)
Vary: Accept-Encoding
Connection: close
Content-Type: text/html
Hello, World.
Note that I have also tried to see if sending "\r\n" instead of just
"\n" in my "hello world" example would work around the issue: it
doesn't, unfortunately.
I am at a loss as where to go from here, to be honest. The patch
(attached) at least fixes the segfault, which resolves the primary issue
at hand here (DoS by crashing processes!) but it would be nice to
actually fix the ErrorDocument as well..
Any ideas?
Thanks in advance,
A.
Acknowledgement sent
to Antoine Beaupré <anarcat@orangeseeds.org>:
Extra info received and forwarded to list. Copy sent to Debian Apache Maintainers <debian-apache@lists.debian.org>.
(Wed, 19 Jul 2017 19:51:03 GMT) (full text, mbox, link).
And then, obviously, I forget the patch.
Sorry for the noise.
--
The secret of life is to have no fear; it's the only way to function.
- Stokely Carmichael
Acknowledgement sent
to Stefan Fritsch <sf@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Apache Maintainers <debian-apache@lists.debian.org>.
(Fri, 21 Jul 2017 07:33:03 GMT) (full text, mbox, link).
Cc: Brian Kroth <bpkroth@gmail.com>, 858373@bugs.debian.org, Doran Moppert <dmoppert@redhat.com>, debian-apache@lists.debian.org, Chris Lamb <lamby@debian.org>, Raphael Hertzog <hertzog@debian.org>, debian-lts@lists.debian.org, mejo@debian.org
Subject: Re: help needed to complete regression fix for apache2 Bug#858373
Date: Fri, 21 Jul 2017 09:24:00 +0200
Hi Antoine,
On Wednesday, 19 July 2017 15:45:20 CEST Antoine Beaupre wrote:
> As I mentioned in the #858373 bug report, I started looking at fixing
> the regression introduced by the 2.2.22-13+deb7u8 upload, part of
> DLA-841-1. The problem occurs when a CGI(d) ErrorDocument is configured
> to handle 400 error messages that can be triggered with a simple "GET /
> HTTP/1.0\n\n". Such a request segfaults Apache in Wheezy right now.
> Unfortunately, re-introducing the protocol initialization code isn't
> sufficient: it does fix the segfaults, but the ErrorDocument handling is
> not quite working yet. Instead of seeing the output of the
> ErrorDocument, after 10 seconds, I get the raw 400 message, doubled with
> a 500 error document warning:
> Note that I have also tried to see if sending "\r\n" instead of just
> "\n" in my "hello world" example would work around the issue: it
> doesn't, unfortunately.
>
> I am at a loss as where to go from here, to be honest. The patch
> (attached) at least fixes the segfault, which resolves the primary issue
> at hand here (DoS by crashing processes!) but it would be nice to
> actually fix the ErrorDocument as well..
This sounds familiar. Maybe it's simply broken in 2.2.22. Can you compare with
2.2.22-13+deb7u7 if that bug has been there already?
In 2.2.30, there is this fix, which is obviously missing from 2.2.22:
*) core, modules: Avoid error response/document handling by the core if some
handler or input filter already did it while reading the request (causing
a double response body). [Yann Ylavic]
I could not find a changelog entry about the 10s delay, but it's possible that
that has been fixed as well. If the issue is not a regression, you should
simply release the patch that you have. The fix for the error document seems
rather invasive:
https://svn.apache.org/r1683808
Cheers,
Stefan
Acknowledgement sent
to Antoine Beaupré <anarcat@orangeseeds.org>:
Extra info received and forwarded to list. Copy sent to Debian Apache Maintainers <debian-apache@lists.debian.org>.
(Fri, 21 Jul 2017 13:48:03 GMT) (full text, mbox, link).
Cc: Brian Kroth <bpkroth@gmail.com>, 858373@bugs.debian.org, Doran Moppert <dmoppert@redhat.com>, debian-apache@lists.debian.org, Chris Lamb <lamby@debian.org>, Raphael Hertzog <hertzog@debian.org>, debian-lts@lists.debian.org, mejo@debian.org
Subject: Re: help needed to complete regression fix for apache2 Bug#858373
Date: Fri, 21 Jul 2017 09:44:38 -0400
TL;DR: New proposed package (deb7u11) doesn't actually show a new
regression, please test:
https://people.debian.org/~anarcat/debian/wheezy-lts/apache2_2.2.22-13+deb7u11_amd64.changes
In particular, Brian Kroth: are you *sure* you had that ErrorDocument
400 working in apache2_2.2.22-13+deb7u7 (ie. before the DLA-841-1
upload)? In my tests, it didn't actually work at all. It wouldn't
trigger a segfault, but the CGI script wouldn't get called either. In
the above package, we don't segfault anymore, but we yield a 400 + 500
error message (because the ErrorDocument fails). The solution, here, is
obviously to update to a later Apache version (e.g. update to jessie,
really) to get that functionality working, from my perspective.
More technical details follow.
On 2017-07-21 09:24:00, Stefan Fritsch wrote:
> Hi Antoine,
>
> On Wednesday, 19 July 2017 15:45:20 CEST Antoine Beaupre wrote:
>> As I mentioned in the #858373 bug report, I started looking at fixing
>> the regression introduced by the 2.2.22-13+deb7u8 upload, part of
>> DLA-841-1. The problem occurs when a CGI(d) ErrorDocument is configured
>> to handle 400 error messages that can be triggered with a simple "GET /
>> HTTP/1.0\n\n". Such a request segfaults Apache in Wheezy right now.
>
>> Unfortunately, re-introducing the protocol initialization code isn't
>> sufficient: it does fix the segfaults, but the ErrorDocument handling is
>> not quite working yet. Instead of seeing the output of the
>> ErrorDocument, after 10 seconds, I get the raw 400 message, doubled with
>> a 500 error document warning:
>
>> Note that I have also tried to see if sending "\r\n" instead of just
>> "\n" in my "hello world" example would work around the issue: it
>> doesn't, unfortunately.
>>
>> I am at a loss as where to go from here, to be honest. The patch
>> (attached) at least fixes the segfault, which resolves the primary issue
>> at hand here (DoS by crashing processes!) but it would be nice to
>> actually fix the ErrorDocument as well..
>
> This sounds familiar. Maybe it's simply broken in 2.2.22. Can you compare with
> 2.2.22-13+deb7u7 if that bug has been there already?
Well, the problem is - how do I reproduce this? I can't generate the
same 400 error message in deb7u7 (I tried!) with the previous techniques
because the new request handling code isn't there. That is, the
following query just works:
# printf "GET / HTTP/1.0\n\n" | nc localhost 80 | head -1
HTTP/1.1 200 OK
Furthermore, generating a 400 error, when it works in deb7u7, doesn't
trigger the ErrorDocument - not sure why:
# printf "G ET / HTTP/1.0\r\n\r\n" | nc localhost 80
HTTP/1.1 400 Bad Request
Date: Fri, 21 Jul 2017 13:40:48 GMT
Server: Apache/2.2.22 (Debian)
Vary: Accept-Encoding
Content-Length: 302
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
<hr>
<address>Apache/2.2.22 (Debian) Server at wheezy.raw Port 80</address>
</body></html>
Logs show the following:
[Fri Jul 21 13:40:48 2017] [error] [client 127.0.0.1] Invalid URI in request G ET / HTTP/1.0
... whether or not the 400 ErrorDocument directive is present. Notice
how the ErrorDocument isn't triggered at all here.
Of course, a 404 ErrorDocument still works correctly:
# printf "GET /wtf HTTP/1.0\r\n\r\n" | nc localhost 80
HTTP/1.1 404 Not Found
Date: Fri, 21 Jul 2017 13:23:46 GMT
Server: Apache/2.2.22 (Debian)
Vary: Accept-Encoding
Connection: close
Content-Type: text/plain
Hello, World.
I get this behavior consistently with deb7u7 and the proposed deb7u11
(which only adds a 500 error document to *certain* 400 errors,
basically). I find that is an acceptable compromise to fix a segfault,
and, from my perspective, doesn't introduce a regression.
> In 2.2.30, there is this fix, which is obviously missing from 2.2.22:
>
> *) core, modules: Avoid error response/document handling by the core if some
> handler or input filter already did it while reading the request (causing
> a double response body). [Yann Ylavic]
>
> I could not find a changelog entry about the 10s delay, but it's possible that
> that has been fixed as well. If the issue is not a regression, you should
> simply release the patch that you have. The fix for the error document seems
> rather invasive:
>
> https://svn.apache.org/r1683808
But that's another big patch to backport:
20 files changed, 196 insertions(+), 129 deletions(-)
Not sure we want to pile yet another backport on top of the pile we
already have. Now I really regret not updating to 2.2.34. :(
Since this issue doesn't seem to be a regression (the ErrorDocument
didn't seem to get called at all, previously), I think I'll just post a
test package with the regression fix and be done with it for now.
I'm more confident in the upload now, and hopefully it won't break too
many things now. At least we don't segfault. ;)
I'll be available to upload the test package tomorrow or by the end of
next week, if there are no regressions. I'd be glad if someone else
could do some more smoke tests here, in particular the original
submitter who has such a great test environment. :)
Thanks for the feedback, Stefan!
A.
--
Dans vos mensonges de pierre
Vous gaspillez le soleil
- Gilles Vigneault
Acknowledgement sent
to Antoine Beaupré <anarcat@orangeseeds.org>:
Extra info received and forwarded to list. Copy sent to Debian Apache Maintainers <debian-apache@lists.debian.org>.
(Sat, 29 Jul 2017 17:09:03 GMT) (full text, mbox, link).
Cc: Brian Kroth <bpkroth@gmail.com>, 858373@bugs.debian.org, Doran Moppert <dmoppert@redhat.com>, debian-apache@lists.debian.org, Chris Lamb <lamby@debian.org>, Raphael Hertzog <hertzog@debian.org>, debian-lts@lists.debian.org, mejo@debian.org
Subject: Re: help needed to complete regression fix for apache2 Bug#858373
Date: Sat, 29 Jul 2017 13:07:37 -0400
Control: fixed 858373 2.2.22-13+deb7u7
Control: tags 858373 +pending +patch
On 2017-07-21 09:44:38, Antoine Beaupré wrote:
> TL;DR: New proposed package (deb7u11) doesn't actually show a new
> regression, please test:
>
> https://people.debian.org/~anarcat/debian/wheezy-lts/apache2_2.2.22-13+deb7u11_amd64.changes
>
> In particular, Brian Kroth: are you *sure* you had that ErrorDocument
> 400 working in apache2_2.2.22-13+deb7u7 (ie. before the DLA-841-1
> upload)? In my tests, it didn't actually work at all. It wouldn't
> trigger a segfault, but the CGI script wouldn't get called either. In
> the above package, we don't segfault anymore, but we yield a 400 + 500
> error message (because the ErrorDocument fails). The solution, here, is
> obviously to update to a later Apache version (e.g. update to jessie,
> really) to get that functionality working, from my perspective.
Timing out on this one: I will assume that 2.2.22-13+deb7u7 didn't
segfault, but then didn't yield a proper ErrorDocument either (because I
cannot reproduce that behavior).
I have uploaded deb7u11 and will send the associated DLA-841-2
regression update when it hits the archives.
A.
--
Seul a un caractère scientifique ce qui peut être réfuté. Ce qui n'est
pas réfutable relève de la magie ou de la mystique.
- Karl Popper
Marked as fixed in versions 2.2.22-13+deb7u7.
Request was from Antoine Beaupré <anarcat@orangeseeds.org>
to 858373-submit@bugs.debian.org.
(Sat, 29 Jul 2017 17:09:03 GMT) (full text, mbox, link).
Added tag(s) pending.
Request was from Antoine Beaupré <anarcat@orangeseeds.org>
to 858373-submit@bugs.debian.org.
(Sat, 29 Jul 2017 17:09:03 GMT) (full text, mbox, link).
Added tag(s) patch.
Request was from Antoine Beaupré <anarcat@orangeseeds.org>
to 858373-submit@bugs.debian.org.
(Sat, 29 Jul 2017 17:09:04 GMT) (full text, mbox, link).
Acknowledgement sent
to Brian Kroth <bpkroth@gmail.com>:
Extra info received and forwarded to list. Copy sent to Debian Apache Maintainers <debian-apache@lists.debian.org>.
(Sun, 30 Jul 2017 01:06:03 GMT) (full text, mbox, link).
Hi, sorry for the delay. Gmail filed this one into spam :-(
Unfortunately, I don't have access to that environment anymore to confirm.
I'll pass this on to the folks that do so hopefully they can.
My recollection from this issue was that I'd tested it against different
package versions and the 400 ErrorDocuments had worked beforehand (we used
them for ModSec types of things primarily, and I'm confident that mode was
working well before hand and after), though possibly not in that particular
protocol error context. I vaguely recall having issues reproducing a
working ErrorDocument with non-cgi methods in that protocol error mode
style test as well, but I don't recall if rhat was only in the newer
versions of the software that I had been testing with or true before that
update as well.
Anyways, thanks much for following up. Sorry I don't have more info to
offer at the moment.
Cheers,
Brian
On Fri, Jul 21, 2017, 08:44 Antoine Beaupré <anarcat@orangeseeds.org> wrote:
> TL;DR: New proposed package (deb7u11) doesn't ctually show a new
> regression, please test:
>
>
> https://people.debian.org/~anarcat/debian/wheezy-lts/apache2_2.2.22-13+deb7u11_amd64.changes
>
> In particular, Brian Kroth: are you *sure* you had that ErrorDocument
> 400 working in apache2_2.2.22-13+deb7u7 (ie. before the DLA-841-1
> upload)? In my tests, it didn't actually work at all. It wouldn't
> trigger a segfault, but the CGI script wouldn't get called either. In
> the above package, we don't segfault anymore, but we yield a 400 + 500
> error message (because the ErrorDocument fails). The solution, here, is
> obviously to update to a later Apache version (e.g. update to jessie,
> really) to get that functionality working, from my perspective.
>
> More technical details follow.
>
> On 2017-07-21 09:24:00, Stefan Fritsch wrote:
> > Hi Antoine,
> >
> > On Wednesday, 19 July 2017 15:45:20 CEST Antoine Beaupre wrote:
> >> As I mentioned in the #858373 bug report, I started looking at fixing
> >> the regression introduced by the 2.2.22-13+deb7u8 upload, part of
> >> DLA-841-1. The problem occurs when a CGI(d) ErrorDocument is configured
> >> to handle 400 error messages that can be triggered with a simple "GET /
> >> HTTP/1.0\n\n". Such a request segfaults Apache in Wheezy right now.
> >
> >> Unfortunately, re-introducing the protocol initialization code isn't
> >> sufficient: it does fix the segfaults, but the ErrorDocument handling is
> >> not quite working yet. Instead of seeing the output of the
> >> ErrorDocument, after 10 seconds, I get the raw 400 message, doubled with
> >> a 500 error document warning:
> >
> >> Note that I have also tried to see if sending "\r\n" instead of just
> >> "\n" in my "hello world" example would work around the issue: it
> >> doesn't, unfortunately.
> >>
> >> I am at a loss as where to go from here, to be honest. The patch
> >> (attached) at least fixes the segfault, which resolves the primary issue
> >> at hand here (DoS by crashing processes!) but it would be nice to
> >> actually fix the ErrorDocument as well..
> >
> > This sounds familiar. Maybe it's simply broken in 2.2.22. Can you
> compare with
> > 2.2.22-13+deb7u7 if that bug has been there already?
>
> Well, the problem is - how do I reproduce this? I can't generate the
> same 400 error message in deb7u7 (I tried!) with the previous techniques
> because the new request handling code isn't there. That is, the
> following query just works:
>
> # printf "GET / HTTP/1.0\n\n" | nc localhost 80 | head -1
> HTTP/1.1 200 OK
>
>
> Furthermore, generating a 400 error, when it works in deb7u7, doesn't
> trigger the ErrorDocument - not sure why:
>
> # printf "G ET / HTTP/1.0\r\n\r\n" | nc localhost 80
> HTTP/1.1 400 Bad Request
> Date: Fri, 21 Jul 2017 13:40:48 GMT
> Server: Apache/2.2.22 (Debian)
> Vary: Accept-Encoding
> Content-Length: 302
> Connection: close
> Content-Type: text/html; charset=iso-8859-1
>
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>400 Bad Request</title>
> </head><body>
> <h1>Bad Request</h1>
> <p>Your browser sent a request that this server could not understand.<br />
> </p>
> <hr>
> <address>Apache/2.2.22 (Debian) Server at wheezy.raw Port 80</address>
> </body></html>
>
> Logs show the following:
>
> [Fri Jul 21 13:40:48 2017] [error] [client 127.0.0.1] Invalid URI in
> request G ET / HTTP/1.0
>
> ... whether or not the 400 ErrorDocument directive is present. Notice
> how the ErrorDocument isn't triggered at all here.
>
> Of course, a 404 ErrorDocument still works correctly:
>
> # printf "GET /wtf HTTP/1.0\r\n\r\n" | nc localhost 80
> HTTP/1.1 404 Not Found
> Date: Fri, 21 Jul 2017 13:23:46 GMT
> Server: Apache/2.2.22 (Debian)
> Vary: Accept-Encoding
> Connection: close
> Content-Type: text/plain
>
> Hello, World.
>
> I get this behavior consistently with deb7u7 and the proposed deb7u11
> (which only adds a 500 error document to *certain* 400 errors,
> basically). I find that is an acceptable compromise to fix a segfault,
> and, from my perspective, doesn't introduce a regression.
>
> > In 2.2.30, there is this fix, which is obviously missing from 2.2.22:
> >
> > *) core, modules: Avoid error response/document handling by the core
> if some
> > handler or input filter already did it while reading the request
> (causing
> > a double response body). [Yann Ylavic]
> >
> > I could not find a changelog entry about the 10s delay, but it's
> possible that
> > that has been fixed as well. If the issue is not a regression, you should
> > simply release the patch that you have. The fix for the error document
> seems
> > rather invasive:
> >
> > https://svn.apache.org/r1683808
>
> But that's another big patch to backport:
>
> 20 files changed, 196 insertions(+), 129 deletions(-)
>
> Not sure we want to pile yet another backport on top of the pile we
> already have. Now I really regret not updating to 2.2.34. :(
>
> Since this issue doesn't seem to be a regression (the ErrorDocument
> didn't seem to get called at all, previously), I think I'll just post a
> test package with the regression fix and be done with it for now.
>
> I'm more confident in the upload now, and hopefully it won't break too
> many things now. At least we don't segfault. ;)
>
> I'll be available to upload the test package tomorrow or by the end of
> next week, if there are no regressions. I'd be glad if someone else
> could do some more smoke tests here, in particular the original
> submitter who has such a great test environment. :)
>
> Thanks for the feedback, Stefan!
>
> A.
>
> --
> Dans vos mensonges de pierre
> Vous gaspillez le soleil
> - Gilles Vigneault
>
Debbugs is free software and licensed under the terms of the GNU
Public License version 2. The current version can be obtained
from https://bugs.debian.org/debbugs-source/.