Debian Bug report logs - #342141
apache2: Apache2 weird forks (owned by root)

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Fotos Georgiadis <fotos@trwn.gr>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: apache2: Apache2 weird forks (owned by root)

Date: Mon, 05 Dec 2005 20:47:54 +0200

Package: apache2
Version: 2.0.54-5
Severity: grave
Tags: security
Justification: renders package unusable


(What follow is the original messege I posted first on
the users@httpd.apache.org, but no-one replied after a few days.
So now I report this as a Debian bug...)

Hello!

A strange problem occurs in our apache 2 installation. I have the  
prefork MPM where the expected (and documented) behavior is one root  
process open for managing the privileges ports (etc.) that spawns  
children, with the privileges of the User and Group directives  
specified (www-data in my case), in order to serve the requests.

Well, in our system what happens in that the children also have root  
privileges despite the User www-data option. But what is more strange  
is that this doesn't happen all the time. Sometimes the server starts  
with 3 children owned by www-data and the rest 2 (out of a  
MinSpareServers 5) are owned by root. The number varies between  
restarts from all owned by root to all owned by www-data. (Don't  
mention the security implications of the situation).

Processes owned by root are not serving pages (nor that I would want  
the root user to serve pages...), and that means the capabilities of  
the server are reduced. When all processes are owned by root, and a  
client opens a connection it hangs there indefinitely. When only 1 or  
2 processes are owned by www-data the server is really slow,  
otherwise it behaves nicely under a moderate load.

Also issuing a reload (apache2ctl graceful) seems to zombie the child  
processes and only SIGKILL can make them rest in peace. The problem  
possibly lies in the forking section of apache 2. Killing the root- 
owned children spawns new, sometimes owned by root, sometimes owned  
by www-data. Killing enough root processes eventually allows us to  
have all processes owned by www-data!

Now that I explained the situation and the problem, details about the  
system follow:
Debian Sarge (3.1) up to date

Standard debian package for apache 2:
% apache2ctl -V:
Server version: Apache/2.0.54
Server built:   Sep  5 2005 11:15:09
Server's Module Magic Number: 20020903:9
Architecture:   32-bit
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D HTTPD_ROOT=""
-D SUEXEC_BIN="/usr/lib/apache2/suexec2"
-D DEFAULT_PIDLOG="/var/run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
-D SERVER_CONFIG_FILE="/etc/apache2/apache2.conf"

Using the prefork MPM as already mentioned.

Server signature and other modules:
Apache/2.0.54 (Debian GNU/Linux)
PHP/4.3.10-16
mod_ssl/2.0.54
OpenSSL/0.9.7e Server

The configuration is a mixed IP-Based (using ip aliasing) and Name- 
Based Virtual Hosting, with SSL enabled. Details (and full  
configuration) can be given as requested. The configuration is not  
the problem as the problem occurs even with no sites enabled. Also  
the server works as advertised, excluding the problem I mentioned.

The only "custom" thing on the server is the kernel which is Linux  
2.4.30 (being a Dell server it wouldn't boot with a debian vanilla  
kernel). But I fail to see how the kernel can be a problem in this  
situation (since every other service works perfectly (and there are a  
lot)). It look like a race condition problem.

This on a SMP machine (Xeon @ 3.2Ghz, with 1GB ram), otherwise stable  
as a rock.

I already searched the archives and couldn't find anything similar.
Perhaps I should also post a bug report here (oneline):
http://bugs.debian.org/cgi-bin/pkgreport.cgi? \
which=pkg&data=apache2&archive=no&version=&dist=stable

but I don't know if this is debian specific or an apache 2 problem.

That's all I can think of. Has anybody seen something similar?
Any developer comments? How can I help you trace this (ugly) bug?

Thanks in advance for reading all this (*wow* thats big)!
-fot

PS. On a side-note, as a joke we started placing bets on how many www- 
data owned processes there will be on the next apache 2 restart! :) :)


-- System Information:
Debian Release: 3.1
Architecture: i386 (i686)
Kernel: Linux 2.4.30
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)

Versions of packages apache2 depends on:
ii  apache2-mpm-prefork           2.0.54-5   traditional model for Apache2

-- no debconf information

Message #16 received at 342141@bugs.debian.org (full text, mbox, reply):

From: Josip Rodin <joy@debbugs.entuzijast.net>

To: Fotos Georgiadis <fotos@trwn.gr>

Cc: 342141@bugs.debian.org

Subject: Re: apache2: Apache2 weird forks (owned by root)

Date: Sun, 15 Oct 2006 23:39:11 +0200

On Mon, Dec 05, 2005 at 08:47:54PM +0200, Fotos Georgiadis wrote:
> Package: apache2
> Version: 2.0.54-5
> 
> Well, in our system what happens in that the children also have root  
> privileges despite the User www-data option. But what is more strange  
> is that this doesn't happen all the time.

> Processes owned by root are not serving pages (nor that I would want  
> the root user to serve pages...), and that means the capabilities of  
> the server are reduced.

> That's all I can think of. Has anybody seen something similar?

I have seen this bug several times on a moderately-used production server
running sarge's apache2. The only peculiar thing about that installation
is that it runs a RewriteMap, but other than that, it's all pretty vanilla.

Here the problem could always be worked around simply by restarting apache.

> It look like a race condition problem.

(To me it looks like what the Jargon File would describe as a mandelbug. :)

-- 
     2. That which causes joy or happiness.

Message #21 received at 342141@bugs.debian.org (full text, mbox, reply):

From: Stefan Fritsch <sf@sfritsch.de>

To: 342141-submitter@bugs.debian.org

Cc: 342141@bugs.debian.org, Josip Rodin <joy@debbugs.entuzijast.net>

Subject: apache2: Apache2 weird forks (owned by root)

Date: Fri, 7 Sep 2007 22:11:57 +0200

Has somebody seen this with 2.2.x? If not, I will close this bug 
report.

TIA,
Stefan

Message #31 received at 342141@bugs.debian.org (full text, mbox, reply):

From: Josip Rodin <joy@debbugs.entuzijast.net>

To: Stefan Fritsch <sf@sfritsch.de>

Cc: 342141-submitter@bugs.debian.org, 342141@bugs.debian.org

Subject: Re: apache2: Apache2 weird forks (owned by root)

Date: Sat, 8 Sep 2007 10:35:03 +0200

On Fri, Sep 07, 2007 at 10:11:57PM +0200, Stefan Fritsch wrote:
> Has somebody seen this with 2.2.x? If not, I will close this bug 
> report.

I'm hoping to upgrade the machine experiencing it to etch soon, and then
I can say for sure.

We actually experienced the problem again this morning (with 2.0).

-- 
     2. That which causes joy or happiness.

Message #39 received at 342141@bugs.debian.org (full text, mbox, reply):

From: Stefan Fritsch <sf@sfritsch.de>

To: Josip Rodin <joy@debbugs.entuzijast.net>

Cc: 342141-submitter@bugs.debian.org, 342141@bugs.debian.org

Subject: Re: apache2: Apache2 weird forks (owned by root)

Date: Sat, 8 Sep 2007 11:07:57 +0200

On Saturday 08 September 2007, Josip Rodin wrote:
> On Fri, Sep 07, 2007 at 10:11:57PM +0200, Stefan Fritsch wrote:
> > Has somebody seen this with 2.2.x? If not, I will close this bug
> > report.
>
> I'm hoping to upgrade the machine experiencing it to etch soon, and
> then I can say for sure.
>
> We actually experienced the problem again this morning (with 2.0).

Possible ways to debug this would be stracing the root owned processes 
to see where they hang, or restarting under strace until the problem 
appears again.

BUT this bug is not going to be fixed in sarge (being oldstable now). 
It is unfortunate that this has not been investigated earlier, but
I think now it makes more sense to spend time on 2.2.x.

Message #47 received at 342141-done@bugs.debian.org (full text, mbox, reply):

From: Stefan Fritsch <sf@sfritsch.de>

To: 342141-done@bugs.debian.org

Subject: #342141 apache2: Apache2 weird forks (owned by root)

Date: Sun, 27 Jul 2008 17:12:45 +0200 (CEST)

Version: 2.2.3-1

Since there is no indication that this bug affects 2.2.x, I am closing it.

Send a report that this bug log contains spam.