Debian Bug report logs - #378261
slapd fails under heavy load due to descriptor limit

version graph

Package: slapd; Maintainer for slapd is Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>; Source for slapd is src:openldap.

Reported by: Chris Adams <cadams@salk.edu>

Date: Fri, 14 Jul 2006 19:03:06 UTC

Severity: important

Tags: confirmed

Found in version openldap2.2/2.2.23-8

Fixed in version 2.3.35-1

Done: Russ Allbery <rra@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#378261; Package slapd. Full text and rfc822 format available.

Acknowledgement sent to Chris Adams <cadams@salk.edu>:
New Bug report received and forwarded. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Chris Adams <cadams@salk.edu>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: slapd fails under heavy load due to descriptor limit
Date: Fri, 14 Jul 2006 11:36:20 -0700
Package: slapd
Version: 2.2.23-8
Severity: important


slapd can fail under heavy load with the following message:

Jul 14 10:12:04 economo slapd[16502]: daemon: 1024 beyond descriptor table size 1024


There are several problems here:
1) slapd uses more file descriptors than it arguably should (#159776)
2) The table size is artificially capped (servers/slapd/daemon.c lines
1026:1030) to FD_SETSIZE (1024 on current systems), well below the 8192
limit set by the ulimit command in /etc/init.d/slapd. 

http://www.openldap.org/faq/data/cache/1126.html suggests that this
can be raised to 8192 using a compile flag. This patch accomplishes
that:

5a6
> CPPFLAGS=-DFD_SETSIZE=8192
32c33
<       cd $(builddir) && CFLAGS="$(CFLAGS)" $(rootdir)/configure \
---
>       cd $(builddir) && CFLAGS="$(CFLAGS)" CPPFLAGS="$(CPPFLAGS)"
>       $(rootdir)/configure \

3) slapd should never accept a connection which it cannot process.  This
causes visible service interruptions (there's a separate bug in
libnss-ldap/libpam-ldap which cause them to ignore defined LDAP
secondaries in this situation) and keeps the backlog stats from
accurately reflecting the number of connections slapd is unable to
handle.

-- System Information:
Debian Release: 3.1
Architecture: i386 (x86_64)
Kernel: Linux 2.6.8-12-amd64-k8-smp
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)

Versions of packages slapd depends on:
ii  coreutils [fileutils] 5.2.1-2            The GNU core utilities
ii  debconf               1.4.30.13          Debian configuration management sy
ii  libc6                 2.3.2.ds1-22sarge3 GNU C Library: Shared libraries an
ii  libdb4.2              4.2.52-18          Berkeley v4.2 Database Libraries [
ii  libiodbc2             3.52.2-3           iODBC Driver Manager
ii  libldap-2.2-7         2.2.23-8           OpenLDAP libraries
ii  libltdl3              1.5.6-6            A system independent dlopen wrappe
ii  libperl5.8            5.8.4-8sarge4      Shared Perl library
ii  libsasl2              2.1.19-1.5sarge1   Authentication abstraction library
ii  libslp1               1.0.11a-2          OpenSLP libraries
ii  libssl0.9.7           0.9.7e-3sarge1     SSL shared libraries
ii  libwrap0              7.6.dbs-8          Wietse Venema's TCP wrappers libra
ii  perl [libmime-base64- 5.8.4-8sarge4      Larry Wall's Practical Extraction 
ii  psmisc                21.5-1             Utilities that use the proc filesy

-- debconf information:
  slapd/password_mismatch:
  slapd/fix_directory: true
  slapd/invalid_config: true
  slapd/upgrade_slapcat_failure:
  slapd/upgrade_slapadd_failure:
  slapd/backend: BDB
  slapd/dump_database: when needed
  slapd/allow_ldap_v2: false
  slapd/no_configuration: false
  slapd/migrate_ldbm_to_bdb: true
  slapd/move_old_database: true
  slapd/suffix_change: false
  slapd/slave_databases_require_updateref:
  slapd/dump_database_destdir: /var/backups/slapd-VERSION
  slapd/autoconf_modules: true
  slapd/purge_database: false



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#378261; Package slapd. Full text and rfc822 format available.

Acknowledgement sent to Quanah Gibson-Mount <quanah@stanford.edu>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. Full text and rfc822 format available.

Message #10 received at 378261@bugs.debian.org (full text, mbox):

From: Quanah Gibson-Mount <quanah@stanford.edu>
To: Chris Adams <cadams@salk.edu>, 378261@bugs.debian.org
Subject: Re: [Pkg-openldap-devel] Bug#378261: slapd fails under heavy load due to descriptor limit
Date: Sun, 16 Jul 2006 18:16:59 -0700

--On Friday, July 14, 2006 11:36 AM -0700 Chris Adams <cadams@salk.edu> 
wrote:

> Package: slapd
> Version: 2.2.23-8
> Severity: important
>
>
> slapd can fail under heavy load with the following message:
>
> Jul 14 10:12:04 economo slapd[16502]: daemon: 1024 beyond descriptor
> table size 1024
>
>
> There are several problems here:
> 1) slapd uses more file descriptors than it arguably should (#159776)
> 2) The table size is artificially capped (servers/slapd/daemon.c lines
> 1026:1030) to FD_SETSIZE (1024 on current systems), well below the 8192
> limit set by the ulimit command in /etc/init.d/slapd.
>
> http://www.openldap.org/faq/data/cache/1126.html suggests that this
> can be raised to 8192 using a compile flag. This patch accomplishes
> that:

If the debian build links against tcp wrappers, that may be the actual 
problem.  This is rather well discussed on the OpenLDAP archives.

--Quanah

--
Quanah Gibson-Mount
Principal Software Developer
ITS/Shared Application Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#378261; Package slapd. Full text and rfc822 format available.

Acknowledgement sent to Chris Adams <cadams@salk.edu>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. Full text and rfc822 format available.

Message #15 received at 378261@bugs.debian.org (full text, mbox):

From: Chris Adams <cadams@salk.edu>
To: Quanah Gibson-Mount <quanah@stanford.edu>, 378261@bugs.debian.org
Subject: Re: [Pkg-openldap-devel] Bug#378261: slapd fails under heavy load due to descriptor limit
Date: Thu, 28 Sep 2006 11:49:07 -0700
[Message part 1 (text/plain, inline)]
On 2006-07-16, at 6:16 PM, Quanah Gibson-Mount wrote:
> If the debian build links against tcp wrappers, that may be the  
> actual problem.  This is rather well discussed on the OpenLDAP  
> archives.

I couldn't find much which seemed relevant in the archives but I've  
confirmed that this is not the problem by rebuilding slapd with -- 
disable-wrappers (verified with ldd) and confirming that the same  
trivial DoS exists:

#!/usr/bin/perl -w

use strict;
use Socket;

my $sockaddr = sockaddr_in(389, inet_aton("ldap"));
my $proto = getprotobyname('tcp');
my @Sockets;

for (my $n = 0; $n < 4096; $n++) {
        socket($Sockets[$n], PF_INET, SOCK_STREAM, $proto) or die 
("Couldn't create socket $n: $!");
        connect($Sockets[$n], $sockaddr) or die("Couldn't connect  
socket #$n: $!");
        print "$n\n";
}

Rebuilding slapd after making the following addition to debian/rules  
successfully raises the limit:
CFLAGS = -Wall -g -D_FILE_OFFSET_BITS=64 -DOPENLDAP_FD_SETSIZE=8192 - 
DFD_SETSIZE=8192

At this point the DoS script no longer works - suggesting that this  
should become part of the default build since the failure mode is  
severe with e.g. pam-ldap.

Chris

[smime.p7s (application/pkcs7-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#378261; Package slapd. Full text and rfc822 format available.

Acknowledgement sent to Quanah Gibson-Mount <quanah@stanford.edu>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. Full text and rfc822 format available.

Message #20 received at 378261@bugs.debian.org (full text, mbox):

From: Quanah Gibson-Mount <quanah@stanford.edu>
To: Chris Adams <cadams@salk.edu>, 378261@bugs.debian.org
Subject: Re: [Pkg-openldap-devel] Bug#378261: slapd fails under heavy load due to descriptor limit
Date: Thu, 28 Sep 2006 13:11:17 -0700

--On Thursday, September 28, 2006 11:49 AM -0700 Chris Adams 
<cadams@salk.edu> wrote:

> Rebuilding slapd after making the following addition to
> debian/rulessuccessfully raises the limit:
> CFLAGS = -Wall -g -D_FILE_OFFSET_BITS=64 -DOPENLDAP_FD_SETSIZE=8192
> DFD_SETSIZE=8192
>
> At this point the DoS script no longer works - suggesting that thisshould
> become part of the default build since the failure mode issevere with
> e.g. pam-ldap.

Hi Chris,

I found that it was unnecessary to build OpenLDAP with the 
OPENLDAP_FD_SETSIZE parameter set at all to get around this issue.  In my 
slapd startup script, I have:

# Bump up file descriptors to 4096
ulimit -n 4096

which is all that is necessary to allow slapd to use more file descriptors. 
With this in place:

Sep 28 13:07:31 ldap-test2 slapd[3473]: fd=4095 DENIED from 171.64.11.148 
(171.64.11.148)
Sep 28 13:07:31 ldap-test2 slapd[3473]: warning: cannot open 
/etc/hosts.allow: Too many open files


is where it fails.

If I change the ulimit to 1024, then it fails at:

Sep 28 13:09:59 ldap-test2 slapd[29388]: warning: cannot open 
/etc/hosts.allow: Too many open files
Sep 28 13:09:59 ldap-test2 slapd[29388]: error: bad option name: 
"171.64.11.148"
Sep 28 13:09:59 ldap-test2 slapd[29388]: fd=1023 DENIED from 171.64.11.148 
(171.64.11.148)


So basically, this is something that can easily be overcome by the user if 
they need to, and doesn't require any particular compile options on the 
servers side.  I don't really see this as any sort of DoS issue, but a user 
configuration issue.  But that's my 2c.

--Quanah


--
Quanah Gibson-Mount
Principal Software Developer
ITS/Shared Application Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#378261; Package slapd. Full text and rfc822 format available.

Acknowledgement sent to Chris Adams <cadams@salk.edu>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. Full text and rfc822 format available.

Message #25 received at 378261@bugs.debian.org (full text, mbox):

From: Chris Adams <cadams@salk.edu>
To: Quanah Gibson-Mount <quanah@stanford.edu>
Cc: 378261@bugs.debian.org
Subject: Re: [Pkg-openldap-devel] Bug#378261: slapd fails under heavy load due to descriptor limit
Date: Thu, 28 Sep 2006 13:46:10 -0700
[Message part 1 (text/plain, inline)]
On 2006-09-28, at 1:11 PM, Quanah Gibson-Mount wrote:
> If I change the ulimit to 1024, then it fails at:
>
> Sep 28 13:09:59 ldap-test2 slapd[29388]: warning: cannot open /etc/ 
> hosts.allow: Too many open files
> Sep 28 13:09:59 ldap-test2 slapd[29388]: error: bad option name:  
> "171.64.11.148"
> Sep 28 13:09:59 ldap-test2 slapd[29388]: fd=1023 DENIED from  
> 171.64.11.148 (171.64.11.148)
>
> So basically, this is something that can easily be overcome by the  
> user if they need to, and doesn't require any particular compile  
> options on the servers side.  I don't really see this as any sort  
> of DoS issue, but a user configuration issue.  But that's my 2c.

That's a different error than I get - which is why I don't think it's  
a tcp wrappers issue. The problem which we see looks like this:

Sep 28 06:30:01 economo slapd[26971]: daemon: 1024 beyond descriptor  
table size 1024

/etc/init.d/slapd has ulimit -n 8192 (at least since January when I  
customized it to deal with #340266); it's also in the dpkg-default  
version as well) but slapd will reliably start rejecting anything  
beyond the first 1023 connections unless it's built with  
OPENLDAP_FD_SETSIZE set to a higher value.

Chris
[smime.p7s (application/pkcs7-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#378261; Package slapd. Full text and rfc822 format available.

Acknowledgement sent to Quanah Gibson-Mount <quanah@stanford.edu>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. Full text and rfc822 format available.

Message #30 received at 378261@bugs.debian.org (full text, mbox):

From: Quanah Gibson-Mount <quanah@stanford.edu>
To: Chris Adams <cadams@salk.edu>
Cc: 378261@bugs.debian.org
Subject: Re: [Pkg-openldap-devel] Bug#378261: slapd fails under heavy load due to descriptor limit
Date: Thu, 28 Sep 2006 14:08:43 -0700

--On Thursday, September 28, 2006 1:46 PM -0700 Chris Adams 
<cadams@salk.edu> wrote:

>
> On 2006-09-28, at 1:11 PM, Quanah Gibson-Mount wrote:
>> If I change the ulimit to 1024, then it fails at:
>>
>> Sep 28 13:09:59 ldap-test2 slapd[29388]: warning: cannot open /etc/
>> hosts.allow: Too many open files
>> Sep 28 13:09:59 ldap-test2 slapd[29388]: error: bad option name:
>> "171.64.11.148"
>> Sep 28 13:09:59 ldap-test2 slapd[29388]: fd=1023 DENIED from
>> 171.64.11.148 (171.64.11.148)
>>
>> So basically, this is something that can easily be overcome by the
>> user if they need to, and doesn't require any particular compile
>> options on the servers side.  I don't really see this as any sort
>> of DoS issue, but a user configuration issue.  But that's my 2c.
>
> That's a different error than I get - which is why I don't think it'sa
> tcp wrappers issue. The problem which we see looks like this:
>
> Sep 28 06:30:01 economo slapd[26971]: daemon: 1024 beyond descriptortable
> size 1024
>
> /etc/init.d/slapd has ulimit -n 8192 (at least since January when
> Icustomized it to deal with #340266); it's also in the
> dpkg-defaultversion as well) but slapd will reliably start rejecting
> anythingbeyond the first 1023 connections unless it's built
> withOPENLDAP_FD_SETSIZE set to a higher value.

Hm, that's odd.  Because it doesn't do that at all for me.  Although I'm 
running on a 64-bit platform, so maybe that's why?

--Quanah

--
Quanah Gibson-Mount
Principal Software Developer
ITS/Shared Application Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#378261; Package slapd. Full text and rfc822 format available.

Acknowledgement sent to Chris Adams <cadams@salk.edu>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. Full text and rfc822 format available.

Message #35 received at 378261@bugs.debian.org (full text, mbox):

From: Chris Adams <cadams@salk.edu>
To: 378261@bugs.debian.org
Subject: #378261 - slapd fails under heavy load due to descriptor limit - Debian Bug report logs
Date: Wed, 2 May 2007 13:06:41 -0700
[Message part 1 (text/plain, inline)]
I've verified that the same fault happens on 2.3.30-5 on a clean Etch  
install. With nfiles set to 8192 the following will occur quite  
easily in testing:

slapd[5574]: daemon: 4096 beyond descriptor table size 4096

When this happens the connection is still accepted so the client will  
wait until its timeout kicks in.

Because slapd is compiled with tcp wrappers this also causes these  
messages:
 slapd[2871]: warning: cannot open /etc/hosts.deny: Too many open files

(Does it still make sense for slapd to use tcp wrappers? It seems  
like that'd be accomplished with less overhead using iptables for  
sites which need it.)
[smime.p7s (application/pkcs7-signature, attachment)]

Tags added: confirmed Request was from Russ Allbery <rra@debian.org> to control@bugs.debian.org. (Sat, 02 Jun 2007 00:24:01 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#378261; Package slapd. Full text and rfc822 format available.

Acknowledgement sent to Howard Chu <hyc@openldap.org>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. Full text and rfc822 format available.

Message #42 received at 378261@bugs.debian.org (full text, mbox):

From: Howard Chu <hyc@openldap.org>
To: 378261@bugs.debian.org
Subject: slapd fails under heavy load due to descriptor limit
Date: Mon, 03 Sep 2007 21:28:31 -0700
> I've verified that the same fault happens on 2.3.30-5 on a clean Etch  
> install. With nfiles set to 8192 the following will occur quite  
> easily in testing:
> 
> slapd[5574]: daemon: 4096 beyond descriptor table size 4096
> 
> When this happens the connection is still accepted so the client will  
> wait until its timeout kicks in.

In fact the accept() completes before that message is logged, and then slapd 
closes the connection immediately after logging this message so the client 
should free up right away. I don't see anything here that will keep the client 
waiting in limbo.

IMO, keeping the connection unacknowledged (by stopping the accepts) would be 
worse because clients would just hang until something else frees up. At least 
this way, when you *do* have an alternate server to contact, you can get 
bounced over to it quickly.
-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP     http://www.openldap.org/project/



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#378261; Package slapd. Full text and rfc822 format available.

Acknowledgement sent to Russ Allbery <rra@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. Full text and rfc822 format available.

Message #47 received at 378261@bugs.debian.org (full text, mbox):

From: Russ Allbery <rra@debian.org>
To: 378261@bugs.debian.org
Subject: Re: Bug#378261: slapd fails under heavy load due to descriptor limit
Date: Thu, 20 Dec 2007 22:32:16 -0800
The problem here is only on 32-bit Linux, where the file descriptor set
size is capped by default at 1024.  It looks like OpenLDAP is already
increasing this to 4096, since I see no sign of that number anywhere in
glibc's default headers and include/ac/fdset.h has:

#if !defined( OPENLDAP_FD_SETSIZE ) && !defined( FD_SETSIZE )
#  define OPENLDAP_FD_SETSIZE 4096
#endif

#ifdef OPENLDAP_FD_SETSIZE
    /* assume installer desires to enlarge fd_set */
#  ifdef HAVE_BITS_TYPES_H
#    include <bits/types.h>
#  endif
#  ifdef __FD_SETSIZE
#    undef __FD_SETSIZE
#    define __FD_SETSIZE OPENLDAP_FD_SETSIZE
#  else
#    define FD_SETSIZE OPENLDAP_FD_SETSIZE
#  endif
#endif

I don't see much drawback to bumping this to 8192, which seems to be what
the thread is requesting.  Did I miss anything here?

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>




Reply sent to Russ Allbery <rra@debian.org>:
You have taken responsibility. Full text and rfc822 format available.

Notification sent to Chris Adams <cadams@salk.edu>:
Bug acknowledged by developer. Full text and rfc822 format available.

Message #52 received at 378261-done@bugs.debian.org (full text, mbox):

From: Russ Allbery <rra@debian.org>
To: 378261-done@bugs.debian.org
Subject: slapd now uses epoll
Date: Tue, 12 Feb 2008 13:51:21 -0800
Version: 2.3.35-1

Per upstream, the FD_SETSIZE limitations are only applicable if built with
select instead of epoll.  We used to do that (and did that for the version
against which this bug was reported) when we were still supporting Linux
2.4, but now that Debian requires Linux 2.6, we dropped that change.
slapd now uses epoll, so should no longer suffer from the FD_SETSIZE
limitation.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>




Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Wed, 12 Mar 2008 07:37:02 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Wed Apr 16 11:39:42 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.