Debian Bug report logs - #673038
slapd: slapcat output truncated every now and then

version graph

Package: slapd; Maintainer for slapd is Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>; Source for slapd is src:openldap.

Reported by: Axel Beckert <beckert@phys.ethz.ch>

Date: Tue, 15 May 2012 16:39:01 UTC

Severity: grave

Tags: fixed-upstream, patch, upstream

Found in versions openldap/2.4.23-7.2, openldap/2.4.23-7

Fixed in version openldap/2.4.31-1+nmu1

Done: Michael Gilbert <mgilbert@debian.org>

Bug is archived. No further changes may be made.

Forwarded to http://www.openldap.org/lists/openldap-technical/201301/msg00195.html

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, beckert@phys.ethz.ch, abe@debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Tue, 15 May 2012 16:39:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Axel Beckert <beckert@phys.ethz.ch>:
New Bug report received and forwarded. Copy sent to beckert@phys.ethz.ch, abe@debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Tue, 15 May 2012 16:39:04 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Axel Beckert <beckert@phys.ethz.ch>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: slapd: slapcat output truncated every now and then
Date: Tue, 15 May 2012 18:33:57 +0200
Package: slapd
Severity: grave
Version: 2.4.23-7.2
Justification: Causes data loss

Every now and then slapcat's output does not contain the whole content
of the LDAP but is truncated at some LDIF entry border (i.e. all printed
LDIF records seem complete).

To reproduce run "while sleep 1; do slapcat | wc; done" on a moderately
busy LDAP server (writes every few seconds to minutes; about 10000
entries) while slapd is running and notice the occasionally occurring
huge change in wc's printed values, e.g.:

# while sleep 1; do slapcat | wc; done
 471698 1015498 15336677
 471698 1015498 15336677
 471698 1015498 15336677
 471698 1015498 15336677
 471698 1015498 15336677
 471698 1015497 15336630
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015497 15336634
 471698 1015498 15336677
 471698 1015497 15336635
 471698 1015497 15336635
 471698 1015497 15336635
 281829  606820 8632165  <--
 471698 1015497 15336635
 471698 1015497 15336635
 471698 1015497 15336635
 308627  664573 9476751  <--
 471698 1015497 15336635
 471698 1015497 15336635
 471698 1015497 15336637
 471698 1015497 15336637
 471698 1015497 15336637
 471698 1015497 15336637
 471698 1015497 15336631
 471698 1015497 15336631
 471698 1015498 15336673
 471698 1015498 15336673
 471698 1015498 15336725
 471698 1015498 15336725
 471698 1015497 15336682
 471698 1015498 15336725
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015497 15336683
 471698 1015498 15336719
 471698 1015497 15336680
 471698 1015497 15336682
 471698 1015497 15336682
 471698 1015497 15336742
 471698 1015497 15336742
 471698 1015497 15336742
 471698 1015497 15336742
 471698 1015497 15336742
 471698 1015498 15336781
 471698 1015498 15336781
 471698 1015498 15336781
 471698 1015498 15336781
 471698 1015498 15336781
 471698 1015498 15336779
 471698 1015498 15336779
 471698 1015497 15336740
 471698 1015497 15336740
 471698 1015497 15336740
 471698 1015497 15336740
 471698 1015496 15336724
 471698 1015496 15336724
 471698 1015496 15336724
 471698 1015496 15336724
 471698 1015496 15336724
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015496 15336727
 471698 1015497 15336763
 471698 1015497 15336763

According to the slapcat man page it should be "always safe to run
slapcat with the slapd-bdb(5) ... backends" even if slapd runs. We do
use a BDB backend.

Using "slapcat -c" instead of just "slapcat" seems only to lower the
error rate a little bit, but it may also be that we just haven't tested
an significant amount of slapcat calls.

Even running "slapcat -d -1" gives no error message and it always exits
with exit code zero, so except checking the output's length there seems
no chance of catching a truncated output.

As slapcat is used to make backups of LDAP database, having an
unreliable slapcat means to have unreliable backups, too. Seems to have
happened with Lenny back then, too:

[...]
-rw-r--r-- 1 root root  1397053 Jul 17  2010 ldif.2010-07-17.gz
-rw-r--r-- 1 root root  1397255 Jul 18  2010 ldif.2010-07-18.gz
-rw-r--r-- 1 root root  1397523 Jul 19  2010 ldif.2010-07-19.gz
-rw-r--r-- 1 root root    89419 Jul 20  2010 ldif.2010-07-20.gz
-rw-r--r-- 1 root root  1398508 Jul 21  2010 ldif.2010-07-21.gz
-rw-r--r-- 1 root root  1397746 Jul 22  2010 ldif.2010-07-22.gz
-rw-r--r-- 1 root root  1398243 Jul 23  2010 ldif.2010-07-23.gz
[...]

Same counts for cases where the slapcat output is used to export the
LDAP content to other database formats like NIS.

-- System Information:
Debian Release: 6.0.5
  APT prefers stable
  APT policy: (990, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.26-2-xen-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages slapd depends on:
ii  adduser                3.112+nmu2        add and remove users and groups
ii  coreutils              8.5-1             GNU core utilities
ii  debconf [debconf-2.0]  1.5.36.1          Debian configuration management sy
ii  libc6                  2.11.3-3          Embedded GNU C Library: Shared lib
ii  libdb4.8               4.8.30-2          Berkeley v4.8 Database Libraries [
ii  libgnutls26            2.8.6-1+squeeze2  the GNU TLS library - runtime libr
ii  libldap-2.4-2          2.4.23-7.2        OpenLDAP libraries
ii  libltdl7               2.2.6b-2          A system independent dlopen wrappe
ii  libperl5.10            5.10.1-17squeeze3 shared Perl library
ii  libsasl2-2             2.1.23.dfsg1-7    Cyrus SASL - authentication abstra
ii  libslp1                1.2.1-7.8         OpenSLP libraries
ii  libwrap0               7.6.q-19          Wietse Venema's TCP wrappers libra
ii  lsb-base               3.2-23.2squeeze1  Linux Standard Base 3.2 init scrip
ii  perl [libmime-base64-p 5.10.1-17squeeze3 Larry Wall's Practical Extraction 
ii  psmisc                 22.11-1           utilities that use the proc file s
ii  unixodbc               2.2.14p2-1        ODBC tools libraries

Versions of packages slapd recommends:
pn  libsasl2-modules              <none>     (no description available)

Versions of packages slapd suggests:
ii  ldap-utils                    2.4.23-7.2 OpenLDAP utilities

-- Configuration Files:
/etc/default/slapd changed:
SLAPD_USER="openldap"
SLAPD_GROUP="openldap"
SLAPD_PIDFILE=
SLURPD_START=auto
SLAPD_SERVICES="ldap:// ldap://<someip>:<someport>/ ldaps:///"
SLAPD_OPTIONS=""
SLURPD_OPTIONS=""


-- debconf information:
  slapd/allow_ldap_v2: false
  slapd/password_mismatch:
  slapd/tlsciphersuite:
  slapd/fix_directory: true
  slapd/invalid_config: true
  shared/organization: ethz.ch
  slapd/upgrade_slapcat_failure:
  slapd/slurpd_obsolete:
  slapd/no_configuration: false
  slapd/migrate_ldbm_to_bdb: false
  slapd/move_old_database: true
  slapd/upgrade_slapadd_failure:
  slapd/suffix_change: false
  slapd/slave_databases_require_updateref:
  slapd/dump_database_destdir: /var/backups/slapd-VERSION
  slapd/autoconf_modules: true
  slapd/purge_database: false
  slapd/domain: ethz.ch
  slapd/backend: BDB
  slapd/dump_database: when needed




Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Sun, 17 Jun 2012 02:33:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steve Langasek <vorlon@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Sun, 17 Jun 2012 02:33:03 GMT) Full text and rfc822 format available.

Message #10 received at 673038@bugs.debian.org (full text, mbox):

From: Steve Langasek <vorlon@debian.org>
To: Axel Beckert <beckert@phys.ethz.ch>, 673038@bugs.debian.org
Subject: Re: [Pkg-openldap-devel] Bug#673038: slapd: slapcat output truncated every now and then
Date: Sat, 16 Jun 2012 19:30:41 -0700
[Message part 1 (text/plain, inline)]
Hi Axel,

On Tue, May 15, 2012 at 06:33:57PM +0200, Axel Beckert wrote:
> Every now and then slapcat's output does not contain the whole content
> of the LDAP but is truncated at some LDIF entry border (i.e. all printed
> LDIF records seem complete).

> To reproduce run "while sleep 1; do slapcat | wc; done" on a moderately
> busy LDAP server (writes every few seconds to minutes; about 10000
> entries) while slapd is running and notice the occasionally occurring
> huge change in wc's printed values,

> According to the slapcat man page it should be "always safe to run
> slapcat with the slapd-bdb(5) ... backends" even if slapd runs. We do
> use a BDB backend.

Note that the HDB backend is the one recommended upstream and the Debian
default.  Is this issue reproducible with HDB, or only with BDB?

Can you reproduce this issue with the slapd 2.4.28 package in wheezy?

Are there any noteworthy commonalities in the entries that are being
dropped?  Or perhaps commonalities in the last entries that are being
printed?  (Assumes that there is some consistent ordering to the output,
which of course may not be the case)

Would you be able to provide a minimal test configuration (/etc/ldap/slapd.d
and LDIF of a directory, plus BDB DB_CONFIG file) that could be used to
reproduce this?

Thanks,
-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@ubuntu.com                                     vorlon@debian.org
[signature.asc (application/pgp-signature, inline)]

Added tag(s) moreinfo. Request was from Steve Langasek <vorlon@debian.org> to control@bugs.debian.org. (Sun, 17 Jun 2012 05:36:07 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Tue, 19 Jun 2012 12:36:11 GMT) Full text and rfc822 format available.

Acknowledgement sent to Axel Beckert <abe@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Tue, 19 Jun 2012 12:36:15 GMT) Full text and rfc822 format available.

Message #17 received at 673038@bugs.debian.org (full text, mbox):

From: Axel Beckert <abe@debian.org>
To: Steve Langasek <vorlon@debian.org>
Cc: 673038@bugs.debian.org
Subject: Re: [Pkg-openldap-devel] Bug#673038: slapd: slapcat output truncated every now and then
Date: Tue, 19 Jun 2012 14:25:41 +0200
Hi Steve,

Steve Langasek wrote:
> > According to the slapcat man page it should be "always safe to run
> > slapcat with the slapd-bdb(5) ... backends" even if slapd runs. We do
> > use a BDB backend.
> 
> Note that the HDB backend is the one recommended upstream and the Debian
> default.

Well, yeah, that system has been dist-upgraded from at least Etch.
IIRC it started at some time when BDB was still the default.

I wrote that -- according to our backups -- this happened already with
Lenny's slapd. But with Lenny it seemed to have happened less often
(which is why we noticed it only recently).

OTOH we have far more objects in the LDAP for a while now, so it's not
clear if that increase was caused by the dist-upgrade or by going from
approximately 5000 objects to approximately 10000 objects.

> Is this issue reproducible with HDB, or only with BDB?

We will check that in a VM.

> Can you reproduce this issue with the slapd 2.4.28 package in wheezy?

We will check that, too.

> Are there any noteworthy commonalities in the entries that are being
> dropped?

It looks as if always all entries created after some time are dropped,
i.e. more recently created user were affected more often than old
accounts.

> Or perhaps commonalities in the last entries that are being
> printed?

We noticed no commonalities besides the fact that recently created
users are affected more often (the slapcat output seems to be order by
creation time). The place where it stops to output stuff seems to be
random.

> (Assumes that there is some consistent ordering to the output,
> which of course may not be the case)

From what I have in my mind it looked quite consistently ordered.

> Would you be able to provide a minimal test configuration (/etc/ldap/slapd.d
> and LDIF of a directory, plus BDB DB_CONFIG file) that could be used to
> reproduce this?

Unfortunately not (yet) because so far the only place where we
encountered this were our live user database. But we will try to
generate some similar artifical data and check if we can reproduce
this issue with that, too.

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5




Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Tue, 19 Jun 2012 16:39:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Quanah Gibson-Mount <quanah@zimbra.com>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Tue, 19 Jun 2012 16:39:03 GMT) Full text and rfc822 format available.

Message #22 received at 673038@bugs.debian.org (full text, mbox):

From: Quanah Gibson-Mount <quanah@zimbra.com>
To: Axel Beckert <abe@debian.org>, 673038@bugs.debian.org, Steve Langasek <vorlon@debian.org>
Subject: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Tue, 19 Jun 2012 09:27:03 -0700
--On Tuesday, June 19, 2012 2:25 PM +0200 Axel Beckert <abe@debian.org> 
wrote:

> Hi Steve,
>
> Steve Langasek wrote:
>> > According to the slapcat man page it should be "always safe to run
>> > slapcat with the slapd-bdb(5) ... backends" even if slapd runs. We do
>> > use a BDB backend.
>>
>> Note that the HDB backend is the one recommended upstream and the Debian
>> default.
>
> Well, yeah, that system has been dist-upgraded from at least Etch.
> IIRC it started at some time when BDB was still the default.
>
> I wrote that -- according to our backups -- this happened already with
> Lenny's slapd. But with Lenny it seemed to have happened less often
> (which is why we noticed it only recently).

Personally, I would advise you to ask a question about this on 
openldap-technical@openldap.org.  I asked Howard about it, and he had a 
ready answer as to why you were seeing this, but I forget what it is.  In 
any case, this is not a debian specific openldap bug.

--Quanah


--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration




Marked as found in versions openldap/2.4.23-7. Request was from Ivo De Decker <ivo.dedecker@ugent.be> to control@bugs.debian.org. (Sat, 15 Dec 2012 16:45:07 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Sat, 19 Jan 2013 14:15:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Balint Reczey <balint@balintreczey.hu>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Sat, 19 Jan 2013 14:15:05 GMT) Full text and rfc822 format available.

Message #29 received at 673038@bugs.debian.org (full text, mbox):

From: Balint Reczey <balint@balintreczey.hu>
To: Quanah Gibson-Mount <quanah@zimbra.com>
Cc: Axel Beckert <abe@debian.org>, 673038@bugs.debian.org, Steve Langasek <vorlon@debian.org>, control <control@bugs.debian.org>
Subject: Re: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Sat, 19 Jan 2013 15:14:15 +0100
[Message part 1 (text/plain, inline)]
forwarded 673038 technical@openldap.org
thanks

Hi,

I have forwarded the problem to technical@openldap.org but it has not
yet appeared in the list archive.

It worth noting that there exists a script [1] in ldap-git-backup which
can be used for backing up LDAP databases more reliably.

Cheers,
Balint

[1]
https://github.com/elmar/ldap-git-backup/blob/master/README.mdown#safe-ldif

On 06/19/2012 06:27 PM, Quanah Gibson-Mount wrote:
> --On Tuesday, June 19, 2012 2:25 PM +0200 Axel Beckert
> <abe@debian.org> wrote:
>
>> Hi Steve,
>>
>> Steve Langasek wrote:
>>> > According to the slapcat man page it should be "always safe to run
>>> > slapcat with the slapd-bdb(5) ... backends" even if slapd runs. We do
>>> > use a BDB backend.
>>>
>>> Note that the HDB backend is the one recommended upstream and the
>>> Debian
>>> default.
>>
>> Well, yeah, that system has been dist-upgraded from at least Etch.
>> IIRC it started at some time when BDB was still the default.
>>
>> I wrote that -- according to our backups -- this happened already with
>> Lenny's slapd. But with Lenny it seemed to have happened less often
>> (which is why we noticed it only recently).
>
> Personally, I would advise you to ask a question about this on
> openldap-technical@openldap.org.  I asked Howard about it, and he had
> a ready answer as to why you were seeing this, but I forget what it
> is.  In any case, this is not a debian specific openldap bug.
>


[signature.asc (application/pgp-signature, attachment)]

Set Bug forwarded-to-address to 'technical@openldap.org'. Request was from Balint Reczey <balint@balintreczey.hu> to control@bugs.debian.org. (Sat, 19 Jan 2013 14:15:09 GMT) Full text and rfc822 format available.

Changed Bug forwarded-to-address to 'http://www.openldap.org/lists/openldap-technical/201301/msg00195.html' from 'technical@openldap.org' Request was from Balint Reczey <balint@balintreczey.hu> to control@bugs.debian.org. (Sun, 20 Jan 2013 12:18:03 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Fri, 25 Jan 2013 23:12:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to balint@balintreczey.hu:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Fri, 25 Jan 2013 23:12:03 GMT) Full text and rfc822 format available.

Message #38 received at 673038@bugs.debian.org (full text, mbox):

From: Bálint Réczey <balint@balintreczey.hu>
To: Quanah Gibson-Mount <quanah@zimbra.com>
Cc: Axel Beckert <abe@debian.org>, 673038@bugs.debian.org, Steve Langasek <vorlon@debian.org>, control <control@bugs.debian.org>
Subject: Re: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Sat, 26 Jan 2013 00:08:39 +0100
tags 673038 patch upstream - moreinfo
thanks

Hi,

Upstream seems to know about the problem and I provided a fix for them
with a documentation update.
Slapcat's exit code is 1 in case of missing entries thus an
unsuccessful backup attempt can be detected.
Since there is also a helper script for retrying mentioned in me
earlier email I suggest decreasing the severity of this bug removing
it from the RC list.

Cheers,
Balint

PS: Discussion with upstream starts here:
http://www.openldap.org/lists/openldap-technical/201301/msg00195.html

2013/1/19 Balint Reczey <balint@balintreczey.hu>
>
> forwarded 673038 technical@openldap.org
> thanks
>
> Hi,
>
> I have forwarded the problem to technical@openldap.org but it has not
> yet appeared in the list archive.
>
> It worth noting that there exists a script [1] in ldap-git-backup which
> can be used for backing up LDAP databases more reliably.
>
> Cheers,
> Balint
>
> [1]
> https://github.com/elmar/ldap-git-backup/blob/master/README.mdown#safe-ldif
>
> On 06/19/2012 06:27 PM, Quanah Gibson-Mount wrote:
> > --On Tuesday, June 19, 2012 2:25 PM +0200 Axel Beckert
> > <abe@debian.org> wrote:
> >
> >> Hi Steve,
> >>
> >> Steve Langasek wrote:
> >>> > According to the slapcat man page it should be "always safe to run
> >>> > slapcat with the slapd-bdb(5) ... backends" even if slapd runs. We do
> >>> > use a BDB backend.
> >>>
> >>> Note that the HDB backend is the one recommended upstream and the
> >>> Debian
> >>> default.
> >>
> >> Well, yeah, that system has been dist-upgraded from at least Etch.
> >> IIRC it started at some time when BDB was still the default.
> >>
> >> I wrote that -- according to our backups -- this happened already with
> >> Lenny's slapd. But with Lenny it seemed to have happened less often
> >> (which is why we noticed it only recently).
> >
> > Personally, I would advise you to ask a question about this on
> > openldap-technical@openldap.org.  I asked Howard about it, and he had
> > a ready answer as to why you were seeing this, but I forget what it
> > is.  In any case, this is not a debian specific openldap bug.
> >
>
>



Added tag(s) upstream and patch. Request was from Bálint Réczey <balint@balintreczey.hu> to control@bugs.debian.org. (Fri, 25 Jan 2013 23:12:05 GMT) Full text and rfc822 format available.

Removed tag(s) moreinfo. Request was from Bálint Réczey <balint@balintreczey.hu> to control@bugs.debian.org. (Fri, 25 Jan 2013 23:12:05 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Sat, 26 Jan 2013 15:09:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Sat, 26 Jan 2013 15:09:07 GMT) Full text and rfc822 format available.

Message #47 received at 673038@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: balint@balintreczey.hu
Cc: Quanah Gibson-Mount <quanah@zimbra.com>, Axel Beckert <abe@debian.org>, 673038@bugs.debian.org, Steve Langasek <vorlon@debian.org>, control <control@bugs.debian.org>
Subject: Re: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Sat, 26 Jan 2013 15:07:44 +0000
[Message part 1 (text/plain, inline)]
On Sat, 2013-01-26 at 00:08 +0100, Bálint Réczey wrote:
> tags 673038 patch upstream - moreinfo
> thanks
> 
> Hi,
> 
> Upstream seems to know about the problem and I provided a fix for them
> with a documentation update.
> Slapcat's exit code is 1 in case of missing entries thus an
> unsuccessful backup attempt can be detected.
[...]

Really, that's not what it says here:
https://github.com/elmar/ldap-git-backup/blob/master/README.mdown#safe-ldif

Is there an upstream bug fix that makes the exit code non-zero?

Ben.

-- 
Ben Hutchings
Any smoothly functioning technology is indistinguishable from a rigged demo.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Sat, 26 Jan 2013 22:48:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to balint@balintreczey.hu:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Sat, 26 Jan 2013 22:48:03 GMT) Full text and rfc822 format available.

Message #52 received at 673038@bugs.debian.org (full text, mbox):

From: Bálint Réczey <balint@balintreczey.hu>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: Quanah Gibson-Mount <quanah@zimbra.com>, Axel Beckert <abe@debian.org>, 673038@bugs.debian.org, Steve Langasek <vorlon@debian.org>, control <control@bugs.debian.org>
Subject: Re: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Sat, 26 Jan 2013 23:45:17 +0100
Hi Ben,

2013/1/26 Ben Hutchings <ben@decadent.org.uk>:
> On Sat, 2013-01-26 at 00:08 +0100, Bálint Réczey wrote:
>> tags 673038 patch upstream - moreinfo
>> thanks
>>
>> Hi,
>>
>> Upstream seems to know about the problem and I provided a fix for them
>> with a documentation update.
>> Slapcat's exit code is 1 in case of missing entries thus an
>> unsuccessful backup attempt can be detected.
> [...]
>
> Really, that's not what it says here:
> https://github.com/elmar/ldap-git-backup/blob/master/README.mdown#safe-ldif
>
> Is there an upstream bug fix that makes the exit code non-zero?
Looking at slapcat.c the intention is returning 1 on errors while not
all error cases seem to be covered:
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=servers/slapd/slapcat.c;h=6bd9293fe4e980dcd1d7d4a1ccd1a644062fb346;hb=HEAD

IMO the documentation should reflect the intention and the code should
be fixed if it does not do what the documentation says.

Cheers,
Balint



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Sat, 26 Jan 2013 23:33:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Sat, 26 Jan 2013 23:33:03 GMT) Full text and rfc822 format available.

Message #57 received at 673038@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: balint@balintreczey.hu
Cc: Quanah Gibson-Mount <quanah@zimbra.com>, Axel Beckert <abe@debian.org>, 673038@bugs.debian.org, Steve Langasek <vorlon@debian.org>
Subject: Re: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Sat, 26 Jan 2013 23:28:04 +0000
[Message part 1 (text/plain, inline)]
On Sat, 2013-01-26 at 23:45 +0100, Bálint Réczey wrote:
> Hi Ben,
> 
> 2013/1/26 Ben Hutchings <ben@decadent.org.uk>:
> > On Sat, 2013-01-26 at 00:08 +0100, Bálint Réczey wrote:
> >> tags 673038 patch upstream - moreinfo
> >> thanks
> >>
> >> Hi,
> >>
> >> Upstream seems to know about the problem and I provided a fix for them
> >> with a documentation update.
> >> Slapcat's exit code is 1 in case of missing entries thus an
> >> unsuccessful backup attempt can be detected.
> > [...]
> >
> > Really, that's not what it says here:
> > https://github.com/elmar/ldap-git-backup/blob/master/README.mdown#safe-ldif
> >
> > Is there an upstream bug fix that makes the exit code non-zero?
> Looking at slapcat.c the intention is returning 1 on errors while not
> all error cases seem to be covered:
> http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=servers/slapd/slapcat.c;h=6bd9293fe4e980dcd1d7d4a1ccd1a644062fb346;hb=HEAD
> 
> IMO the documentation should reflect the intention and the code should
> be fixed if it does not do what the documentation says.

I think we're all in agreement that the code should be fixed.  Please
help to do that, if you can.

The manual page in unstable doesn't say anything either way though the
'principle of least surprise' implies that slapcat should behave like
most Unix commands do: use stderr and non-zero exit code to report
failure.

Ben.

-- 
Ben Hutchings
Any smoothly functioning technology is indistinguishable from a rigged demo.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Sun, 27 Jan 2013 23:24:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to balint@balintreczey.hu:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Sun, 27 Jan 2013 23:24:03 GMT) Full text and rfc822 format available.

Message #62 received at 673038@bugs.debian.org (full text, mbox):

From: Bálint Réczey <balint@balintreczey.hu>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: Quanah Gibson-Mount <quanah@zimbra.com>, Axel Beckert <abe@debian.org>, 673038@bugs.debian.org, Steve Langasek <vorlon@debian.org>
Subject: Re: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Mon, 28 Jan 2013 00:20:22 +0100
tags 673038 - patch
thanks


2013/1/27 Ben Hutchings <ben@decadent.org.uk>:
> On Sat, 2013-01-26 at 23:45 +0100, Bálint Réczey wrote:
>> Hi Ben,
>>
>> 2013/1/26 Ben Hutchings <ben@decadent.org.uk>:
>> > On Sat, 2013-01-26 at 00:08 +0100, Bálint Réczey wrote:
>> >> tags 673038 patch upstream - moreinfo
>> >> thanks
>> >>
>> >> Hi,
>> >>
>> >> Upstream seems to know about the problem and I provided a fix for them
>> >> with a documentation update.
>> >> Slapcat's exit code is 1 in case of missing entries thus an
>> >> unsuccessful backup attempt can be detected.
>> > [...]
>> >
>> > Really, that's not what it says here:
>> > https://github.com/elmar/ldap-git-backup/blob/master/README.mdown#safe-ldif
>> >
>> > Is there an upstream bug fix that makes the exit code non-zero?
>> Looking at slapcat.c the intention is returning 1 on errors while not
>> all error cases seem to be covered:
>> http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=servers/slapd/slapcat.c;h=6bd9293fe4e980dcd1d7d4a1ccd1a644062fb346;hb=HEAD
>>
>> IMO the documentation should reflect the intention and the code should
>> be fixed if it does not do what the documentation says.
>
> I think we're all in agreement that the code should be fixed.  Please
> help to do that, if you can.
Upstream has rejected the proposed fix.
Since it seems I'm not familiar enough with upstream's plans and
coding practices I'm not the best person to provide a fix.

Upstream BTW don't see this issue as an important one and recommends
using the mdb back-end which is expected to be much faster in most
cases and also does not exhibit this problem.

Regarding the Debian Project IMO the best option would be living with
the bug, relying on the workaround for backups and releasing Wheezy
with this bug present.

Cheers,
Balint



Removed tag(s) patch. Request was from Bálint Réczey <balint@balintreczey.hu> to control@bugs.debian.org. (Sun, 27 Jan 2013 23:24:05 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Thu, 07 Feb 2013 18:51:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to balint@balintreczey.hu:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Thu, 07 Feb 2013 18:51:06 GMT) Full text and rfc822 format available.

Message #69 received at 673038@bugs.debian.org (full text, mbox):

From: Bálint Réczey <balint@balintreczey.hu>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: Quanah Gibson-Mount <quanah@zimbra.com>, Axel Beckert <abe@debian.org>, 673038@bugs.debian.org, Steve Langasek <vorlon@debian.org>, control <control@bugs.debian.org>
Subject: Re: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Thu, 7 Feb 2013 19:45:54 +0100
tags 673038 + patch fixed-upstream
thanks

Hi All,

2013/1/28 Bálint Réczey <balint@balintreczey.hu>:
...
>>
>> I think we're all in agreement that the code should be fixed.  Please
>> help to do that, if you can.
> Upstream has rejected the proposed fix.
> Since it seems I'm not familiar enough with upstream's plans and
> coding practices I'm not the best person to provide a fix.
Upstream (Howard Chu, thanks!) has committed and alternate fix [1] [2].
Please consider back-porting it to Debian instead of using my patch.

Cheers,
Balint

[1]: http://www.openldap.org/lists/openldap-bugs/201302/msg00016.html
[2]: http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blobdiff;f=servers/slapd/back-bdb/init.c;h=57690fc532a3c3c803ceeb1b404ad5952b5df734;hp=1d8527332a956d4c4122ae94a69f07ba9f4da9a1;hb=HEAD;hpb=1c99711ede25860b24750aab9c529a49c53756ae



Added tag(s) fixed-upstream and patch. Request was from Bálint Réczey <balint@balintreczey.hu> to control@bugs.debian.org. (Thu, 07 Feb 2013 18:51:08 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Thu, 07 Feb 2013 19:00:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Quanah Gibson-Mount <quanah@zimbra.com>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Thu, 07 Feb 2013 19:00:03 GMT) Full text and rfc822 format available.

Message #76 received at 673038@bugs.debian.org (full text, mbox):

From: Quanah Gibson-Mount <quanah@zimbra.com>
To: balint@balintreczey.hu, Ben Hutchings <ben@decadent.org.uk>
Cc: Axel Beckert <abe@debian.org>, 673038@bugs.debian.org, Steve Langasek <vorlon@debian.org>
Subject: Re: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Thu, 07 Feb 2013 10:48:28 -0800
--On Thursday, February 07, 2013 7:45 PM +0100 Bálint Réczey 
<balint@balintreczey.hu> wrote:

> tags 673038 + patch fixed-upstream
> thanks
>
> Hi All,
>
> 2013/1/28 Bálint Réczey <balint@balintreczey.hu>:
> ...
>>>
>>> I think we're all in agreement that the code should be fixed.  Please
>>> help to do that, if you can.
>> Upstream has rejected the proposed fix.
>> Since it seems I'm not familiar enough with upstream's plans and
>> coding practices I'm not the best person to provide a fix.
> Upstream (Howard Chu, thanks!) has committed and alternate fix [1] [2].
> Please consider back-porting it to Debian instead of using my patch.

As noted in the follow up, this fix needs to be *tested* by someone who is 
affected.  Not just grabbed and applied.

--Quanah

--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Thu, 07 Feb 2013 19:00:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to balint@balintreczey.hu:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Thu, 07 Feb 2013 19:00:05 GMT) Full text and rfc822 format available.

Message #81 received at 673038@bugs.debian.org (full text, mbox):

From: Bálint Réczey <balint@balintreczey.hu>
To: Quanah Gibson-Mount <quanah@zimbra.com>
Cc: Ben Hutchings <ben@decadent.org.uk>, Axel Beckert <abe@debian.org>, 673038@bugs.debian.org, Steve Langasek <vorlon@debian.org>
Subject: Re: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Thu, 7 Feb 2013 19:57:45 +0100
2013/2/7 Quanah Gibson-Mount <quanah@zimbra.com>:
> --On Thursday, February 07, 2013 7:45 PM +0100 Bálint Réczey
> <balint@balintreczey.hu> wrote:
>
>> tags 673038 + patch fixed-upstream
>> thanks
...
>> Upstream (Howard Chu, thanks!) has committed and alternate fix [1] [2].
>> Please consider back-porting it to Debian instead of using my patch.
>
>
> As noted in the follow up, this fix needs to be *tested* by someone who is
> affected.  Not just grabbed and applied.
Usually the considering part includes waiting for testers or testing
the fix. ;-)
I just wanted to note that the proposed patch (for which I set the
patch tag again)
is not the one I submitted.

Cheers,
Balint



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Wed, 13 Feb 2013 17:15:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Giovanni Biscuolo <g@xelera.eu>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Wed, 13 Feb 2013 17:15:03 GMT) Full text and rfc822 format available.

Message #86 received at 673038@bugs.debian.org (full text, mbox):

From: Giovanni Biscuolo <g@xelera.eu>
To: 673038@bugs.debian.org
Subject: please confirm if back-hdb is affected too
Date: Wed, 13 Feb 2013 18:08:42 +0100
[Message part 1 (text/plain, inline)]
Hello,

does the fact that the proposed patch (message #69
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=673038#69)

is just for back-bdb mean that back-hdb is not affected?

Thanks in advance
Giovanni

-- 
Giovanni Biscuolo

Xelera - IT infrastructures
http://xelera.eu/contact-us/
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Wed, 13 Feb 2013 18:09:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Quanah Gibson-Mount <quanah@zimbra.com>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Wed, 13 Feb 2013 18:09:03 GMT) Full text and rfc822 format available.

Message #91 received at 673038@bugs.debian.org (full text, mbox):

From: Quanah Gibson-Mount <quanah@zimbra.com>
To: Giovanni Biscuolo <g@xelera.eu>, 673038@bugs.debian.org
Subject: Re: [Pkg-openldap-devel] Bug#673038: please confirm if back-hdb is affected too
Date: Wed, 13 Feb 2013 10:04:38 -0800
--On Wednesday, February 13, 2013 6:08 PM +0100 Giovanni Biscuolo 
<g@xelera.eu> wrote:

> Hello,
>
> does the fact that the proposed patch (message #69
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=673038#69)
>
> is just for back-bdb mean that back-hdb is not affected?

back-bdb and back-hdb share 99% or more of their code, including the source 
files.  Thus a fix to the "back-bdb" location is generally a fix to both 
backends.

--Quanah


--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Tue, 16 Apr 2013 18:45:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Adam D. Barratt" <adam@adam-barratt.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Tue, 16 Apr 2013 18:45:04 GMT) Full text and rfc822 format available.

Message #96 received at 673038@bugs.debian.org (full text, mbox):

From: "Adam D. Barratt" <adam@adam-barratt.org.uk>
To: balint@balintreczey.hu, 673038@bugs.debian.org
Cc: Quanah Gibson-Mount <quanah@zimbra.com>, Ben Hutchings <ben@decadent.org.uk>, Axel Beckert <abe@debian.org>, Steve Langasek <vorlon@debian.org>
Subject: Re: Bug#673038: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Tue, 16 Apr 2013 19:40:13 +0100
On Thu, 2013-02-07 at 19:57 +0100, Bálint Réczey wrote:
> 2013/2/7 Quanah Gibson-Mount <quanah@zimbra.com>:
> > --On Thursday, February 07, 2013 7:45 PM +0100 Bálint Réczey
> > <balint@balintreczey.hu> wrote:
> >> Upstream (Howard Chu, thanks!) has committed and alternate fix [1] [2].
> >> Please consider back-porting it to Debian instead of using my patch.
> >
> >
> > As noted in the follow up, this fix needs to be *tested* by someone who is
> > affected.  Not just grabbed and applied.
> Usually the considering part includes waiting for testers or testing
> the fix. ;-)

Has anyone had chance to look at this? It's getting quite late for a fix
for wheezy.

Regards,

Adam




Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Tue, 16 Apr 2013 18:45:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steve Langasek <vorlon@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Tue, 16 Apr 2013 18:45:07 GMT) Full text and rfc822 format available.

Message #101 received at 673038@bugs.debian.org (full text, mbox):

From: Steve Langasek <vorlon@debian.org>
To: "Adam D. Barratt" <adam@adam-barratt.org.uk>
Cc: balint@balintreczey.hu, 673038@bugs.debian.org, Quanah Gibson-Mount <quanah@zimbra.com>, Ben Hutchings <ben@decadent.org.uk>, Axel Beckert <abe@debian.org>
Subject: Re: Bug#673038: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Tue, 16 Apr 2013 11:43:31 -0700
[Message part 1 (text/plain, inline)]
On Tue, Apr 16, 2013 at 07:40:13PM +0100, Adam D. Barratt wrote:
> On Thu, 2013-02-07 at 19:57 +0100, Bálint Réczey wrote:
> > 2013/2/7 Quanah Gibson-Mount <quanah@zimbra.com>:
> > > --On Thursday, February 07, 2013 7:45 PM +0100 Bálint Réczey
> > > <balint@balintreczey.hu> wrote:
> > >> Upstream (Howard Chu, thanks!) has committed and alternate fix [1] [2].
> > >> Please consider back-porting it to Debian instead of using my patch.

> > > As noted in the follow up, this fix needs to be *tested* by someone who is
> > > affected.  Not just grabbed and applied.
> > Usually the considering part includes waiting for testers or testing
> > the fix. ;-)

> Has anyone had chance to look at this? It's getting quite late for a fix
> for wheezy.

Not much I can do without a reproducer case for this.  Axel, can you test
the proposed upstream fix?

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@ubuntu.com                                     vorlon@debian.org
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Tue, 16 Apr 2013 18:57:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Axel Beckert <abe@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Tue, 16 Apr 2013 18:57:04 GMT) Full text and rfc822 format available.

Message #106 received at 673038@bugs.debian.org (full text, mbox):

From: Axel Beckert <abe@debian.org>
To: Steve Langasek <vorlon@debian.org>
Cc: "Adam D. Barratt" <adam@adam-barratt.org.uk>, balint@balintreczey.hu, 673038@bugs.debian.org, Quanah Gibson-Mount <quanah@zimbra.com>, Ben Hutchings <ben@decadent.org.uk>
Subject: Re: Bug#673038: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Tue, 16 Apr 2013 20:53:04 +0200
Hi,

Steve Langasek wrote:
> On Tue, Apr 16, 2013 at 07:40:13PM +0100, Adam D. Barratt wrote:
> > On Thu, 2013-02-07 at 19:57 +0100, Bálint Réczey wrote:
> > > 2013/2/7 Quanah Gibson-Mount <quanah@zimbra.com>:
> > > > --On Thursday, February 07, 2013 7:45 PM +0100 Bálint Réczey
> > > > <balint@balintreczey.hu> wrote:
> > > >> Upstream (Howard Chu, thanks!) has committed and alternate fix [1] [2].
> > > >> Please consider back-porting it to Debian instead of using my patch.
> 
> > > > As noted in the follow up, this fix needs to be *tested* by someone who is
> > > > affected.  Not just grabbed and applied.
> > > Usually the considering part includes waiting for testers or testing
> > > the fix. ;-)
> 
> > Has anyone had chance to look at this? It's getting quite late for a fix
> > for wheezy.
> 
> Not much I can do without a reproducer case for this.  Axel, can you test
> the proposed upstream fix?

I will try, but I can't promise that I'm able to reproduce the issue
with just artificial load.

Currently trying to figure out what I need to reproduce the issue.

Can't test it on the production server where we found the issue,
sorry. (We're not in urgent need anymore for a fix as we workarounded
it as mentioned earlier by running slapcat as often as necessary to
get two identical results.)

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Tue, 16 Apr 2013 21:36:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Axel Beckert <abe@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Tue, 16 Apr 2013 21:36:04 GMT) Full text and rfc822 format available.

Message #111 received at 673038@bugs.debian.org (full text, mbox):

From: Axel Beckert <abe@debian.org>
To: Steve Langasek <vorlon@debian.org>
Cc: "Adam D. Barratt" <adam@adam-barratt.org.uk>, balint@balintreczey.hu, 673038@bugs.debian.org, Quanah Gibson-Mount <quanah@zimbra.com>, Ben Hutchings <ben@decadent.org.uk>
Subject: Re: Bug#673038: Re: [Pkg-openldap-devel] Bug#673038: Bug#673038: slapd: slapcat output truncated every now and then
Date: Tue, 16 Apr 2013 23:32:18 +0200
Hi,

Axel Beckert wrote:
> > > > > --On Thursday, February 07, 2013 7:45 PM +0100 Bálint Réczey
> > > > > <balint@balintreczey.hu> wrote:
> > > > >> Upstream (Howard Chu, thanks!) has committed and alternate fix [1] [2].
> > > > >> Please consider back-porting it to Debian instead of using my patch.

This is from
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=673038#69 -- I used
the patch mentioned in there via quilt and build the package in a
pbuilder Sid chroot:

$ debdiff openldap_2.4.31-1.dsc openldap_2.4.31-2~abetest1.dsc
diff -u openldap-2.4.31/debian/changelog openldap-2.4.31/debian/changelog
--- openldap-2.4.31/debian/changelog
+++ openldap-2.4.31/debian/changelog
@@ -1,3 +1,9 @@
+openldap (2.4.31-2~abetest1) unstable; urgency=low
+
+  * Testing fix for #673038
+
+ -- Axel Beckert <abe@debian.org>  Tue, 16 Apr 2013 21:25:13 +0200
+
 openldap (2.4.31-1) unstable; urgency=low
 
   * New upstream release.
diff -u openldap-2.4.31/debian/patches/series openldap-2.4.31/debian/patches/series
--- openldap-2.4.31/debian/patches/series
+++ openldap-2.4.31/debian/patches/series
@@ -21,0 +22 @@
+slapcat-673038
only in patch2:
unchanged:
--- openldap-2.4.31.orig/debian/patches/slapcat-673038
+++ openldap-2.4.31/debian/patches/slapcat-673038
@@ -0,0 +1,21 @@
+X-Git-Url: http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blobdiff_plain;f=servers%2Fslapd%2Fback-bdb%2Finit.c;h=57690fc532a3c3c803ceeb1b404ad5952b5df734;hp=1d8527332a956d4c4122ae94a69f07ba9f4da9a1;hb=HEAD;hpb=1c99711ede25860b24750aab9c529a49c53756ae
+
+diff --git a/servers/slapd/back-bdb/init.c b/servers/slapd/back-bdb/init.c
+index 1d85273..57690fc 100644
+--- a/servers/slapd/back-bdb/init.c
++++ b/servers/slapd/back-bdb/init.c
+@@ -529,7 +529,13 @@ shm_retry:
+       }
+ 
+       if ( !quick ) {
+-              TXN_BEGIN(bdb->bi_dbenv, NULL, &bdb->bi_cache.c_txn, DB_READ_COMMITTED | DB_TXN_NOWAIT);
++              int txflag = DB_READ_COMMITTED;
++              /* avoid deadlocks in server; tools should
++               * wait since they have no deadlock retry mechanism.
++               */
++              if ( slapMode & SLAP_SERVER_MODE )
++                      txflag |= DB_TXN_NOWAIT;
++              TXN_BEGIN(bdb->bi_dbenv, NULL, &bdb->bi_cache.c_txn, txflag);
+       }
+ 
+       entry_prealloc( bdb->bi_cache.c_maxsize );

> I will try, but I can't promise that I'm able to reproduce the issue
> with just artificial load.

That was surprisingly easy. Additionally needed packages for my test
setup: ldapvi and makepasswd. I installed slapd from Wheezy, set a
password and added one additional entry (see below) using ldapvi.

Due to http://bugs.debian.org/550843 I had to dump the following
"editor" into a script:

---begin ldapvi-editor.sh---
#!/bin/sh
sed -e 's/userPassword: .*/userPassword: '`makepasswd`'/' -i "$2"
---end ldapvi-editor.sh---

The initial slapd content looked like this:

---begin slapcat output---
dn: dc=ethz,dc=ch
objectClass: top
objectClass: dcObject
objectClass: organization
o: ethz.ch
dc: ethz
structuralObjectClass: organization
entryUUID: c1f21214-3b12-1032-9c28-b763fe97c760
creatorsName: cn=admin,dc=ethz,dc=ch
createTimestamp: 20130416185405Z
entryCSN: 20130416185405.984971Z#000000#000#000000
modifiersName: cn=admin,dc=ethz,dc=ch
modifyTimestamp: 20130416185405Z

dn: cn=admin,dc=ethz,dc=ch
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: admin
description: LDAP administrator
userPassword:: <admin password encoded>
structuralObjectClass: organizationalRole
entryUUID: c1fe26bc-3b12-1032-9c29-b763fe97c760
creatorsName: cn=admin,dc=ethz,dc=ch
createTimestamp: 20130416185406Z
entryCSN: 20130416185406.064157Z#000000#000#000000
modifiersName: cn=admin,dc=ethz,dc=ch
modifyTimestamp: 20130416185406Z

dn: cn=test,dc=ethz,dc=ch
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: test
description: #673038 test object
structuralObjectClass: organizationalRole
entryUUID: 8b242f46-3b13-1032-9176-ad0e06c74528
creatorsName: cn=admin,dc=ethz,dc=ch
createTimestamp: 20130416185943Z
userPassword:: akhzcFVvWTNzTg==
entryCSN: 20130416211030.076749Z#000000#000#000000
modifiersName: cn=admin,dc=ethz,dc=ch
modifyTimestamp: 20130416211030Z
---end slapcat output---

Then I started the following onliner in a root shell:

# while :; do slapcat | wc -l; done

It writes out a lot of "42". (Yay! The answer to life, the universe
and everything! ;-)

Then I started on a user shell the following oneliner:

$ while :; do env EDITOR=./ldapvi-editor.sh ldapvi --noquestions -h localhost -D cn=admin,dc=ethz,dc=ch -w '<admin password>' -v -b dc=ethz,dc=ch cn=test; done

This updates the user password of cn=test,dc=ethz,dc=ch on every
cycle, i.e. does more or less constantly write accesses to the LDAP
database.

With the current package version in Wheezy/Sid, occassionally (guessed
average around every 30th time) the printed number of the slapcat loop
wasn't "42" but "0" or, very seldom, also numbers between 0 and 42. (I
e.g. once saw a "14".)

With the package build with the above patch I always had "42" and no
more other numbers.

For me, this confirms that the patch from
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blobdiff;f=servers/slapd/back-bdb/init.c;h=57690fc532a3c3c803ceeb1b404ad5952b5df734;hp=1d8527332a956d4c4122ae94a69f07ba9f4da9a1;hb=HEAD;hpb=1c99711ede25860b24750aab9c529a49c53756ae
indeed solves the issue.

The build time test suite of the package ran through successfully.

There was only one issue I encountered while running this test: When
running my test with the Wheezy version, I first Ctrl-C'ed the slapcat
loop. This made the ldapvi loop to hang (but not abort). I Ctrl-Z'ed
the ldapvi loop and "kill %1"'ed it. After that "service slapd stop"
always failed. I wasn't able to "kill -TERM" slapd, I had to "kill
-KILL" it. After that the package upgrade went fine and slapd
successfully recovered the unclean shutdown.

HTH.

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Wed, 17 Apr 2013 01:00:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Michael Gilbert <mgilbert@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Wed, 17 Apr 2013 01:00:04 GMT) Full text and rfc822 format available.

Message #116 received at 673038@bugs.debian.org (full text, mbox):

From: Michael Gilbert <mgilbert@debian.org>
To: 673038@bugs.debian.org
Subject: re: slapcat output truncated every now and then
Date: Tue, 16 Apr 2013 20:56:48 -0400
> Has anyone had chance to look at this? It's getting quite late for a fix
> for wheezy.

For what its worth, this was tested and confirmed working upstream a
couple days ago:
http://www.mail-archive.com/openldap-its@openldap.org/msg08242.html

Best wishes,
Mike



Information forwarded to debian-bugs-dist@lists.debian.org, Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>:
Bug#673038; Package slapd. (Fri, 19 Apr 2013 00:15:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to Michael Gilbert <mgilbert@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>. (Fri, 19 Apr 2013 00:15:07 GMT) Full text and rfc822 format available.

Message #121 received at 673038@bugs.debian.org (full text, mbox):

From: Michael Gilbert <mgilbert@debian.org>
To: 673038@bugs.debian.org
Subject: Re: slapcat output truncated every now and then
Date: Thu, 18 Apr 2013 20:12:35 -0400
[Message part 1 (text/plain, inline)]
On Tue, Apr 16, 2013 at 8:56 PM, Michael Gilbert wrote:
>> Has anyone had chance to look at this? It's getting quite late for a fix
>> for wheezy.
>
> For what its worth, this was tested and confirmed working upstream a
> couple days ago:
> http://www.mail-archive.com/openldap-its@openldap.org/msg08242.html

I've uploaded an nmu to delayed/2 fixing this.  Please let me know if
I should delay longer.  Patch attached.

Best wishes,
Mike
[openldap.patch (application/octet-stream, attachment)]

Reply sent to Michael Gilbert <mgilbert@debian.org>:
You have taken responsibility. (Sun, 21 Apr 2013 00:36:05 GMT) Full text and rfc822 format available.

Notification sent to Axel Beckert <beckert@phys.ethz.ch>:
Bug acknowledged by developer. (Sun, 21 Apr 2013 00:36:05 GMT) Full text and rfc822 format available.

Message #126 received at 673038-close@bugs.debian.org (full text, mbox):

From: Michael Gilbert <mgilbert@debian.org>
To: 673038-close@bugs.debian.org
Subject: Bug#673038: fixed in openldap 2.4.31-1+nmu1
Date: Sun, 21 Apr 2013 00:32:49 +0000
Source: openldap
Source-Version: 2.4.31-1+nmu1

We believe that the bug you reported is fixed in the latest version of
openldap, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 673038@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Michael Gilbert <mgilbert@debian.org> (supplier of updated openldap package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Tue, 16 Apr 2013 03:35:31 +0000
Source: openldap
Binary: slapd slapd-smbk5pwd ldap-utils libldap-2.4-2 libldap-2.4-2-dbg libldap2-dev slapd-dbg
Architecture: source amd64
Version: 2.4.31-1+nmu1
Distribution: unstable
Urgency: medium
Maintainer: Debian OpenLDAP Maintainers <pkg-openldap-devel@lists.alioth.debian.org>
Changed-By: Michael Gilbert <mgilbert@debian.org>
Description: 
 ldap-utils - OpenLDAP utilities
 libldap-2.4-2 - OpenLDAP libraries
 libldap-2.4-2-dbg - Debugging information for OpenLDAP libraries
 libldap2-dev - OpenLDAP development libraries
 slapd      - OpenLDAP server (slapd)
 slapd-dbg  - Debugging information for the OpenLDAP server (slapd)
 slapd-smbk5pwd - Keeps Samba and Kerberos passwords in sync within slapd.
Closes: 673038
Changes: 
 openldap (2.4.31-1+nmu1) unstable; urgency=medium
 .
   * Non-maintainer upload.
   * Avoid deadlocks in back-bdb that truncate slapcat output (closes: #673038).
Checksums-Sha1: 
 2ef007b480a44b751596aea8ff11c8ec7859f7c8 3441 openldap_2.4.31-1+nmu1.dsc
 c2d2f92570f007604bf6a147dbb30adb42bd5adc 161504 openldap_2.4.31-1+nmu1.diff.gz
 ba0a6c436c3cbe467388d37e9620c02b89927dff 1776646 slapd_2.4.31-1+nmu1_amd64.deb
 80f0e19537e7bf611693558619da354412fb3f06 79274 slapd-smbk5pwd_2.4.31-1+nmu1_amd64.deb
 673ce0f93df7fb782df3da72060c97ee59063a62 344302 ldap-utils_2.4.31-1+nmu1_amd64.deb
 996744efc296299618fea676fb3184be74422eca 242492 libldap-2.4-2_2.4.31-1+nmu1_amd64.deb
 8f9dd74e198ec7575154a8c118696f3437312f77 474500 libldap-2.4-2-dbg_2.4.31-1+nmu1_amd64.deb
 eb5e8622839be9d487ce261808eb16edff40f91c 554994 libldap2-dev_2.4.31-1+nmu1_amd64.deb
 2729a5a77744cf16d9c27c3a6e36b51862a23061 5520892 slapd-dbg_2.4.31-1+nmu1_amd64.deb
Checksums-Sha256: 
 e5aae6f95e59806cd231309f83c8308c7a5f734af429998c0c0212cd2c661d4f 3441 openldap_2.4.31-1+nmu1.dsc
 f2f2a8d21cba6ca15f90f29ddd4d92fe88a497b2b9d1d4980d7aa9b7737cecd6 161504 openldap_2.4.31-1+nmu1.diff.gz
 48de4ad94d33e18584d8c33790ebd7f834c9279d30159e7c5ab7bcb5667d63af 1776646 slapd_2.4.31-1+nmu1_amd64.deb
 33cbca3d83d4313c8808e0814677da501035a15f6be46caa92953abaf078446f 79274 slapd-smbk5pwd_2.4.31-1+nmu1_amd64.deb
 b03b95b402e82d53c455f0203dc5c1ef0ab5974e6b3c9eb16a9319bf3a5985f6 344302 ldap-utils_2.4.31-1+nmu1_amd64.deb
 ada3ffd135b2c0fb42b8563c3b4e380a80e37f58825df1bc091e73eac854035b 242492 libldap-2.4-2_2.4.31-1+nmu1_amd64.deb
 c6535de348de35acb550a882fdc07e0464ded43d8632cd07774d2246792a20e6 474500 libldap-2.4-2-dbg_2.4.31-1+nmu1_amd64.deb
 6b1c710b1ef7af829d9501411c80b23113dd296fa61bd07f92edd31029532b48 554994 libldap2-dev_2.4.31-1+nmu1_amd64.deb
 af23c24c471958bba626ae1aa39f5acf3fb626ccc228282ed5f18fbda8151096 5520892 slapd-dbg_2.4.31-1+nmu1_amd64.deb
Files: 
 df74d45dbc5e202f75007288c4657bbf 3441 net optional openldap_2.4.31-1+nmu1.dsc
 7173b719cbf35a9fe7d7d5733eaaa6c5 161504 net optional openldap_2.4.31-1+nmu1.diff.gz
 cb195bc8aa062dc3022467f4ea3f4def 1776646 net optional slapd_2.4.31-1+nmu1_amd64.deb
 54e027716788f93934ca24d56bbebdcd 79274 net extra slapd-smbk5pwd_2.4.31-1+nmu1_amd64.deb
 4f8fa7f8066493c2db826e7702afbede 344302 net optional ldap-utils_2.4.31-1+nmu1_amd64.deb
 33cf08ccb0f9cc5c9e1924de84c35811 242492 libs standard libldap-2.4-2_2.4.31-1+nmu1_amd64.deb
 5de5b869556194eee26819a75a9bb867 474500 debug extra libldap-2.4-2-dbg_2.4.31-1+nmu1_amd64.deb
 76f9c93c577372359e61ccda7e154556 554994 libdevel extra libldap2-dev_2.4.31-1+nmu1_amd64.deb
 0bf7a0a71166da33025453477d95332c 5520892 debug extra slapd-dbg_2.4.31-1+nmu1_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQQcBAEBCAAGBQJRb1dMAAoJELjWss0C1vRzFJQf/3lzigE1P5mMRDZ8Kk+1yg/p
IX0HCn3NzrgVTjXzytq8lSh4GD3IcN7EcjzTwjac5y9NzfMjpaf3OheTIGxCZRL2
0T1iIPpVre3KeroDWCwLlo3mijMHbbsUc41DGYJqSZlqEsJukxR05CXjVNMlGJuY
1zeyaKfV3b5cYKdthQDgquFc7TdZw34boeWnmP48lYHnoDafF75Ql9Zt+CyNUacr
4CPngSa3GW2Y9ujHsZ7SYkl41ijKXIInMO/1U5gkqEFqgt2o1gn30P+GR1qm91Be
dTN0AYmvd+UUk5Cf9YIysFG6un/S8JevZryEY3Nrt96AS9/gDLTADKblNRCSRx2e
WcSASDYFdHfWvRV+ZB625dJbOjIURNQiXYcV/FFRTV+ZMzJm0r1qaUzQ0JsRnoGV
Ierzgnus4z5/AS5dhxT1WgCcicaieHPdU8+9tZ/bTUb+sxfsgtRBc4pEIryt7wc1
lLFB6PPaEP7uM0wClMgDmy3bRfQtgWHzM5ZQ5aYwaoRDA5GrjsKi6VFiXB823med
QRoWZ03cVB3nKyT9x2AY8uSgCETAXc5T9+jFliLFbw/t+K7DuZKl9jk4LQYj2GsM
0r9uHZvPQs1aJ0VNg/yCJVElCQMb85xDPMTISgZ/q8iLmIIxnaGwjEwUTSydSSXn
IvVFp5kGo3ApgtStoyWaoaCq9rgiDU9DZ993VwMbPpI0K3VGqLczexoWy8fHuqYg
JLPbxomq7YCQGu+1firpgTURW96Vdi25+FoolnFb67hBQMMojmmnGO1q5zoc+Fkk
ZO7ERE8P6sizt6g8FpUyCxVLVFa6fX9c57Xvl0FGAxSIrVaH3iJGqnIpmIFbSnI1
n5O66SE2vs2vZX+pZJ0MMWlK9BbyZuO7vUutj5IHARpzPzn3Zv3R1sLX2Ua2sLSC
xVXTNWbn21zfpnytvQiCwPfzbVTC0XRgE+H8bb2KNnoM6XoMs2SbZtnwLjxwzixx
2HWI2cTvUbO2WUF3i+3w1Sv151SOK//TjmwSa+BFDJBJnmcoRZ9JgHGvIyy4sScb
xiXPAfv8AWO+12wVuOphIhDrY0A/GmSbqPVv+FufMC55iLDZKvd5lY5kUfDjp9qb
9dFro6PfS6YwYI04zrIaglin3sPjD8iKjDAthSw3hm94anHV0pLFaZsHa5YsF3d/
RxIm66gaSR5LywWQWbRGhB+GzyXNHBqeYuYxR3UCt7A8QZotBuvxC7mFTiCrcbfS
vqn7kKT4J6k0zsY0MLfYQvYL817VI+FbFDjAOFaccfeGtSysiwYOWH3Ro07FmFlj
n5HcxCdpuvEuaXVAaktsUqgfBRF6mZEEkVcD9DOrKlmGdhMR3izIKfmI3Fl4sN8=
=Ad3x
-----END PGP SIGNATURE-----




Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Sun, 06 Oct 2013 07:29:54 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Mon Apr 21 16:07:41 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.