Debian Bug report logs - #670581
openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Andris Kalnozols <andris@hpl.hp.com>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Thu, 26 Apr 2012 15:28:17 -0700

Package: openssl
Version: 0.9.8o-4squeeze12
Severity: important
Tags: squeeze

The NTP daemon on our stratum-1 time server `clepsydra.dec.com' keeps
failing with this log message:

Apr 26 12:27:17 clepsydra kernel: [  635.455671] ntpd[2598]: segfault at 20 ip 00007f727f118ec3 sp 00007fff1ecb9a78 error 4 in libcrypto.so.0.9.8[7f727f039000+175000]

When linking with libcrypto.a, crashes still occur:

Apr 26 14:20:19 clepsydra kernel: [ 2191.670043] ntpd[2596]: segfault at 20 ip 0000000000470aa3 sp 00007fff49a93e78 error 4 in ntpd[400000+14f000]

Despite the following:

  clepsydra# ulimit -a
  core file size          (blocks, -c) unlimited

  running the process as root instead of uid=ntp

I am unable to obtain a core file.

This problem is somewhat recent and the workaround seemed to be
a system reboot.  However, after applying the latest openssl
update, the ntp daemon is now esentially broken.

The ntpd process is built from the Debian source using
`apt-get source ntp' with minimal patches to support
the Spectracom TSync GPS reference clock.

For now, I have recompiled the NTP package without OpenSSL
support and it is running fine.

If you need more data or can assist me in getting a core file,
let me know.

Thanks,
Andris


-- System Information:
Debian Release: 6.0.4
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-amd64 (SMP w/12 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/bash

Versions of packages openssl depends on:
ii  libc6                  2.11.3-3          Embedded GNU C Library: Shared lib
ii  libssl0.9.8            0.9.8o-4squeeze12 SSL shared libraries
ii  zlib1g                 1:1.2.3.4.dfsg-3  compression library - runtime

openssl recommends no packages.

Versions of packages openssl suggests:
ii  ca-certificates    20090814+nmu3squeeze1 Common CA certificates

-- no debconf information

Message #10 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Kurt Roeckx <kurt@roeckx.be>

To: Andris Kalnozols <andris@hpl.hp.com>, 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Fri, 27 Apr 2012 20:44:20 +0200

On Thu, Apr 26, 2012 at 03:28:17PM -0700, Andris Kalnozols wrote:
> Package: openssl
> Version: 0.9.8o-4squeeze12
> Severity: important
> Tags: squeeze
> 
> The NTP daemon on our stratum-1 time server `clepsydra.dec.com' keeps
> failing with this log message:
> 
> Apr 26 12:27:17 clepsydra kernel: [  635.455671] ntpd[2598]: segfault at 20 ip 00007f727f118ec3 sp 00007fff1ecb9a78 error 4 in libcrypto.so.0.9.8[7f727f039000+175000]
> 
> When linking with libcrypto.a, crashes still occur:
> 
> Apr 26 14:20:19 clepsydra kernel: [ 2191.670043] ntpd[2596]: segfault at 20 ip 0000000000470aa3 sp 00007fff49a93e78 error 4 in ntpd[400000+14f000]

Did you try this with 0.9.8o-4squeeze11?  I assume
0.9.8o-4squeeze7 didn't have a problem?

In any case I can't reproduce your problem.

As far as I know, ntp only uses openssl for autokey, but I also
don't have a setup with autokey.

> Despite the following:
> 
>   clepsydra# ulimit -a
>   core file size          (blocks, -c) unlimited
> 
>   running the process as root instead of uid=ntp

I assume you removed the -u option?

How about starting it in gdb?  With the -n option it
should not fork, but I think gdb has an option to
follow the fork.



Kurt

Message #15 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Andris Kalnozols <andris@hpl.hp.com>

To: Kurt Roeckx <kurt@roeckx.be>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Fri, 27 Apr 2012 12:36:00 -0700

On 4/27/2012 11:44 AM, Kurt Roeckx wrote:
> On Thu, Apr 26, 2012 at 03:28:17PM -0700, Andris Kalnozols wrote:
>> Package: openssl
>> Version: 0.9.8o-4squeeze12
>> Severity: important
>> Tags: squeeze
>>
>> The NTP daemon on our stratum-1 time server `clepsydra.dec.com' keeps
>> failing with this log message:
>>
>> Apr 26 12:27:17 clepsydra kernel: [  635.455671] ntpd[2598]: segfault at 20 ip 00007f727f118ec3 sp 00007fff1ecb9a78 error 4 in libcrypto.so.0.9.8[7f727f039000+175000]
>>
>> When linking with libcrypto.a, crashes still occur:
>>
>> Apr 26 14:20:19 clepsydra kernel: [ 2191.670043] ntpd[2596]: segfault at 20 ip 0000000000470aa3 sp 00007fff49a93e78 error 4 in ntpd[400000+14f000]
>
> Did you try this with 0.9.8o-4squeeze11?  I assume
> 0.9.8o-4squeeze7 didn't have a problem?

I see that the squeeze7 release is still available:

  apt-get install openssl=0.9.8o-4squeeze7
  Reading package lists... Done
  Building dependency tree
  Reading state information... Done
  The following packages will be DOWNGRADED:
    openssl

but trying to install the squeeze11 version gives this:

  E: Version '0.9.8o-4squeeze11' for 'openssl' was not found

I appears that this is just for the kreebsd-amd64 architecture.

From the changelog, squeeze7 came out in January and I'm pretty
sure that the ntpd process never segfaulted until last month
when subsequent versions of the openssl package were released.
However, apt-get reports that the squeeze{8,9,10} versions are
unavailable to me.

>
>> Despite the following:
>>
>>    clepsydra# ulimit -a
>>    core file size          (blocks, -c) unlimited
>>
>>    running the process as root instead of uid=ntp
>
> I assume you removed the -u option?
>
> How about starting it in gdb?  With the -n option it
> should not fork, but I think gdb has an option to
> follow the fork.

Yes, I modified the init.d startup script to strip out the "-u"
option and run ntpd as root.  After the process aborted, I searched
the entire system for files named "core" and came up empty.

I will try to launch the daemon using gdb(1) and taking advantage
of the "set follow-fork-mode child" option.  I'll use ntpd with
the libcrypto library statically linked.  After getting a stack
backtrace (I hope), I'll downgrade to the squeeze7 version of
openssl as a baseline test to make sure that my assumption of
it not segfaulting is correct.

Thanks,
Andris

Message #20 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Raphael Geissert <geissert@debian.org>

To: Andris Kalnozols <andris@hpl.hp.com>

Cc: 670581@bugs.debian.org

Subject: Re: Bug#670581: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Fri, 27 Apr 2012 15:07:50 -0500

On Friday 27 April 2012 14:36:00 Andris Kalnozols wrote:
> I see that the squeeze7 release is still available:
> 
>    apt-get install openssl=0.9.8o-4squeeze7
>    Reading package lists... Done
>    Building dependency tree
>    Reading state information... Done
>    The following packages will be DOWNGRADED:
>      openssl
> 
> but trying to install the squeeze11 version gives this:
> 
>    E: Version '0.9.8o-4squeeze11' for 'openssl' was not found
> 
> I appears that this is just for the kreebsd-amd64 architecture.

Please try 0.9.8o-4squeeze12, it is in the security archive.

Cheers,
-- 
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net

Message #25 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Andris Kalnozols <andris@hpl.hp.com>

To: 670581@bugs.debian.org

Cc: Raphael Geissert <geissert@debian.org>

Subject: Re: Bug#670581: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Fri, 27 Apr 2012 15:02:09 -0700

On 4/27/2012 1:07 PM, Raphael Geissert wrote:
> On Friday 27 April 2012 14:36:00 Andris Kalnozols wrote:
>> I see that the squeeze7 release is still available:
>>
>>     apt-get install openssl=0.9.8o-4squeeze7
>>     Reading package lists... Done
>>     Building dependency tree
>>     Reading state information... Done
>>     The following packages will be DOWNGRADED:
>>       openssl
>>
>> but trying to install the squeeze11 version gives this:
>>
>>     E: Version '0.9.8o-4squeeze11' for 'openssl' was not found
>>
>> I appears that this is just for the kreebsd-amd64 architecture.
> Please try 0.9.8o-4squeeze12, it is in the security archive.
>
> Cheers,

I do keep the security archive in my `sources.list' file so this latest 
version is
the one that is installed currently:

dpkg -l | grep openssl
ii  openssl                     0.9.8o-4squeeze12           Secure 
Socket Layer (SSL) binary and related cryptographic tools
ii  openssl-blacklist           0.5-2                       list of 
blacklisted OpenSSL RSA keys

------
Andris

Message #30 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Kurt Roeckx <kurt@roeckx.be>

To: Andris Kalnozols <andris@hpl.hp.com>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Sat, 28 Apr 2012 00:49:04 +0200

On Fri, Apr 27, 2012 at 12:36:00PM -0700, Andris Kalnozols wrote:
> On 4/27/2012 11:44 AM, Kurt Roeckx wrote:
> >Did you try this with 0.9.8o-4squeeze11?  I assume
> >0.9.8o-4squeeze7 didn't have a problem?
>
> From the changelog, squeeze7 came out in January and I'm pretty
> sure that the ntpd process never segfaulted until last month
> when subsequent versions of the openssl package were released.
> However, apt-get reports that the squeeze{8,9,10} versions are
> unavailable to me.

8, 9 and 10 were never released.  12 came just after 11, and the
binaries probably aren't really available on the mirrors anymore.

So it would be nice if you can at least confirm that this is a
regression between squeeze7 and squeeze12.  But I don't see
anything in the diff that ntp would be using.  In any case
would a backtrace be useful.


Kurt

Message #35 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Andris Kalnozols <andris@hpl.hp.com>

To: Kurt Roeckx <kurt@roeckx.be>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Fri, 27 Apr 2012 15:56:03 -0700

On 4/27/2012 11:44 AM, Kurt Roeckx wrote:
> On Thu, Apr 26, 2012 at 03:28:17PM -0700, Andris Kalnozols wrote:
>> Package: openssl
>> Version: 0.9.8o-4squeeze12
>> Severity: important
>> Tags: squeeze
>>
>> The NTP daemon on our stratum-1 time server `clepsydra.dec.com' keeps
>> failing with this log message:
>>
>> Apr 26 12:27:17 clepsydra kernel: [  635.455671] ntpd[2598]: segfault at 20 ip 00007f727f118ec3 sp 00007fff1ecb9a78 error 4 in libcrypto.so.0.9.8[7f727f039000+175000]
>>
>> When linking with libcrypto.a, crashes still occur:
>>
>> Apr 26 14:20:19 clepsydra kernel: [ 2191.670043] ntpd[2596]: segfault at 20 ip 0000000000470aa3 sp 00007fff49a93e78 error 4 in ntpd[400000+14f000]

OK, some progress.  I was able to get a stack backtrace with gdb:

clepsydra# gdb /usr/sbin/ntpd
Reading symbols from /usr/sbin/ntpd...done.
(gdb) set set follow-fork-mode child
No symbol "set" in current context.
(gdb) set follow-fork-mode child
(gdb) run -4 -g
Starting program: /usr/sbin/ntpd -4 -g
[Thread debugging using libthread_db enabled]
[New process 4749]
[Thread debugging using libthread_db enabled]
[tcsetpgrp failed in terminal_inferior: No such process]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fe9700 (LWP 4749)]
0x00000000004776d3 in EVP_DigestUpdate ()
(gdb) bt
#0  0x00000000004776d3 in EVP_DigestUpdate ()
#1  0x000000000041c5ee in session_key (srcadr=0x7d50f0, dstadr=0x7db1b0,
    keyno=0, private=<value optimized out>, lifetime=0) at ntp_crypto.c:233
#2  0x00000000004254db in fast_xmit (rbufp=0x7d50e0,
    xmode=<value optimized out>, xkeyid=510781053, flags=<value 
optimized out>)
    at ntp_proto.c:3320
#3  0x00000000004260a7 in receive (rbufp=0x7d50e0) at ntp_proto.c:476
#4  0x00000000004128c9 in ntpdmain (argc=3, argv=0x7fffffffe678) at 
ntpd.c:1172
#5  0x00007ffff6fecc8d in __libc_start_main () from /lib/libc.so.6
#6  0x00000000004047a9 in _start ()


Appended is the session_key() function from `ntp_crypto.c' where the failure
is occurring.  Let me know whatever debugging steps you would like me to do.

Thanks,
Andris



/*
 * session_key - generate session key
 *
 * This routine generates a session key from the source address,
 * destination address, key ID and private value. The value of the
 * session key is the MD5 hash of these values, while the next key ID is
 * the first four octets of the hash.
 *
 * Returns the next key ID or 0 if there is no destination address.
 */
keyid_t
session_key(
        sockaddr_u *srcadr,     /* source address */
        sockaddr_u *dstadr,     /* destination address */
        keyid_t keyno,          /* key ID */
        keyid_t private,        /* private value */
        u_long  lifetime        /* key lifetime */
        )
{
        EVP_MD_CTX ctx;         /* message digest context */
        u_char dgst[EVP_MAX_MD_SIZE]; /* message digest */
        keyid_t keyid;          /* key identifer */
        u_int32 header[10];     /* data in network byte order */
        u_int   hdlen, len;

        if (!dstadr)
                return 0;

        /*
         * Generate the session key and key ID. If the lifetime is
         * greater than zero, install the key and call it trusted.
         */
        hdlen = 0;
        switch(AF(srcadr)) {
        case AF_INET:
                header[0] = NSRCADR(srcadr);
                header[1] = NSRCADR(dstadr);
                header[2] = htonl(keyno);
                header[3] = htonl(private);
                hdlen = 4 * sizeof(u_int32);
                break;

        case AF_INET6:
                memcpy(&header[0], PSOCK_ADDR6(srcadr),
                    sizeof(struct in6_addr));
                memcpy(&header[4], PSOCK_ADDR6(dstadr),
                    sizeof(struct in6_addr));
                header[8] = htonl(keyno);
                header[9] = htonl(private);
                hdlen = 10 * sizeof(u_int32);
                break;
        }
        EVP_DigestInit(&ctx, EVP_get_digestbynid(crypto_nid));
--->    EVP_DigestUpdate(&ctx, (u_char *)header, hdlen);
        EVP_DigestFinal(&ctx, dgst, &len);
        memcpy(&keyid, dgst, 4);
        keyid = ntohl(keyid);
        if (lifetime != 0) {
                MD5auth_setkey(keyno, crypto_nid, dgst, len);
                authtrust(keyno, lifetime);
        }
        DPRINTF(2, ("session_key: %s > %s %08x %08x hash %08x life %lu\n",
                    stoa(srcadr), stoa(dstadr), keyno,
                    private, keyid, lifetime));

        return (keyid);
}

Message #40 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Kurt Roeckx <kurt@roeckx.be>

To: Andris Kalnozols <andris@hpl.hp.com>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Sat, 28 Apr 2012 01:11:39 +0200

On Fri, Apr 27, 2012 at 03:56:03PM -0700, Andris Kalnozols wrote:
> On 4/27/2012 11:44 AM, Kurt Roeckx wrote:
> >On Thu, Apr 26, 2012 at 03:28:17PM -0700, Andris Kalnozols wrote:
> >>Package: openssl
> >>Version: 0.9.8o-4squeeze12
> >>Severity: important
> >>Tags: squeeze
> >>
> >>The NTP daemon on our stratum-1 time server `clepsydra.dec.com' keeps
> >>failing with this log message:
> >>
> >>Apr 26 12:27:17 clepsydra kernel: [  635.455671] ntpd[2598]: segfault at 20 ip 00007f727f118ec3 sp 00007fff1ecb9a78 error 4 in libcrypto.so.0.9.8[7f727f039000+175000]
> >>
> >>When linking with libcrypto.a, crashes still occur:
> >>
> >>Apr 26 14:20:19 clepsydra kernel: [ 2191.670043] ntpd[2596]: segfault at 20 ip 0000000000470aa3 sp 00007fff49a93e78 error 4 in ntpd[400000+14f000]
> 
> OK, some progress.  I was able to get a stack backtrace with gdb:
> 
> clepsydra# gdb /usr/sbin/ntpd
> Reading symbols from /usr/sbin/ntpd...done.
> (gdb) set set follow-fork-mode child
> No symbol "set" in current context.
> (gdb) set follow-fork-mode child
> (gdb) run -4 -g
> Starting program: /usr/sbin/ntpd -4 -g
> [Thread debugging using libthread_db enabled]
> [New process 4749]
> [Thread debugging using libthread_db enabled]
> [tcsetpgrp failed in terminal_inferior: No such process]
> 
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffff7fe9700 (LWP 4749)]
> 0x00000000004776d3 in EVP_DigestUpdate ()
> (gdb) bt
> #0  0x00000000004776d3 in EVP_DigestUpdate ()
> #1  0x000000000041c5ee in session_key (srcadr=0x7d50f0, dstadr=0x7db1b0,
>     keyno=0, private=<value optimized out>, lifetime=0) at ntp_crypto.c:233
> #2  0x00000000004254db in fast_xmit (rbufp=0x7d50e0,
>     xmode=<value optimized out>, xkeyid=510781053, flags=<value
> optimized out>)
>     at ntp_proto.c:3320
> #3  0x00000000004260a7 in receive (rbufp=0x7d50e0) at ntp_proto.c:476
> #4  0x00000000004128c9 in ntpdmain (argc=3, argv=0x7fffffffe678) at
> ntpd.c:1172
> #5  0x00007ffff6fecc8d in __libc_start_main () from /lib/libc.so.6
> #6  0x00000000004047a9 in _start ()
> 
> 
> Appended is the session_key() function from `ntp_crypto.c' where the failure
> is occurring.  Let me know whatever debugging steps you would like me to do.

header and hdlen from frame 1 would be nice.

Could you also install the libssl0.9.8-dbg package?  You'll get
debug symbols then.


Kurt

Message #45 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Kurt Roeckx <kurt@roeckx.be>

To: Andris Kalnozols <andris@hpl.hp.com>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Sat, 28 Apr 2012 01:13:19 +0200

On Sat, Apr 28, 2012 at 01:11:39AM +0200, Kurt Roeckx wrote:
> 
> header and hdlen from frame 1 would be nice.

I guess ctx and crypto_nid would also be nice.


Kurt

Message #50 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Andris Kalnozols <andris@hpl.hp.com>

To: Kurt Roeckx <kurt@roeckx.be>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Fri, 27 Apr 2012 17:42:07 -0700

On 4/27/2012 4:11 PM, Kurt Roeckx wrote:
> header and hdlen from frame 1 would be nice.
>
> Could you also install the libssl0.9.8-dbg package?  You'll get
> debug symbols then.
>

I'm not a real pro with the debugger.  After installing the `dbg' 
package, is
this all that's needed?:

gdb /usr/sbin/ntpd
Reading symbols from /usr/sbin/ntpd...done.
(gdb) add-symbol-file /usr/lib/debug/usr/lib/libcrypto.so.0.9.8
The address where /usr/lib/debug/usr/lib/libcrypto.so.0.9.8 has been 
loaded is missing

This doesn't seem like it would work.  I could be wrong.

This doesn't work either:

clepsydra# export LD_LIBRARY_PATH=/usr/lib/debug/usr/lib:/usr/lib
clepsydra# ldd /usr/sbin/ntpd
/usr/bin/ldd: line 118: 30380 Segmentation fault      (core dumped) 
LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= 
LD_LIBRARY_VERSION=$verify_out LD_VERBOSE= "$@"

Modifying the Makefile to link ntpd using libcrypto in 
/usr/lib/debug/usr/lib
required me to also install the zlib1g-dbg package to resolve some 
missing symbols.
Like the attempted LD_LIBRARY_PATH hack, this also segfaulted when 
starting ntpd.

Help.

------
Andris

Message #55 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Kurt Roeckx <kurt@roeckx.be>

To: Andris Kalnozols <andris@hpl.hp.com>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Sat, 28 Apr 2012 10:57:26 +0200

On Fri, Apr 27, 2012 at 05:42:07PM -0700, Andris Kalnozols wrote:
> On 4/27/2012 4:11 PM, Kurt Roeckx wrote:
> >header and hdlen from frame 1 would be nice.
> >
> >Could you also install the libssl0.9.8-dbg package?  You'll get
> >debug symbols then.
> >
> 
> I'm not a real pro with the debugger.  After installing the `dbg'
> package, is
> this all that's needed?:

gdb should look load the debug symbols from
/usr/lib/debug/usr/lib/libcrypto.so.0.9.8 automaticly,
there is no need to add the symbol files in gdb.

> Modifying the Makefile to link ntpd using libcrypto in
> /usr/lib/debug/usr/lib

There is also no need for that, and it won't work, because it
only contains the debug symbols.

The library in /usr/lib/ tells that it's debug symbols have been
removed and placed in the file in /usr/lib/debug/usr/lib/, and gdb
should automaticly pick this up.



Kurt

Message #60 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Andris Kalnozols <andris@hpl.hp.com>

To: Kurt Roeckx <kurt@roeckx.be>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Sun, 29 Apr 2012 17:13:39 -0700

On 4/28/2012 1:57 AM, Kurt Roeckx wrote:
> On Fri, Apr 27, 2012 at 05:42:07PM -0700, Andris Kalnozols wrote:
>> On 4/27/2012 4:11 PM, Kurt Roeckx wrote:
>>> header and hdlen from frame 1 would be nice.
>>>
>>> Could you also install the libssl0.9.8-dbg package?  You'll get
>>> debug symbols then.
>>>
>>
>> I'm not a real pro with the debugger.  After installing the `dbg'
>> package, is
>> this all that's needed?:
>
> gdb should look load the debug symbols from
> /usr/lib/debug/usr/lib/libcrypto.so.0.9.8 automaticly,
> there is no need to add the symbol files in gdb.
>
>> Modifying the Makefile to link ntpd using libcrypto in
>> /usr/lib/debug/usr/lib
>
> There is also no need for that, and it won't work, because it
> only contains the debug symbols.
>
> The library in /usr/lib/ tells that it's debug symbols have been
> removed and placed in the file in /usr/lib/debug/usr/lib/, and gdb
> should automaticly pick this up.

Thanks, Kurt.

Before I got your reply, I went to "Plan B" and did the following:

  * apt-get source openssl
  * changed the optimization from -O3 to -O0 in the Configure script
    for the "debian-amd64" section
  * make -f debian/rules
  * relinked the ntpd program with the libcrypto.static library

I also added the following code:

--- ntp_crypto.c.orig   2009-12-08 23:36:35.000000000 -0800
+++ ntp_crypto.c        2012-04-29 15:16:50.181208921 -0700
@@ -230,6 +230,38 @@
                break;
        }
        EVP_DigestInit(&ctx, EVP_get_digestbynid(crypto_nid));
+       /*
+        * Begin debugging output
+        */
+       switch(AF(srcadr)) {
+       char src_buf[INET6_ADDRSTRLEN],
+            dst_buf[INET6_ADDRSTRLEN];
+
+       case AF_INET:
+               inet_ntop(AF_INET, &header[0], src_buf, INET6_ADDRSTRLEN);
+               inet_ntop(AF_INET, &header[1], dst_buf, INET6_ADDRSTRLEN);
+               msyslog(LOG_INFO, "session_key() IPv4 data: header[0]=%u, "
+                                 "src_addr=%s, header[1]=%u, dst_addr=%s",
+                                 header[0], src_buf, header[1], dst_buf);
+               break;
+       case AF_INET6:
+               inet_ntop(AF_INET6, &header[0], src_buf, INET6_ADDRSTRLEN);
+               inet_ntop(AF_INET6, &header[4], dst_buf, INET6_ADDRSTRLEN);
+               msyslog(LOG_INFO, "session_key() IPv6 data: 
header[0-3]=%u, "
+                                 "%u, %u, %u, src_addr=%s, 
header[4-7]=%u, "
+                                 "%u, %u, %u, dst_addr=%s", header[0],
+                                 header[1], header[2], header[3], src_buf,
+                                 header[4], header[5], header[6], 
header[7],
+                                 dst_buf);
+               break;
+       }
+       msyslog(LOG_INFO, "remaining session_key() data [host byte order]: "
+                         "hdlen=%u, keyno=%u, private=%u, crypto_nid=%u, "
+                         "ctx.flags=%lu",
+                         hdlen, keyno, private, crypto_nid, ctx.flags);
+       /*
+        * End debugging output
+        */
        EVP_DigestUpdate(&ctx, (u_char *)header, hdlen);
        EVP_DigestFinal(&ctx, dgst, &len);
        memcpy(&keyid, dgst, 4);

This logged the following output:

29 Apr 16:06:29 ntpd[23221]: session_key() IPv4 data: 
header[0]=3096497361, src_addr=209.204.144.184,
   header[1]=84048844, dst_addr=204.123.2.5
29 Apr 16:06:29 ntpd[23221]: remaining session_key() data [host byte 
order]: hdlen=16, keyno=0,
   private=3604904099, crypto_nid=4, ctx.flags=0

A full backtrace of the first two frames after the segfault shows:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fe9700 (LWP 23221)]
0x0000000000497c02 in EVP_DigestUpdate (ctx=0x7fffffffda20,
    data=0x7fffffffd9b0, count=16) at digest.c:325
325             return ctx->digest->update(ctx,data,count);
(gdb) backtrace full
#0  0x0000000000497c02 in EVP_DigestUpdate (ctx=0x7fffffffda20,
    data=0x7fffffffd9b0, count=16) at digest.c:325
No locals.
#1  0x000000000041d23f in session_key (srcadr=0x8110f0, dstadr=0x8171b0,
    keyno=0, private=3604904099, lifetime=0) at ntp_crypto.c:265
        ctx = {digest = 0x0, engine = 0x0, flags = 0, md_data = 0x0}
        dgst = "%\000\000\000\000\000\000\000\035JB\000\000\000\000
                \000\004\000\000\000\377\177\000\000P\266\377\367
                \377\177\000\000\000\003G\323\000\000\000\000<\002A
                \000\000\000\000\000P\332\377\377\377\177\000\000
                \000\000\000\000\000\000\000"
        keyid = 32767
        header = {3096497361, 84048844, 0, 2742607574, 4294960736,
                  32767, 0, 0, 0, 0}
        hdlen = 16
        len = 32767

The other notable thing is that it's always the same source IP address
[209.204.144.184] which appears before the server aborts.  A packet
capture of traffic from/to this IP address is at:

  ftp://ftp.hpl.hp.com/outgoing/andris/openssl/ntpd.pcap

This client IP address did not abort the server right away.  Five NTP
queries were answered.  The sixth query is what caused this abort.

Regards
Andris

Message #65 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Kurt Roeckx <kurt@roeckx.be>

To: Andris Kalnozols <andris@hpl.hp.com>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Mon, 30 Apr 2012 12:07:05 +0200

On Sun, Apr 29, 2012 at 05:13:39PM -0700, Andris Kalnozols wrote:
> 
> I also added the following code:
> 
> --- ntp_crypto.c.orig   2009-12-08 23:36:35.000000000 -0800
> +++ ntp_crypto.c        2012-04-29 15:16:50.181208921 -0700
> @@ -230,6 +230,38 @@
>                 break;
>         }

Can you add this line:
	EVP_MD_CTX_init(&ctx);

Before this line:
	
>         EVP_DigestInit(&ctx, EVP_get_digestbynid(crypto_nid));

The documentation clearly says that ctx must be initialized before
calling EVP_DigestInit{_ex}



Kurt

Message #70 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Andris Kalnozols <andris@hpl.hp.com>

To: Kurt Roeckx <kurt@roeckx.be>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Mon, 30 Apr 2012 17:25:44 -0700

On 4/30/2012 3:07 AM, Kurt Roeckx wrote:
> On Sun, Apr 29, 2012 at 05:13:39PM -0700, Andris Kalnozols wrote:
>> I also added the following code:
>>
>> --- ntp_crypto.c.orig   2009-12-08 23:36:35.000000000 -0800
>> +++ ntp_crypto.c        2012-04-29 15:16:50.181208921 -0700
>> @@ -230,6 +230,38 @@
>>                  break;
>>          }
> Can you add this line:
> 	EVP_MD_CTX_init(&ctx);
>
> Before this line:
> 	
>>          EVP_DigestInit(&ctx, EVP_get_digestbynid(crypto_nid));
> The documentation clearly says that ctx must be initialized before
> calling EVP_DigestInit{_ex}
>
>
>
> Kurt
>

I added the EVP_MD_CTX_init() routine as suggested but still get the
same failure after a while:


Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fe8700 (LWP 2771)]
0x00007ffff767eec3 in EVP_DigestUpdate (ctx=0x7fffffffda30,
    data=0x7fffffffd9c0, count=16) at digest.c:325
325     digest.c: No such file or directory.
        in digest.c
(gdb) bt full
#0  0x00007ffff767eec3 in EVP_DigestUpdate (ctx=0x7fffffffda30,
    data=0x7fffffffd9c0, count=16) at digest.c:325
No locals.
#1  0x000000000041e99b in session_key (srcadr=0x7070f0, dstadr=0x70d1b0,
    keyno=0, private=1378153285, lifetime=0) at ntp_crypto.c:266
        ctx = {digest = 0x0, engine = 0x0, flags = 0, md_data = 0x0}
        dgst = "\016\000\000\000\000\000\000\000\320\341\377\377\377\177\000\000P\337\377\377\377\177\000\000P\266\377\367\377\177\000\000\370\337\377\377\377\177\000\000\214\031A\000\000\000\000\000`\332\377\377\377\177\000\000\000\000\000\000\000\000\000"
        keyid = 32767
        header = {3096497361, 84048844, 0, 1173562450, 0, 0, 4779203, 0, 4779203, 0}
        hdlen = 16
        len = 32767


30 Apr 13:30:59 ntpd[2771]: session_key() IPv4 data: header[0]=3096497361, src_addr=209.204.144.184,
   header[1]=84048844, dst_addr=204.123.2.5
30 Apr 13:30:59 ntpd[2771]: remaining session_key() data [host byte order]: hdlen=16, keyno=0,
   private=1378153285, crypto_nid=4, ctx.flags=0

According to the EVP_DigestInit(3SSL) man page installed on my system, I see this:

       EVP_DigestInit() behaves in the same way as EVP_DigestInit_ex() except
       the passed context ctx does not have to be initialized, and it always
       uses the default digest implementation.

So it appears that NTP's session_key() is calling the digest routines correctly.

In `digest.c', the failing line of code seems to involve a function pointer
if I'm interpreting the EVP_MD structure correctly from `crypto/evp/evp.h:

  int (*update)(EVP_MD_CTX *ctx,const void *data,size_t count);

I made a guess that *update pointed to HASH_UPDATE() in `crypto/md32_common.h'
and put in some debugging statements there.  However, when running the NTP daemon
in the foreground, there was no output from any of the HASH_UPDATE() printf()
statements prior to the segfault.

Regards,
Andris

Message #75 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Andris Kalnozols <andris@hpl.hp.com>

To: Kurt Roeckx <kurt@roeckx.be>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Mon, 30 Apr 2012 23:54:20 -0700

I believe I found the smoking gun by breaking out the call
to EVP_get_digestbynid(crypto_nid) and testing its return
value:

> --- ntp_crypto.c.orig   2009-12-08 23:36:35.000000000 -0800
> +++ ntp_crypto.c        2012-04-30 22:44:46.094602274 -0700
> @@ -197,6 +197,7 @@
>         )
>  {
>         EVP_MD_CTX ctx;         /* message digest context */
> +       const EVP_MD *type;
>         u_char dgst[EVP_MAX_MD_SIZE]; /* message digest */
>         keyid_t keyid;          /* key identifer */
>         u_int32 header[10];     /* data in network byte order */
> @@ -229,7 +230,43 @@
>                 hdlen = 10 * sizeof(u_int32);
>                 break;
>         }
> -       EVP_DigestInit(&ctx, EVP_get_digestbynid(crypto_nid));
> +       if ((type = EVP_get_digestbynid(crypto_nid)) == NULL)
> +               msyslog(LOG_ERR, "EVP_get_digestbynid(KEY_TYPE_MD5) "
> +                                "returned NULL");
> +       EVP_DigestInit(&ctx, type);
>         EVP_DigestUpdate(&ctx, (u_char *)header, hdlen);
>         EVP_DigestFinal(&ctx, dgst, &len);
>         memcpy(&keyid, dgst, 4);

Sure enough, this routine returns NULL before the program segfaults.
A web search led me to this:
  > 
http://stackoverflow.com/questions/6762054/evp-get-digestbyname-what-is-this

Indeed, the Debian man page for evp_get_digestbynid(3) has this 
important information:

> EVP_get_digestbyname(), EVP_get_digestbynid() and EVP_get_digestbyobj() return
> an EVP_MD structure when passed a digest name, a digest NID or an 
ASN1_OBJECT
> structure respectively. The digest table must be initialized using, 
for example,
> OpenSSL_add_all_digests() for these functions to work.

My search of the NTP source code for references to
"OpenSSL_add_all_*" showed the following two locations:

libntp/ssl_init.c:
------------------

> #ifdef OPENSSL
> #include "openssl/err.h"
> #include "openssl/rand.h"
>
>
> int ssl_init_done;
>
> void
> ssl_init(void)
> {
>         if (ssl_init_done)
>                 return;
>
>         ERR_load_crypto_strings();
>         OpenSSL_add_all_algorithms();
>
>         ssl_init_done = 1;
> }
>
>
> void
> ssl_check_version(void)
> {
>         INIT_SSL();
> }
> #endif  /* OPENSSL */

include/ntp_stdlib.h:
---------------------

> /* ssl_init.c */
> #ifdef OPENSSL
> extern  void    ssl_init                (void);
> extern  void    ssl_check_version       (void);
> extern  int     ssl_init_done;
> #define INIT_SSL()                              \
>         do {                                    \
>                 if (!ssl_init_done)             \
>                         ssl_init();             \
>         } while (0)
> #else   /* !OPENSSL follows */
> #define INIT_SSL()              do {} while (0)
> #endif

However, I can't find where the ntpd program ever calls ssl_init()
or the INIT_SSL() macro.  So I'm betting that this problem will go
away once I find the proper place to call this important routine.

Perhaps you can keep this ticket open until I can confirm my hunch.
It shouldn't take too long.

Thanks,
Andris

Message #80 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Kurt Roeckx <kurt@roeckx.be>

To: Andris Kalnozols <andris@hpl.hp.com>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Tue, 1 May 2012 11:40:58 +0200

On Mon, Apr 30, 2012 at 05:25:44PM -0700, Andris Kalnozols wrote:
> 
> In `digest.c', the failing line of code seems to involve a function pointer
> if I'm interpreting the EVP_MD structure correctly from `crypto/evp/evp.h:
> 
>   int (*update)(EVP_MD_CTX *ctx,const void *data,size_t count);
> 
> I made a guess that *update pointed to HASH_UPDATE() in `crypto/md32_common.h'
> and put in some debugging statements there.  However, when running the NTP daemon
> in the foreground, there was no output from any of the HASH_UPDATE() printf()
> statements prior to the segfault.

It should be using the update function from the struct it gets
with EVP_get_digestbynid(crypto_nid), which will be a static
function in openssl.

It's all making very little sense to me.


Kurt

Message #85 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Andris Kalnozols <andris@hpl.hp.com>

To: Kurt Roeckx <kurt@roeckx.be>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Tue, 01 May 2012 09:52:06 -0700

On 4/30/2012 11:54 PM, Andris Kalnozols wrote:

> However, I can't find where the ntpd program ever calls ssl_init()
> or the INIT_SSL() macro. So I'm betting that this problem will go
> away once I find the proper place to call this important routine.

I took a look at the latest development code base (ntp-dev-4.2.7p275)
from ntp.org and it appears that the missing SSL initialization call
is now present.  Backporting the fix to the proper location gives this
for the version of NTP (4.2.6p2) for Debian squeeze:

> --- ntpd.c.orig 2012-04-22 00:33:29.000000000 -0700
> +++ ntpd.c      2012-05-01 00:04:55.770602014 -0700
> @@ -836,10 +836,11 @@
>         /*
>          * Call the init_ routines to initialize the data structures.
>          *
>          * Exactly what command-line options are we expecting here?
>          */
> +       INIT_SSL();
>         init_auth();
>         init_util();
>         init_restrict();
>         init_mon();
>         init_timer();

This does indeed fix the problem.

My apologies for the false bug report but with your help I was able
to dig deeper into the OpenSSL internals that I could have otherwise.

I will notify the Debian maintainers for NTP of this important patch.

Best regards,
Andris

Message #90 received at 670581@bugs.debian.org (full text, mbox, reply):

From: Kurt Roeckx <kurt@roeckx.be>

To: Andris Kalnozols <andris@hpl.hp.com>

Cc: 670581@bugs.debian.org

Subject: Re: [Pkg-openssl-devel] Bug#670581: openssl: ntpd segfaults with error 4 in libcrypto.so.0.9.8 on Debian squeeze

Date: Tue, 1 May 2012 19:05:51 +0200

reassign 670581 src:ntp 1:4.2.6.p2+dfsg-1
thanks

On Tue, May 01, 2012 at 09:52:06AM -0700, Andris Kalnozols wrote:
> On 4/30/2012 11:54 PM, Andris Kalnozols wrote:
> 
> >However, I can't find where the ntpd program ever calls ssl_init()
> >or the INIT_SSL() macro. So I'm betting that this problem will go
> >away once I find the proper place to call this important routine.
> 
> I took a look at the latest development code base (ntp-dev-4.2.7p275)
> from ntp.org and it appears that the missing SSL initialization call
> is now present.  Backporting the fix to the proper location gives this
> for the version of NTP (4.2.6p2) for Debian squeeze:
> 
> >--- ntpd.c.orig 2012-04-22 00:33:29.000000000 -0700
> >+++ ntpd.c      2012-05-01 00:04:55.770602014 -0700
> >@@ -836,10 +836,11 @@
> >        /*
> >         * Call the init_ routines to initialize the data structures.
> >         *
> >         * Exactly what command-line options are we expecting here?
> >         */
> >+       INIT_SSL();
> >        init_auth();
> >        init_util();
> >        init_restrict();
> >        init_mon();
> >        init_timer();
> 
> This does indeed fix the problem.

Thanks for looking into this.

> I will notify the Debian maintainers for NTP of this important patch.

I've just reassign it to the proper package then.

PS: I'm also one of the ntp maintainers.


Kurt

Message #101 received at 670581-done@bugs.debian.org (full text, mbox, reply):

From: Adrian Bunk <bunk@debian.org>

To: 632844-done@bugs.debian.org, 586271-done@bugs.debian.org, 507399-done@bugs.debian.org, 577072-done@bugs.debian.org, 579536-done@bugs.debian.org, 580646-done@bugs.debian.org, 580956-done@bugs.debian.org, 581882-done@bugs.debian.org, 586054-done@bugs.debian.org, 588772-done@bugs.debian.org, 592412-done@bugs.debian.org, 593332-done@bugs.debian.org, 595250-done@bugs.debian.org, 597304-done@bugs.debian.org, 610593-done@bugs.debian.org, 615128-done@bugs.debian.org, 615521-done@bugs.debian.org, 616056-done@bugs.debian.org, 616192-done@bugs.debian.org, 618313-done@bugs.debian.org, 627760-done@bugs.debian.org, 631812-done@bugs.debian.org, 636092-done@bugs.debian.org, 645653-done@bugs.debian.org, 645889-done@bugs.debian.org, 646362-done@bugs.debian.org, 670581-done@bugs.debian.org, 673053-done@bugs.debian.org, 679015-done@bugs.debian.org, 697592-done@bugs.debian.org, 700916-done@bugs.debian.org, 718431-done@bugs.debian.org, 633644-done@bugs.debian.org, 725622-done@bugs.debian.org, 611398-done@bugs.debian.org, 627811-done@bugs.debian.org, 248437-done@bugs.debian.org, 273816-done@bugs.debian.org, 377494-done@bugs.debian.org, 577027-done@bugs.debian.org, 577930-done@bugs.debian.org, 579186-done@bugs.debian.org, 579295-done@bugs.debian.org, 580258-done@bugs.debian.org, 580849-done@bugs.debian.org, 581595-done@bugs.debian.org, 581786-done@bugs.debian.org, 582292-done@bugs.debian.org, 583823-done@bugs.debian.org, 584246-done@bugs.debian.org, 584539-done@bugs.debian.org, 584577-done@bugs.debian.org, 584668-done@bugs.debian.org, 585721-done@bugs.debian.org, 586049-done@bugs.debian.org, 586283-done@bugs.debian.org, 586661-done@bugs.debian.org, 590283-done@bugs.debian.org, 593379-done@bugs.debian.org, 597383-done@bugs.debian.org, 597457-done@bugs.debian.org, 598521-done@bugs.debian.org, 599205-done@bugs.debian.org, 602927-done@bugs.debian.org, 603123-done@bugs.debian.org, 604156-done@bugs.debian.org, 605010-done@bugs.debian.org, 607718-done@bugs.debian.org, 610740-done@bugs.debian.org, 611218-done@bugs.debian.org, 611244-done@bugs.debian.org, 611313-done@bugs.debian.org, 613206-done@bugs.debian.org, 613627-done@bugs.debian.org, 614767-done@bugs.debian.org, 616719-done@bugs.debian.org, 617519-done@bugs.debian.org, 621094-done@bugs.debian.org, 633095-done@bugs.debian.org, 633159-done@bugs.debian.org, 636916-done@bugs.debian.org, 638410-done@bugs.debian.org, 640844-done@bugs.debian.org, 641029-done@bugs.debian.org, 642997-done@bugs.debian.org, 645092-done@bugs.debian.org, 645093-done@bugs.debian.org, 650392-done@bugs.debian.org, 650490-done@bugs.debian.org, 651751-done@bugs.debian.org, 653242-done@bugs.debian.org, 655493-done@bugs.debian.org, 658214-done@bugs.debian.org, 665205-done@bugs.debian.org, 665481-done@bugs.debian.org, 666403-done@bugs.debian.org, 687149-done@bugs.debian.org, 699600-done@bugs.debian.org, 767203-done@bugs.debian.org, 503915-done@bugs.debian.org, 594545-done@bugs.debian.org, 599129-done@bugs.debian.org, 646534-done@bugs.debian.org, 660197-done@bugs.debian.org, 583873-done@bugs.debian.org, 585008-done@bugs.debian.org, 631186-done@bugs.debian.org, 663920-done@bugs.debian.org, 698485-done@bugs.debian.org, 674894-done@bugs.debian.org, 581982-done@bugs.debian.org, 582269-done@bugs.debian.org, 583877-done@bugs.debian.org, 598567-done@bugs.debian.org, 614863-done@bugs.debian.org, 645757-done@bugs.debian.org, 646509-done@bugs.debian.org, 646583-done@bugs.debian.org, 698964-done@bugs.debian.org, 680276-done@bugs.debian.org, 596792-done@bugs.debian.org, 561359-done@bugs.debian.org, 575412-done@bugs.debian.org, 578868-done@bugs.debian.org, 579591-done@bugs.debian.org, 585049-done@bugs.debian.org, 596879-done@bugs.debian.org, 596947-done@bugs.debian.org, 599533-done@bugs.debian.org, 605400-done@bugs.debian.org, 613926-done@bugs.debian.org, 630380-done@bugs.debian.org, 668034-done@bugs.debian.org, 675371-done@bugs.debian.org, 700578-done@bugs.debian.org, 489441-done@bugs.debian.org, 580126-done@bugs.debian.org, 587502-done@bugs.debian.org, 607497-done@bugs.debian.org, 620907-done@bugs.debian.org, 630888-done@bugs.debian.org, 648811-done@bugs.debian.org, 594885-done@bugs.debian.org, 598760-done@bugs.debian.org, 657483-done@bugs.debian.org, 680393-done@bugs.debian.org

Subject: Closing squeeze-only bugs

Date: Mon, 27 Feb 2017 21:34:36 +0200

Dear submitter,

these bug are tagged squeeze without any wheezy/jessie/stretch tag
implying that the bug is not present in more recent Debian releases.

squeeze is no longer supported.

We are sorry that we couldn't deal with your issue in squeeze.

If this bug was incorrectly tagged squeeze, please reopen the bug
and remove the squeeze tag.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Send a report that this bug log contains spam.