Debian Bug report logs - #509292
rsyslog: random crashes on multi core machines

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Juha Koho <jmcsa00@gmail.com>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: rsyslog: random crashes with remote logging

Date: Sun, 21 Dec 2008 00:26:39 +0200

Package: rsyslog
Version: 3.18.5-1
Severity: critical
Justification: causes serious data loss

Hello,

I have noticed that rsyslog randomly crashes with remote logging 
(client) enabled. This happens almost once a day and all log messages 
are lost until rsyslog is started again. I don't know if this is related 
to remote logging but all my other installations with no remote logging 
work fine.

Normally there is nothing special in the log files when rsyslog crashes 
and I have not been able to find out the cause. Today I noticed the 
following:

*** syslog
*** glibc detected *** /usr/sbin/rsyslogd: malloc(): memory corruption (fast): 0x00007f633001d33f ***
======= Backtrace: =========
/lib/libc.so.6[0x7f63378c3948]
/lib/libc.so.6[0x7f63378c667f]
/lib/libc.so.6(__libc_malloc+0x98)[0x7f63378c7a78]
/usr/sbin/rsyslogd(rsCStrGetSzStr+0x5d)[0x414b1d]
/usr/sbin/rsyslogd(rsCStrConvSzStrAndDestruct+0x24)[0x414cc4]
/usr/sbin/rsyslogd(logmsg+0x50c)[0x40ba4c]
/usr/sbin/rsyslogd(printline+0xed)[0x40be8d]
/usr/sbin/rsyslogd(parseAndSubmitMessage+0x126)[0x40c046]
/usr/lib/rsyslog/imuxsock.so[0x7f633703b2d5]
/usr/lib/rsyslog/imuxsock.so[0x7f633703b4a7]
/usr/sbin/rsyslogd[0x416470]
/lib/libpthread.so.0[0x7f6337fb6fc7]
/lib/libc.so.6(clone+0x6d)[0x7f633791f5ad]
======= Memory map: ========
00400000-0043b000 r-xp 00000000 fd:02 49160                              /usr/sbin/rsyslogd
0063a000-0063e000 rw-p 0003a000 fd:02 49160                              /usr/sbin/rsyslogd
0063e000-0063f000 rw-p 0063e000 00:00 0 
02058000-0211f000 rw-p 02058000 00:00 0                                  [heap]
408ed000-408ee000 ---p 408ed000 00:00 0 
408ee000-410ee000 rw-p 408ee000 00:00 0 
410ee000-410ef000 ---p 410ee000 00:00 0 
410ef000-418ef000 rw-p 410ef000 00:00 0 
41b14000-41b15000 ---p 41b14000 00:00 0 
41b15000-42315000 rw-p 41b15000 00:00 0 
42315000-42316000 ---p 42315000 00:00 0 
42316000-42b16000 rw-p 42316000 00:00 0 
7f6328000000-7f6328021000 rw-p 7f6328000000 00:00 0 
7f6328021000-7f632c000000 ---p 7f6328021000 00:00 0 
7f6330000000-7f6330021000 rw-p 7f6330000000 00:00 0 
7f6330021000-7f6334000000 ---p 7f6330021000 00:00 0 
7f6336c1c000-7f6336c32000 r-xp 00000000 fd:00 8226                       /lib/libgcc_s.so.1
7f6336c32000-7f6336e32000 ---p 00016000 fd:00 8226                       /lib/libgcc_s.so.1
7f6336e32000-7f6336e33000 rw-p 00016000 fd:00 8226                       /lib/libgcc_s.so.1
7f6336e33000-7f6336e38000 r-xp 00000000 fd:02 73841                      /usr/lib/rsyslog/imklog.so
7f6336e38000-7f6337038000 ---p 00005000 fd:02 73841                      /usr/lib/rsyslog/imklog.so
7f6337038000-7f6337039000 rw-p 00005000 fd:02 73841                      /usr/lib/rsyslog/imklog.so
7f6337039000-7f633703a000 rw-p 7f6337039000 00:00 0 
7f633703a000-7f633703c000 r-xp 00000000 fd:02 73834                      /usr/lib/rsyslog/imuxsock.so
7f633703c000-7f633723c000 ---p 00002000 fd:02 73834                      /usr/lib/rsyslog/imuxsock.so
7f633723c000-7f633723d000 rw-p 00002000 fd:02 73834                      /usr/lib/rsyslog/imuxsock.so
7f633723d000-7f633723f000 r-xp 00000000 fd:02 73833                      /usr/lib/rsyslog/lmtcpclt.so
7f633723f000-7f633743e000 ---p 00002000 fd:02 73833                      /usr/lib/rsyslog/lmtcpclt.so
7f633743e000-7f633743f000 rw-p 00001000 fd:02 73833                      /usr/lib/rsyslog/lmtcpclt.so
7f633743f000-7f6337449000 r-xp 00000000 fd:00 8447                       /lib/libnss_files-2.7.so
7f6337449000-7f6337649000 ---p 0000a000 fd:00 8447                       /lib/libnss_files-2.7.so
7f6337649000-7f633764b000 rw-p 0000a000 fd:00 8447                       /lib/libnss_files-2.7.so
7f633764b000-7f633764f000 r-xp 00000000 fd:02 73832                      /usr/lib/rsyslog/lmnet.so
7f633764f000-7f633784f000 ---p 00004000 fd:02 73832                      /usr/lib/rsyslog/lmnet.so
7f633784f000-7f6337850000 rw-p 00004000 fd:02 73832                      /usr/lib/rsyslog/lmnet.so
7f6337850000-7f633799a000 r-xp 00000000 fd:00 8459                       /lib/libc-2.7.so
7f633799a000-7f6337b99000 ---p 0014a000 fd:00 8459                       /lib/libc-2.7.so
7f6337b99000-7f6337b9c000 r--p 00149000 fd:00 8459                       /lib/libc-2.7.so
7f6337b9c000-7f6337b9e000 rw-p 0014c000 fd:00 8459                       /lib/libc-2.7.so
7f6337b9e000-7f6337ba3000 rw-p 7f6337b9e000 00:00 0 
7f6337ba3000-7f6337bab000 r-xp 00000000 fd:00 8498                       /lib/librt-2.7.so
7f6337bab000-7f6337daa000 ---p 00008000 fd:00 8498                       /lib/librt-2.7.so
7f6337daa000-7f6337dac000 rw-p 00007000 fd:00 8498                       /lib/librt-2.7.so
7f6337dac000-7f6337dae000 r-xp 00000000 fd:00 8496                       /lib/libdl-2.7.so
7f6337dae000-7f6337fae000 ---p 00002000 fd:00 8496                       /lib/libdl-2.7.so
7f6337fae000-7f6337fb0000 rw-p 00002000 fd:00 8496                       /lib/libdl-2.7.so
7f6337fb0000-7f6337fc6000 r-xp 00000000 fd:00 8442                       /lib/libpthread-2.7.so
7f6337fc6000-7f63381c6000 ---p 00016000 fd:00 8442                       /lib/libpthread-2.7.so
7f63381c6000-7f63381c8000 rw-p 00016000 fd:00 8442                       /lib/libpthread-2.7.so
7f63381c8000-7f63381cc000 rw-p 7f63381c8000 00:00 0 
7f63381cc000-7f63381e2000 r-xp 00000000 fd:02 58672                      /usr/lib/libz.so.1.2.3.3
7f63381e2000-7f63383e2000 ---p 00016000 fd:02 58672                      /usr/lib/libz.so.1.2.3.3
7f63383e2000-7f63383e3000 rw-p 00016000 fd:02 58672                      /usr/lib/libz.so.1.2.3.3
7f63383e3000-7f63383ff000 r-xp 00000000 fd:00 8495                       /lib/ld-2.7.so
7f63385f3000-7f63385f6000 rw-p 7f63385f3000 00:00 0 
7f63385fa000-7f63385fe000 rw-p 7f63385fa000 00:00 0 
7f63385fe000-7f6338600000 rw-p 0001b000 fd:00 8495                       /lib/ld-2.7.so
7fff405ea000-7fff405ff000 rw-p 7ffffffea000 00:00 0                      [stack]
7fff405ff000-7fff40600000 r-xp 7fff405ff000 00:00 0                      [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]



Here's my rsyslog.conf and remote logging configuration in 
/etc/rsyslog.d/remote-logging.conf:

*** rsyslog.conf
#  /etc/rsyslog.conf	Configuration file for rsyslog v3.
#
#			For more information see 
#			/usr/share/doc/rsyslog-doc/html/rsyslog_conf.html


#################
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd) 
#$ModLoad immark  # provides --MARK-- message capability

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use default timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf


###############
#### RULES ####
###############

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*			/var/log/auth.log
*.*;auth,authpriv.none		-/var/log/syslog
#cron.*				/var/log/cron.log
daemon.*			-/var/log/daemon.log
kern.*				-/var/log/kern.log
lpr.*				-/var/log/lpr.log
mail.*				-/var/log/mail.log
user.*				-/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info			-/var/log/mail.info
mail.warn			-/var/log/mail.warn
mail.err			/var/log/mail.err

#
# Logging for INN news system.
#
news.crit			/var/log/news/news.crit
news.err			/var/log/news/news.err
news.notice			-/var/log/news/news.notice

#
# Some "catch-all" log files.
#
*.=debug;\
	auth,authpriv.none;\
	news.none;mail.none	-/var/log/debug
*.=info;*.=notice;*.=warn;\
	auth,authpriv.none;\
	cron,daemon.none;\
	mail,news.none		-/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg				*

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
#	news.=crit;news.=err;news.=notice;\
#	*.=debug;*.=info;\
#	*.=notice;*.=warn	/dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
# you must invoke `xconsole' with the `-file' option:
# 
#    $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
#      busy site..
#
daemon.*;mail.*;\
	news.err;\
	*.=debug;*.=info;\
	*.=notice;*.=warn	|/dev/xconsole

*** remote-logging.conf
$WorkDirectory /var/log/work  # default location for work (spool) files

$ActionQueueType LinkedList   # use asynchronous processing
$ActionQueueFileName srvrfwd  # set file name, also enables disk mode
$ActionResumeRetryCount -1    # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.*       @@hostname.removed:514


-- System Information:
Debian Release: 5.0
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.26-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=fi_FI.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages rsyslog depends on:
ii  libc6                  2.7-16            GNU C Library: Shared libraries
ii  lsb-base               3.2-20            Linux Standard Base 3.2 init scrip
ii  zlib1g                 1:1.2.3.3.dfsg-12 compression library - runtime

Versions of packages rsyslog recommends:
ii  logrotate                     3.7.1-5    Log rotation utility

Versions of packages rsyslog suggests:
pn  rsyslog-doc                   <none>     (no description available)
pn  rsyslog-mysql | rsyslog-pgsql <none>     (no description available)

-- no debconf information

Message #10 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Juha Koho <jmcsa00@gmail.com>, 509292@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Sun, 21 Dec 2008 13:45:52 +0100

[Message part 1 (text/plain, inline)]

Juha Koho wrote:
> Package: rsyslog
> Version: 3.18.5-1
> Severity: critical
> Justification: causes serious data loss
> 
> Hello,
> 
> I have noticed that rsyslog randomly crashes with remote logging 
> (client) enabled. This happens almost once a day and all log messages 
> are lost until rsyslog is started again. I don't know if this is related 
> to remote logging but all my other installations with no remote logging 
> work fine.
> 
> Normally there is nothing special in the log files when rsyslog crashes 
> and I have not been able to find out the cause. Today I noticed the 
> following:
> 

Could you try to run rsyslog in debug mode:
rsyslogd -c 3 -d


> *** remote-logging.conf
> $WorkDirectory /var/log/work  # default location for work (spool) files
> 
> $ActionQueueType LinkedList   # use asynchronous processing
> $ActionQueueFileName srvrfwd  # set file name, also enables disk mode
> $ActionResumeRetryCount -1    # infinite retries on insert failure
> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down

Do you still get the crashes, if you comment out the $Action* and $WorkDirectory
directives?

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #15 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Juha Koho" <jmcsa00@gmail.com>

To: 509292@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Tue, 23 Dec 2008 11:15:38 +0200

Michael Biebl <biebl@debian.org> wrote:
> Could you try to run rsyslog in debug mode:
> rsyslogd -c 3 -d

I have been running rsyslog in debug mode for a couple of days now
without problems.  But... this morning my remote logging server had
crashed which has never happened before. Server has the latest version
(3.18.6-2) of rsyslog. Again there was nothing in log files. I will
try to run the server in debug mode also.

> Do you still get the crashes, if you comment out the $Action* and $WorkDirectory
> directives?

I have not yet tested this. I will try to get some debugging
information in the debug mode first.

Regards,
Juha

Message #20 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Juha Koho" <jmcsa00@gmail.com>

To: 509292@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Wed, 24 Dec 2008 06:07:10 +0200

[Message part 1 (text/plain, inline)]

Hello,

now my remote logging server had crashed while doing rsyslogd reload
in a daily cron job. Client has been running fine since I started
running it in debug mode. I'll attach the debugging output just before
the crash.

Those $Action* and $WorkDirectory directives are not used in the
server so I assume the client would crash as well with or without
these directives.

Regards,
Juha

[rsyslog.log (text/x-log, attachment)]

Message #25 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Juha Koho" <jmcsa00@gmail.com>

To: 509292@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Mon, 29 Dec 2008 11:11:40 +0200

[Message part 1 (text/plain, inline)]

Hello,

here's some more information regarding this bug. I had been running my
client which had previously crashed almost once a day for several days
in debug mode. Suddenly I noticed that rsyslog was running but no log
entries were written to any log files. I started to investigate the
case and I noticed that rsyslog process was running but it was in some
kind of deadlock state and it was doing nothing. I think this happened
also doing rsyslog reload from a cron job.

I attached the last lines of debug output when I terminated the
process by pressing ctrl-c several times. Too bad that I didn't
realize to check the memory usage of the process.

I have also noticed that if I put $AllowedSender directive in config
file allowing my local subnet to send log messages to the server the
server after a while (say next day or something) somehow forgets this
directive and doesn't allow any client to send log messages. It just
writes log entries like "TCP message from disallowed sender ...
discarded" although this sender has been allowed using the
$AllowedSender directive. I tried using both host names and IP
addresses.

Syntax of the directive I used was simply:

$AllowedSender TCP, 127.0.0.0/8, 192.168.1.0/24

Regards,
Juha

[rsyslog.log (text/x-log, attachment)]

Message #30 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Juha Koho <jmcsa00@gmail.com>, 509292@bugs.debian.org

Cc: Rainer Gerhards <rgerhards@adiscon.com>

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Mon, 29 Dec 2008 14:25:03 +0100

[Message part 1 (text/plain, inline)]

Juha Koho wrote:
> Hello,
> 
> here's some more information regarding this bug. I had been running my
> client which had previously crashed almost once a day for several days
> in debug mode. Suddenly I noticed that rsyslog was running but no log
> entries were written to any log files. I started to investigate the
> case and I noticed that rsyslog process was running but it was in some
> kind of deadlock state and it was doing nothing. I think this happened
> also doing rsyslog reload from a cron job.
> 
> I attached the last lines of debug output when I terminated the
> process by pressing ctrl-c several times. Too bad that I didn't
> realize to check the memory usage of the process.

Could you please build rsyslog with --enable-rtinst to get a more complete debug
output.
I've put Rainer, the upstream author on CC, please keep the CC on replies.


> 
> I have also noticed that if I put $AllowedSender directive in config
> file allowing my local subnet to send log messages to the server the
> server after a while (say next day or something) somehow forgets this
> directive and doesn't allow any client to send log messages. It just
> writes log entries like "TCP message from disallowed sender ...
> discarded" although this sender has been allowed using the
> $AllowedSender directive. I tried using both host names and IP
> addresses.
> 
> Syntax of the directive I used was simply:
> 
> $AllowedSender TCP, 127.0.0.0/8, 192.168.1.0/24

Thats a different issue I think, very likely related to

2627.182158676:action 73 queue[DA]:Reg/w0:  log-server:514/tcp
2627.182451825:action 73 queue[DA]:Reg/w0: create tcp connection failed, reason
Connection refusedno working socket could be obtainederror forwarding via tcp,
suspending

Maybe another process already blocking the tcp 514 port.
Anyways, as this is a separate issue, we should not discuss this in this bug
report (but in a separate bug report, if you can reproduce this).


Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #35 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Juha Koho <jmcsa00@gmail.com>, 509292@bugs.debian.org

Cc: Rainer Gerhards <rgerhards@adiscon.com>

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Mon, 29 Dec 2008 14:28:10 +0100

[Message part 1 (text/plain, inline)]

Michael Biebl wrote:

> I've put Rainer, the upstream author on CC, please keep the CC on replies.

Rainer,

I forgot to mention, as you missed the initial bug report, that you can find the
complete log in the BTS at

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=509292

Cheers,
Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #40 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Juha Koho" <jmcsa00@gmail.com>

To: "Michael Biebl" <biebl@debian.org>

Cc: 509292@bugs.debian.org, "Rainer Gerhards" <rgerhards@adiscon.com>

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Tue, 30 Dec 2008 09:36:26 +0200

Hello,

something strange happened now with my remote logging server after
compiling rsyslog with --enable-rtinst and running it in debug mode.
During last night the server had crashed completely; DHCP server
failed to give IP addresses and I was unable to log in via SSH or
access NFS shares. I have no monitor attached to the server so I was
unable to see what was going on. I had to power it down and restart.
After restart everything was working fine.

This happened again when logrotate runs rsyslogd reload since last
(and only) message in syslog was "kernel: imklog 3.18.6, log source =
/proc/kmsg started.". There was no other log messages although the
server had been running for 6 hours before I restarted it.

Too bad that I lost my debugging output because I was storing this
output to tmp and restart deleted the file. Could it be that rsyslog
was filling up my tmp (5.3G) with debug output log and that had killed
my server?

Regards,
Juha

Message #45 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Juha Koho" <jmcsa00@gmail.com>

To: "Michael Biebl" <biebl@debian.org>

Cc: 509292@bugs.debian.org, "Rainer Gerhards" <rgerhards@adiscon.com>

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Tue, 30 Dec 2008 10:43:37 +0200

[Message part 1 (text/plain, inline)]

Hello again,

now I manually ran rsyslogd reload in the server and I managed to
crash rsyslog and here's the last lines of the debug output. Hopefully
this helps. This happened when I just ran "/etc/init.d/rsyslogd
reload".

Regards,
Juha

[rsyslog.log (text/x-log, attachment)]

Message #50 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Juha Koho <jmcsa00@gmail.com>, 509292@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Sat, 03 Jan 2009 21:22:44 +0100

[Message part 1 (text/plain, inline)]

Juha Koho wrote:
> Hello again,
> 
> now I manually ran rsyslogd reload in the server and I managed to
> crash rsyslog and here's the last lines of the debug output. Hopefully
> this helps. This happened when I just ran "/etc/init.d/rsyslogd
> reload".

Hi Juha,

I can reproduce this segfault when I use the $AllowedSender directive in
rsyslog.conf and reload rsyslogd.
rsyslog is reloaded daily via the cron job, which might explain your regular
crashes.

Could you please verify, that you no longer get the crashes, if you remove the
$AllowedSender directive, so I can be sure that we are on the same track.

Cheers,
Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #55 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Juha Koho" <jmcsa00@gmail.com>

To: "Michael Biebl" <biebl@debian.org>

Cc: 509292@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Sun, 4 Jan 2009 10:09:49 +0200

On Sat, Jan 3, 2009 at 10:22 PM, Michael Biebl <biebl@debian.org> wrote:
> I can reproduce this segfault when I use the $AllowedSender directive in
> rsyslog.conf and reload rsyslogd.
> rsyslog is reloaded daily via the cron job, which might explain your regular
> crashes.
>
> Could you please verify, that you no longer get the crashes, if you remove the
> $AllowedSender directive, so I can be sure that we are on the same track.

Hello Michael,

yes I can confirm that you are on the same track. I removed this
directive because of the problem I mentioned earlier and now my server
has been up since that.

However the original reason for this bug report was that my remote
logging client crashed randomly. Sometimes during normal operation and
not during reload and of course my client does not have the
AllowedSender directive. But for some reason my client has been
running fine since I recompiled rsyslog with more debugging output
(--enable-rtinst).

Regards,
Juha

Message #60 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Juha Koho <jmcsa00@gmail.com>, 509292@bugs.debian.org

Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Sat, 10 Jan 2009 07:14:28 +0100

[Message part 1 (text/plain, inline)]

Juha Koho schrieb:
> On Sat, Jan 3, 2009 at 10:22 PM, Michael Biebl <biebl@debian.org> wrote:
>> I can reproduce this segfault when I use the $AllowedSender directive in
>> rsyslog.conf and reload rsyslogd.
>> rsyslog is reloaded daily via the cron job, which might explain your regular
>> crashes.
>>
>> Could you please verify, that you no longer get the crashes, if you remove the
>> $AllowedSender directive, so I can be sure that we are on the same track.
> 
> Hello Michael,
> 
> yes I can confirm that you are on the same track. I removed this
> directive because of the problem I mentioned earlier and now my server
> has been up since that.
> 
> However the original reason for this bug report was that my remote
> logging client crashed randomly. Sometimes during normal operation and
> not during reload and of course my client does not have the
> AllowedSender directive. But for some reason my client has been
> running fine since I recompiled rsyslog with more debugging output
> (--enable-rtinst).

So we have two issues here:

1.) The segfault on reload (triggered by cron) when using
$AllowedSender. The following forum entry is very likely about the same
issue [1].
This issue is easily reproducible (with 3.18.6 from testing/unstable)

2.) Random crashes in the client when forwarding messages over UDP.
I can't reproduce this problem myself and it's not yet clear, what is
causing this crashes.

Nevertheless, I think we should split up the bug report and deal with
1.) and 2.) separately as they don't seem to be related from what I can see.
What's a it troubling is though, is that you can't reproduce 2.) anymore
with --enable-rtinst. This makes me suspect that it is a timing/race
condition.


Cheers,
Michael


[1]http://kb.monitorware.com/viewtopic.php?f=34&t=8857

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #65 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Juha Koho" <jmcsa00@gmail.com>

To: "Michael Biebl" <biebl@debian.org>

Cc: 509292@bugs.debian.org, "Rainer Gerhards" <rgerhards@hq.adiscon.com>

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Sat, 10 Jan 2009 20:12:51 +0200

On Sat, Jan 10, 2009 at 8:14 AM, Michael Biebl <biebl@debian.org> wrote:
> So we have two issues here:
>
> 1.) The segfault on reload (triggered by cron) when using
> $AllowedSender. The following forum entry is very likely about the same
> issue [1].
> This issue is easily reproducible (with 3.18.6 from testing/unstable)
>
> 2.) Random crashes in the client when forwarding messages over UDP.
> I can't reproduce this problem myself and it's not yet clear, what is
> causing this crashes.
>
> Nevertheless, I think we should split up the bug report and deal with
> 1.) and 2.) separately as they don't seem to be related from what I can see.
> What's a it troubling is though, is that you can't reproduce 2.) anymore
> with --enable-rtinst. This makes me suspect that it is a timing/race
> condition.

Yes I agree with you on this. Do you want me to create a new bug
report for issue number 1. or will you file it by yourself?

For this issue (number 2.) I believe it could be a thread
synchronization issue. The client that has had these problems is a
quad core system and I installed other single core system with exactly
the same configuration not running the recompiled version and it has
been working perfectly since I installed it for at least a week ago. I
don't think these issues are related either because my client used to
crash at random times and not during reload.

By the way. I'm actually using TCP to forward messages and I haven't
tried UDP yet.

Regards,
Juha

Message #70 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Juha Koho <jmcsa00@gmail.com>, 509292@bugs.debian.org, Rainer Gerhards <rgerhards@adiscon.com>

Cc: control@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Mon, 12 Jan 2009 09:55:41 +0100

[Message part 1 (text/plain, inline)]

clone 509292 -1
retitle -1 rsyslog: segfault on reload when using $AllowedSender
thanks

Juha Koho wrote:
> On Sat, Jan 10, 2009 at 8:14 AM, Michael Biebl <biebl@debian.org> wrote:
>> So we have two issues here:
>>
>> 1.) The segfault on reload (triggered by cron) when using
>> $AllowedSender. The following forum entry is very likely about the same
>> issue [1].
>> This issue is easily reproducible (with 3.18.6 from testing/unstable)
>>
>> 2.) Random crashes in the client when forwarding messages over UDP.
>> I can't reproduce this problem myself and it's not yet clear, what is
>> causing this crashes.
>>
>> Nevertheless, I think we should split up the bug report and deal with
>> 1.) and 2.) separately as they don't seem to be related from what I can see.
>> What's a it troubling is though, is that you can't reproduce 2.) anymore
>> with --enable-rtinst. This makes me suspect that it is a timing/race
>> condition.
> 
> Yes I agree with you on this. Do you want me to create a new bug
> report for issue number 1. or will you file it by yourself?

I cloned the bug and retitled it accordingly for 1.), so we have the bug log and
you don't need to open a new bug report.

Cheers,
Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #77 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Rainer Gerhards <rgerhards@hq.adiscon.com>

To: Juha Koho <jmcsa00@gmail.com>

Cc: Michael Biebl <biebl@debian.org>, 509292@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Tue, 13 Jan 2009 04:12:26 +0100

Juha,

I have finally been able to review the material that came with this bug
report. Thanks for all the good info, but it looks everything was
related to the $AllowedSender bug, not to the race condition (which I,
too, think exists).

... more inline below...

On Sat, 2009-01-10 at 20:12 +0200, Juha Koho wrote:
> For this issue (number 2.) I believe it could be a thread
> synchronization issue. The client that has had these problems is a
> quad core system and I installed other single core system with exactly
> the same configuration not running the recompiled version and it has
> been working perfectly since I installed it for at least a week ago.

Definitely. I am trying to track down a nasty race condition (I think it
is one) for a while now. It seems to occur only on machines with at
least four cores and not always. I unfortunately can not reproduced it
myself. This partly due to insufficient hardware, but when I got a
machine for a while, I was able to see the issue only once or twice, but
very, very random and I could not draw any conclusion before I needed to
return the machine. There are few other reports, but for none of them I
have been able to obtain any information that points to the culprit. I
hope we can make better success in your case.

>  I
> don't think these issues are related either because my client used to
> crash at random times and not during reload.

Right, this one is different.

> 
> By the way. I'm actually using TCP to forward messages and I haven't
> tried UDP yet.

This doesn't seem to make a difference. I think I have tracked down it
to either the code that creates or destructs the message object, but not
being able to reliably reproduce, this is just an educated guess. So the
input may make a difference (but I don't think so).

The primary question I have at this time is if you can reproduce the bug
without the $AllowedSender directive (or with the patch I created for
the cloned bug). If so, that would be a very good thing. From there, we
would need to change the config to see if it disappears if some settings
are changed (I am a bit sceptic about the async queue). That than could
lead us to the right path, even when not being able to apply any debug
settings. Oh - did I mention that the bug almost instantly disappears if
rsyslog is compiled for debugging. I initially thought that is an
artifact of limited concurrency due to debug calls, but now I tend to
believe that it actually is due to reduced speed - so on a 8-core system
we may have the issue even with debug mode (someone with a 8 way system
out there? ;)).

I guess the bug is quite basic, but it is very hard to find it not being
able to reproduce it at will or at least once a day and in debug mode...

Feedback appreciated,
Rainer

Message #82 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Juha Koho" <jmcsa00@gmail.com>

To: "Rainer Gerhards" <rgerhards@hq.adiscon.com>

Cc: "Michael Biebl" <biebl@debian.org>, 509292@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Wed, 14 Jan 2009 07:56:22 +0200

On Tue, Jan 13, 2009 at 5:12 AM, Rainer Gerhards
<rgerhards@hq.adiscon.com> wrote:
> The primary question I have at this time is if you can reproduce the bug
> without the $AllowedSender directive (or with the patch I created for
> the cloned bug). If so, that would be a very good thing. From there, we

Hello Rainer,

when I filed this bug report I didn't yet use the $AllowedSender
directive in the server. I started to use it later on and then I
discovered the reload problem. So I'm sure that this problem persists
even when not using the $AllowedSender. And also note that this bug
report actually applies to my _remote logging client_ and not my
server (server is a dual core system). The $AllowedSender segfaults
occured in my server (naturally I don't use this directive in the
client...).

> would need to change the config to see if it disappears if some settings
> are changed (I am a bit sceptic about the async queue). That than could
> lead us to the right path, even when not being able to apply any debug
> settings. Oh - did I mention that the bug almost instantly disappears if
> rsyslog is compiled for debugging. I initially thought that is an
> artifact of limited concurrency due to debug calls, but now I tend to
> believe that it actually is due to reduced speed - so on a 8-core system
> we may have the issue even with debug mode (someone with a 8 way system
> out there? ;)).

I have been now running the recompiled version without -d option for
few days and still no problems. Seems that when rsyslog is compiled
with --enable-rtinst this problem disappears or at least happens very
rarely. I think I need to revert back to the original version to
reproduce this bug.

Do you have any ideas what to try next?

Regards,
Juha

Message #87 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Rainer Gerhards <rgerhards@hq.adiscon.com>, 509292@bugs.debian.org, Juha Koho <jmcsa00@gmail.com>

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Wed, 14 Jan 2009 08:40:56 +0100

[Message part 1 (text/plain, inline)]

Rainer Gerhards wrote:
> Juha,
> 
> I have finally been able to review the material that came with this bug
> report. Thanks for all the good info, but it looks everything was
> related to the $AllowedSender bug, not to the race condition (which I,
> too, think exists).
> 
> ... more inline below...
> 
> On Sat, 2009-01-10 at 20:12 +0200, Juha Koho wrote:
>> For this issue (number 2.) I believe it could be a thread
>> synchronization issue. The client that has had these problems is a
>> quad core system and I installed other single core system with exactly
>> the same configuration not running the recompiled version and it has
>> been working perfectly since I installed it for at least a week ago.
> 
> Definitely. I am trying to track down a nasty race condition (I think it
> is one) for a while now. It seems to occur only on machines with at
> least four cores and not always. I unfortunately can not reproduced it
> myself. This partly due to insufficient hardware, but when I got a
> machine for a while, I was able to see the issue only once or twice, but
> very, very random and I could not draw any conclusion before I needed to
> return the machine. There are few other reports, but for none of them I
> have been able to obtain any information that points to the culprit. I
> hope we can make better success in your case.
> 
>>  I
>> don't think these issues are related either because my client used to
>> crash at random times and not during reload.
> 
> Right, this one is different.
> 
>> By the way. I'm actually using TCP to forward messages and I haven't
>> tried UDP yet.
> 
> This doesn't seem to make a difference. I think I have tracked down it
> to either the code that creates or destructs the message object, but not
> being able to reliably reproduce, this is just an educated guess. So the
> input may make a difference (but I don't think so).
> 
> The primary question I have at this time is if you can reproduce the bug
> without the $AllowedSender directive (or with the patch I created for
> the cloned bug). If so, that would be a very good thing. From there, we

I don't think the $AllowedSender directive has any influence on the crashes Juho
experiences on his rsyslog clients (as he only used those directive on the
rsyslog server).
Why do you suspect that the $AllowedSender fix might have an influence on this?


> would need to change the config to see if it disappears if some settings
> are changed (I am a bit sceptic about the async queue). That than could
> lead us to the right path, even when not being able to apply any debug
> settings. Oh - did I mention that the bug almost instantly disappears if
> rsyslog is compiled for debugging. I initially thought that is an
> artifact of limited concurrency due to debug calls, but now I tend to
> believe that it actually is due to reduced speed - so on a 8-core system
> we may have the issue even with debug mode (someone with a 8 way system
> out there? ;)).
> 
> I guess the bug is quite basic, but it is very hard to find it not being
> able to reproduce it at will or at least once a day and in debug mode...

So, we have the $AllowedSender issue on the server (tracked as #511562), where
Rainer has provided a patch and the fix seems to work for me (waiting for Juho
to confirm this).


Then we have the random crashes on the client (tracked as #509292). Juho's
rsyslog.conf is at [1]. The only clue so far is, that it is related to multi
core machines (>= 4 cores). I'm not convinced that it is related to remote logging.
Juho, could you strip down your rsyslog.conf step by step (i.e. first remove the
remote logging, then the $ActionQueue* directives, then the imklog plugin, then
the different rules), which will help us to narrow down this bug.

Cheers,
Michael


[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=509292#5
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #92 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Juha Koho" <jmcsa00@gmail.com>

To: "Michael Biebl" <biebl@debian.org>

Cc: "Rainer Gerhards" <rgerhards@hq.adiscon.com>, 509292@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Wed, 14 Jan 2009 10:13:29 +0200

On Wed, Jan 14, 2009 at 9:40 AM, Michael Biebl <biebl@debian.org> wrote:
> Then we have the random crashes on the client (tracked as #509292). Juho's
> rsyslog.conf is at [1]. The only clue so far is, that it is related to multi
> core machines (>= 4 cores). I'm not convinced that it is related to remote logging.
> Juho, could you strip down your rsyslog.conf step by step (i.e. first remove the
> remote logging, then the $ActionQueue* directives, then the imklog plugin, then
> the different rules), which will help us to narrow down this bug.

Actually it's Juha... ;)

Well anyway, I reverted back to the original version without
additional debugging information and I removed remote logging. Let's
see if I still get those crashes.

Regards,
Juha

Message #97 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Rainer Gerhards <rgerhards@hq.adiscon.com>

To: Michael Biebl <biebl@debian.org>

Cc: 509292@bugs.debian.org, Juha Koho <jmcsa00@gmail.com>

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Tue, 13 Jan 2009 07:03:41 +0100

On Wed, 2009-01-14 at 08:40 +0100, Michael Biebl wrote:
> I don't think the $AllowedSender directive has any influence on the crashes Juho
> experiences on his rsyslog clients (as he only used those directive on the
> rsyslog server).
> Why do you suspect that the $AllowedSender fix might have an influence on this?

I don't suspect it has - I was wrong ;) I think I was too-focussed on
the quad core issue that I didn't even read the config correctly. I
thought the crashes happened on the server. I now need to reevaluate the
material - or maybe it is simpler if I just follow what happens now.

> Then we have the random crashes on the client (tracked as #509292). Juho's
> rsyslog.conf is at [1]. The only clue so far is, that it is related to multi
> core machines (>= 4 cores). I'm not convinced that it is related to remote logging.
> Juho, could you strip down your rsyslog.conf step by step (i.e. first remove the
> remote logging, then the $ActionQueue* directives, then the imklog plugin, then
> the different rules), which will help us to narrow down this bug.

I'd actually start with removing the $ActionQueue* directives, as they
cause additional asynchronizity. But the important thing is it first get
back to a state where we can (somewhat) reliably reproduce the bug. I
think we should see at least 3 aborts in a row before we do any changes.
The reason is that we otherwise do not know if the config changed caused
the situation to improve or that was just a random non-abort. That's the
issue I have run into all time so far...

Rainer

Message #104 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Juha Koho <jmcsa00@gmail.com>

To: Rainer Gerhards <rgerhards@hq.adiscon.com>

Cc: Michael Biebl <biebl@debian.org>, 509292@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Sat, 24 Jan 2009 13:35:53 +0200

On Tue, Jan 13, 2009 at 8:03 AM, Rainer Gerhards
<rgerhards@hq.adiscon.com> wrote:
> I'd actually start with removing the $ActionQueue* directives, as they
> cause additional asynchronizity. But the important thing is it first get
> back to a state where we can (somewhat) reliably reproduce the bug. I
> think we should see at least 3 aborts in a row before we do any changes.
> The reason is that we otherwise do not know if the config changed caused
> the situation to improve or that was just a random non-abort. That's the
> issue I have run into all time so far...

Hello all,

now this bug gets interesting... It has been over a week now since I
reverted back to the original rsyslog version with remote logging
activated and configuration file as it was originally when I had these
crashes. Well... now there has been no problems although originally I
had these crashes almost once a day. So currently I have been unable
to reproduce this bug anymore.

Could this bug had been caused by a bug in a library that rsyslog uses
that would have been updated in the past week or so?

Regards,
Juha

Message #109 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Juha Koho <jmcsa00@gmail.com>, 509292@bugs.debian.org

Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Sat, 24 Jan 2009 14:06:39 +0100

[Message part 1 (text/plain, inline)]

Juha Koho wrote:

> 
> Could this bug had been caused by a bug in a library that rsyslog uses
> that would have been updated in the past week or so?
> 

Rainer suspected atomic operations to be the root cause. This would mean a
problem in GCC at compile time.
rsyslog basically depends on zlib1g and libc6. I don't expect zlib1g to have any
influence, so you might wanna check if libc6 has been updated on you system.

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #114 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Juha Koho <jmcsa00@gmail.com>

To: Michael Biebl <biebl@debian.org>

Cc: 509292@bugs.debian.org, Rainer Gerhards <rgerhards@hq.adiscon.com>

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Sun, 25 Jan 2009 11:32:09 +0200

[Message part 1 (text/plain, inline)]

On Sat, Jan 24, 2009 at 3:06 PM, Michael Biebl <biebl@debian.org> wrote:
> Rainer suspected atomic operations to be the root cause. This would mean a
> problem in GCC at compile time.
> rsyslog basically depends on zlib1g and libc6. I don't expect zlib1g to have any
> influence, so you might wanna check if libc6 has been updated on you system.

Just when I said I have had no problems... last night (not during cron
job) rsyslog had crashed in my client (last lines of syslog attacted).

So I can still reproduce this bug. Do you want to be sure and wait for
another crash or do you want me to start be removing the $ActionQueue*
directives from configuration?

Regards,
Juha

[syslog (application/octet-stream, attachment)]

Message #119 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Rainer Gerhards" <rgerhards@hq.adiscon.com>

To: "Juha Koho" <jmcsa00@gmail.com>, "Michael Biebl" <biebl@debian.org>

Cc: <509292@bugs.debian.org>

Subject: RE: Bug#509292: rsyslog: random crashes with remote logging

Date: Sun, 25 Jan 2009 10:46:56 +0100

Juha,

that's the problem with this bug, it is hard to reproduce. However, I
made some progress recently and finally am able to reproduce it in my
lab. I am currently posting updates on the progress via twitter at
http://twitter.com/rgerhards in case you'd like to follow - and
eventually add to - what I am doing (It's kind of problematic to
broadcast all the details...).

So in short, I don't suggest anything at this stage. I am pretty
confident, however, that removing the $ActionQueue mode will solve the
issue for the time being, but of course at the cost of reduced
functionality.

Rainer

> -----Original Message-----
> From: Juha Koho [mailto:jmcsa00@gmail.com]
> Sent: Sunday, January 25, 2009 10:32 AM
> To: Michael Biebl
> Cc: 509292@bugs.debian.org; Rainer Gerhards
> Subject: Re: Bug#509292: rsyslog: random crashes with remote logging
> 
> On Sat, Jan 24, 2009 at 3:06 PM, Michael Biebl <biebl@debian.org>
> wrote:
> > Rainer suspected atomic operations to be the root cause. This would
> mean a
> > problem in GCC at compile time.
> > rsyslog basically depends on zlib1g and libc6. I don't expect zlib1g
> to have any
> > influence, so you might wanna check if libc6 has been updated on you
> system.
> 
> Just when I said I have had no problems... last night (not during cron
> job) rsyslog had crashed in my client (last lines of syslog attacted).
> 
> So I can still reproduce this bug. Do you want to be sure and wait for
> another crash or do you want me to start be removing the $ActionQueue*
> directives from configuration?
> 
> Regards,
> Juha

Message #124 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Rainer Gerhards" <rgerhards@hq.adiscon.com>

To: "Juha Koho" <jmcsa00@gmail.com>, "Michael Biebl" <biebl@debian.org>

Cc: <509292@bugs.debian.org>

Subject: RE: Bug#509292: rsyslog: random crashes with remote logging

Date: Tue, 27 Jan 2009 14:34:28 +0100

Is there any chance we could try this with the current v3-stable? Or,
better yet, with the current v4? The reason I ask is that I have run
some valgrind/DRD tests today, and that reminded me that 3.18 had a
couple of "not so nice" sync primitive handlings. They should not be the
issue, but it is pretty hard to use valgrind on that version. So I'll
focus troubleshooting on v4. It may be that I will not be able to
backport a fix, once I find it (but it is still too early to think about
the details ... let's find it first ;)).

Rainer

> -----Original Message-----
> From: Juha Koho [mailto:jmcsa00@gmail.com]
> Sent: Sunday, January 25, 2009 10:32 AM
> To: Michael Biebl
> Cc: 509292@bugs.debian.org; Rainer Gerhards
> Subject: Re: Bug#509292: rsyslog: random crashes with remote logging
> 
> On Sat, Jan 24, 2009 at 3:06 PM, Michael Biebl <biebl@debian.org>
> wrote:
> > Rainer suspected atomic operations to be the root cause. This would
> mean a
> > problem in GCC at compile time.
> > rsyslog basically depends on zlib1g and libc6. I don't expect zlib1g
> to have any
> > influence, so you might wanna check if libc6 has been updated on you
> system.
> 
> Just when I said I have had no problems... last night (not during cron
> job) rsyslog had crashed in my client (last lines of syslog attacted).
> 
> So I can still reproduce this bug. Do you want to be sure and wait for
> another crash or do you want me to start be removing the $ActionQueue*
> directives from configuration?
> 
> Regards,
> Juha

Message #129 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Rainer Gerhards <rgerhards@hq.adiscon.com>

Cc: Juha Koho <jmcsa00@gmail.com>, 509292@bugs.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Tue, 27 Jan 2009 15:03:06 +0100

[Message part 1 (text/plain, inline)]

Rainer Gerhards wrote:
> Is there any chance we could try this with the current v3-stable? Or,
> better yet, with the current v4? The reason I ask is that I have run
> some valgrind/DRD tests today, and that reminded me that 3.18 had a
> couple of "not so nice" sync primitive handlings. They should not be the
> issue, but it is pretty hard to use valgrind on that version. So I'll
> focus troubleshooting on v4. It may be that I will not be able to
> backport a fix, once I find it (but it is still too early to think about
> the details ... let's find it first ;)).
> 

Latest v3-stable (3.20.3) is available from experimental [1].
I could try to provide unofficial Debian packages for v4 for Juha, if that helps
(but I currently only have a i386 to build these packages)

Cheers,
Michael

[1] http://packages.debian.org/experimental/rsyslog
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #134 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Rainer Gerhards" <rgerhards@hq.adiscon.com>

To: "Michael Biebl" <biebl@debian.org>

Cc: "Juha Koho" <jmcsa00@gmail.com>, <509292@bugs.debian.org>

Subject: RE: Bug#509292: rsyslog: random crashes with remote logging

Date: Wed, 28 Jan 2009 12:59:40 +0100

OK, thanks to the help of Lorenzo Catucci, I was able to pinpoint one
problem that can cause a race. I will write up the details soon, but
this requires some time ;) I think I have a fix for the debian_lenny
branch, you can pull it from git and also find it on its gitweb:

http://git.adiscon.com/?p=rsyslog.git;a=commit;h=35673b12c42429786f6229f
f9fcef7001a6b21ab

No other branch has been patched so far.

Rainer

> -----Original Message-----
> From: Michael Biebl [mailto:biebl@debian.org]
> Sent: Tuesday, January 27, 2009 3:03 PM
> To: Rainer Gerhards
> Cc: Juha Koho; 509292@bugs.debian.org
> Subject: Re: Bug#509292: rsyslog: random crashes with remote logging
> 
> Rainer Gerhards wrote:
> > Is there any chance we could try this with the current v3-stable?
Or,
> > better yet, with the current v4? The reason I ask is that I have run
> > some valgrind/DRD tests today, and that reminded me that 3.18 had a
> > couple of "not so nice" sync primitive handlings. They should not be
> the
> > issue, but it is pretty hard to use valgrind on that version. So
I'll
> > focus troubleshooting on v4. It may be that I will not be able to
> > backport a fix, once I find it (but it is still too early to think
> about
> > the details ... let's find it first ;)).
> >
> 
> Latest v3-stable (3.20.3) is available from experimental [1].
> I could try to provide unofficial Debian packages for v4 for Juha, if
> that helps
> (but I currently only have a i386 to build these packages)
> 
> Cheers,
> Michael
> 
> [1] http://packages.debian.org/experimental/rsyslog
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?

Message #139 received at 509292@bugs.debian.org (full text, mbox, reply):

From: "Rainer Gerhards" <rgerhards@hq.adiscon.com>

To: "Rainer Gerhards" <rgerhards@hq.adiscon.com>, <509292@bugs.debian.org>, "Michael Biebl" <biebl@debian.org>

Cc: "Juha Koho" <jmcsa00@gmail.com>

Subject: RE: Bug#509292: rsyslog: random crashes with remote logging

Date: Wed, 28 Jan 2009 13:10:39 +0100

Sorry, there was a copy&paste error in the commit. It is corrected now.
Please pull the latest version from that git branch. 

> -----Original Message-----
> From: Rainer Gerhards [mailto:rgerhards@hq.adiscon.com]
> Sent: Wednesday, January 28, 2009 1:00 PM
> To: Michael Biebl
> Cc: Juha Koho; 509292@bugs.debian.org
> Subject: Bug#509292: rsyslog: random crashes with remote logging
> 
> OK, thanks to the help of Lorenzo Catucci, I was able to pinpoint one
> problem that can cause a race. I will write up the details soon, but
> this requires some time ;) I think I have a fix for the debian_lenny
> branch, you can pull it from git and also find it on its gitweb:
> 
>
http://git.adiscon.com/?p=rsyslog.git;a=commit;h=35673b12c42429786f6229
> f
> f9fcef7001a6b21ab
> 
> No other branch has been patched so far.
> 
> Rainer
> 
> > -----Original Message-----
> > From: Michael Biebl [mailto:biebl@debian.org]
> > Sent: Tuesday, January 27, 2009 3:03 PM
> > To: Rainer Gerhards
> > Cc: Juha Koho; 509292@bugs.debian.org
> > Subject: Re: Bug#509292: rsyslog: random crashes with remote logging
> >
> > Rainer Gerhards wrote:
> > > Is there any chance we could try this with the current v3-stable?
> Or,
> > > better yet, with the current v4? The reason I ask is that I have
> run
> > > some valgrind/DRD tests today, and that reminded me that 3.18 had
a
> > > couple of "not so nice" sync primitive handlings. They should not
> be
> > the
> > > issue, but it is pretty hard to use valgrind on that version. So
> I'll
> > > focus troubleshooting on v4. It may be that I will not be able to
> > > backport a fix, once I find it (but it is still too early to think
> > about
> > > the details ... let's find it first ;)).
> > >
> >
> > Latest v3-stable (3.20.3) is available from experimental [1].
> > I could try to provide unofficial Debian packages for v4 for Juha,
if
> > that helps
> > (but I currently only have a i386 to build these packages)
> >
> > Cheers,
> > Michael
> >
> > [1] http://packages.debian.org/experimental/rsyslog
> > --
> > Why is it that all of the instruments seeking intelligent life in
the
> > universe are pointed away from Earth?
> 
> 
> 

Message #144 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Rainer Gerhards <rgerhards@hq.adiscon.com>

Cc: 509292@bugs.debian.org, Juha Koho <jmcsa00@gmail.com>

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Thu, 29 Jan 2009 01:30:46 +0100

[Message part 1 (text/plain, inline)]

Rainer Gerhards wrote:
> Sorry, there was a copy&paste error in the commit. It is corrected now.
> Please pull the latest version from that git branch. 
> 
>> -----Original Message-----
>> From: Rainer Gerhards [mailto:rgerhards@hq.adiscon.com]
>> Sent: Wednesday, January 28, 2009 1:00 PM
>> To: Michael Biebl
>> Cc: Juha Koho; 509292@bugs.debian.org
>> Subject: Bug#509292: rsyslog: random crashes with remote logging
>>
>> OK, thanks to the help of Lorenzo Catucci, I was able to pinpoint one
>> problem that can cause a race. I will write up the details soon, but
>> this requires some time ;) I think I have a fix for the debian_lenny
>> branch, you can pull it from git and also find it on its gitweb:
>>

Juha, I've prepared packages which include this fix. They are available from
[1], but currently i386 only. If you need to recompile it, the easiest way is

# apt-get install build-essential devscripts
# apt-get build-dep rsyslog
# dget -x http://debs.michaelbiebl.de/rsyslog/rsyslog_3.18.6-4~1.dsc
# cd rsyslog-3.18.6
# dpkg-buildpackage -us -uc

Cheers,
Michael

[1] http://debs.michaelbiebl.de/rsyslog/


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #151 received at 509292@bugs.debian.org (full text, mbox, reply):

From: Mark Purcell <msp@debian.org>

To: Michael Biebl <biebl@debian.org>, 509292@bugs.debian.org

Cc: Rainer Gerhards <rgerhards@hq.adiscon.com>, Juha Koho <jmcsa00@gmail.com>, debian-release@lists.debian.org

Subject: Re: Bug#509292: rsyslog: random crashes with remote logging

Date: Sat, 7 Feb 2009 21:02:01 +1100

[Message part 1 (text/plain, inline)]

On Thursday 29 January 2009 11:30:46 Michael Biebl wrote:
> I've prepared packages which include this fix. They are available
> from [1], but currently i386 only

Michael,

Are you intending to upload these packages to unstable?

One week to proposed lenny release and your package is still shown with a RC 
bug.

Mark

[signature.asc (application/pgp-signature, inline)]

Message #156 received at 509292-close@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: 509292-close@bugs.debian.org

Subject: Bug#509292: fixed in rsyslog 3.18.6-4

Date: Sun, 08 Feb 2009 00:32:09 +0000

Source: rsyslog
Source-Version: 3.18.6-4

We believe that the bug you reported is fixed in the latest version of
rsyslog, which is due to be installed in the Debian FTP archive:

rsyslog-doc_3.18.6-4_all.deb
  to pool/main/r/rsyslog/rsyslog-doc_3.18.6-4_all.deb
rsyslog-mysql_3.18.6-4_i386.deb
  to pool/main/r/rsyslog/rsyslog-mysql_3.18.6-4_i386.deb
rsyslog-pgsql_3.18.6-4_i386.deb
  to pool/main/r/rsyslog/rsyslog-pgsql_3.18.6-4_i386.deb
rsyslog_3.18.6-4.diff.gz
  to pool/main/r/rsyslog/rsyslog_3.18.6-4.diff.gz
rsyslog_3.18.6-4.dsc
  to pool/main/r/rsyslog/rsyslog_3.18.6-4.dsc
rsyslog_3.18.6-4_i386.deb
  to pool/main/r/rsyslog/rsyslog_3.18.6-4_i386.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 509292@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Michael Biebl <biebl@debian.org> (supplier of updated rsyslog package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Sun, 08 Feb 2009 00:54:39 +0100
Source: rsyslog
Binary: rsyslog rsyslog-doc rsyslog-mysql rsyslog-pgsql
Architecture: source all i386
Version: 3.18.6-4
Distribution: unstable
Urgency: medium
Maintainer: Michael Biebl <biebl@debian.org>
Changed-By: Michael Biebl <biebl@debian.org>
Description: 
 rsyslog    - enhanced multi-threaded syslogd
 rsyslog-doc - documentation for rsyslog
 rsyslog-mysql - MySQL output plugin for rsyslog
 rsyslog-pgsql - PostgreSQL output plugin for rsyslog
Closes: 484083 509292 512663 513589
Changes: 
 rsyslog (3.18.6-4) unstable; urgency=medium
 .
   * debian/patches/message_locking_fix.patch
     - Proper message locking on message destruct to avoid a race condition
       which could lead to a segfault. Closes: #509292
       Patch cherry-picked from upstream git.
   * Urgency medium for the RC bug fix.
   * Use the dbconfig-common template mechanism to generate the configuration
     files for rsyslog-mysql and rsyslog-pgsql. This not only simplifies
     postinst quite a bit, but also makes sure we don't read any unset debconf
     values. Closes: #513589
   * debian/README.Debian
     - Add notes about the Debian specific configuration of rsyslog and outline
       some of the changes between rsyslog and sysklogd which should ease the
       migration. Closes: #484083
     - Add instructions how to avoid doubled hostname entries when sending
       syslog messages from rsyslog to a sysklogd server. Closes: #512663
Checksums-Sha1: 
 aca07ad712bcd54271d16562a18b22cb31ea1fcb 1227 rsyslog_3.18.6-4.dsc
 3c84e59594e4eeaba7752aee65f6cbdb5fc5f869 21460 rsyslog_3.18.6-4.diff.gz
 7de74f95539adb727f15d80b5c9f16af20b74eee 229796 rsyslog-doc_3.18.6-4_all.deb
 8903d9a9dcb5054e2202b671a389309d9dbb7da4 168696 rsyslog_3.18.6-4_i386.deb
 7ffbc9c5b3155334ac18a2a28ddcdb93acb7ca54 12356 rsyslog-mysql_3.18.6-4_i386.deb
 ab95b13ae8d0064766fc62c4f5e56e619de9294f 12108 rsyslog-pgsql_3.18.6-4_i386.deb
Checksums-Sha256: 
 95626d3136fad95cdcf62aa2324bd4e924df328257c571ad2701225f6c51e404 1227 rsyslog_3.18.6-4.dsc
 624f5b9bdb8cfadf627bd64cb1a9b79cbe346723337c2684a249f05d0f365855 21460 rsyslog_3.18.6-4.diff.gz
 90f1fa86cac33cb650035778d9e875a21cd9f5fdabc8fc29954508ed06f3d072 229796 rsyslog-doc_3.18.6-4_all.deb
 41b82a6c4dd367ff36992d9c62fb72d890ceca2c9eb21c4e8934ce8bdd14167b 168696 rsyslog_3.18.6-4_i386.deb
 25d1dd9588e7752b578ca8199c66f5b7f1d189952c1d5379d7aec629cab4ec08 12356 rsyslog-mysql_3.18.6-4_i386.deb
 4b6a4ec966067f7dd70da5c72bc23245f9eb46245642cd64cf0d843e9b985023 12108 rsyslog-pgsql_3.18.6-4_i386.deb
Files: 
 04d410dc6eaa3ac6c936eafb556fb913 1227 admin important rsyslog_3.18.6-4.dsc
 87165a982b800aac73c68ed9576d9e49 21460 admin important rsyslog_3.18.6-4.diff.gz
 7dd89674856dc252ddc9147bdadb961c 229796 doc extra rsyslog-doc_3.18.6-4_all.deb
 0b7b1ed221713c0765eaa46d516eedb6 168696 admin important rsyslog_3.18.6-4_i386.deb
 1eebc8f7f755e95e358d9d32cc34ae2a 12356 admin extra rsyslog-mysql_3.18.6-4_i386.deb
 9e63e99548a549c6e81cc9fbef8fbf87 12108 admin extra rsyslog-pgsql_3.18.6-4_i386.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkmOJQ4ACgkQh7PER70FhVQfLgCgvFTSGOTJ4do4oYz8Dvw77Qwe
r1AAoLjBTY1h/8HJtDDgQGvVS4nmSAbn
=6mU/
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.