Debian Bug report logs - #702494
malformed messages/gibberish in logs

Package: linux; Maintainer for linux is Debian Kernel Team <debian-kernel@lists.debian.org>;

Reported by: Tomas Pospisek <tpo_deb@sourcepole.ch>

Date: Thu, 7 Mar 2013 11:03:02 UTC

Severity: wishlist

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Michael Biebl <biebl@debian.org>:
Bug#702494; Package rsyslog. (Thu, 07 Mar 2013 11:03:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Tomas Pospisek <tpo_deb@sourcepole.ch>:
New Bug report received and forwarded. Copy sent to Michael Biebl <biebl@debian.org>. (Thu, 07 Mar 2013 11:03:04 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Tomas Pospisek <tpo_deb@sourcepole.ch>
To: submit@bugs.debian.org
Subject: malformed messages/gibberish in logs
Date: Thu, 7 Mar 2013 11:59:26 +0100 (CET)
Package: rsyslog
Version: 5.8.11-2
Severity: important

On two of our machines we are seeing that rsyslog is filling
up the log with undecipherable gibberish:

>From /var/log/kern.log (other logs are also affected such as */syslog):

  Mar  7 03:29:02 brisi kernel: [236975.910574] EXT3-fs (dm-12): recovery complete
  Mar  7 03:37:06 brisi kernel: 9]X- m2 nnnjn<2596Xfd2mtfseiord e<6>[237459.93863] ET-s(m1) sn nenljra96]trnepeiufn d12>7.6 3p_a:lnnecie1
  Mar  7 03:37:06 brisi kernel: [492xr_ndirei3
  Mar  7 03:41:58 brisi kernel: 39]_nnegfed2>50X 2o st>56X 2c e<>[237712.63]jrlst.ottaes55E3sd1:ot lyewhrr tme15 ulti m elsn.34e3p_negfed9 rhnioe eee
  Mar  7 03:45:07 brisi kernel: >3526Xs- eptam

The corresponding output from dmesg:

  [236975.910574] EXT3-fs (dm-12): recovery complete
  [236975.984204] EXT3-fs (dm-12): mounted filesystem with ordered data mode
  [237050.760055] kjournald starting.  Commit interval 5 seconds
  [237050.789205] EXT3-fs (dm-12): using internal journal
  [237050.790426] EXT3-fs (dm-12): mounted filesystem with ordered data mode
  [237459.946661] kjournald starting.  Commit interval 5 seconds
  [237459.953863] EXT3-fs (dm-12): using internal journal
  [237459.965601] ext3_orphan_cleanup: deleting unreferenced inode 311354
  [237459.966597] ext3_orphan_cleanup: deleting unreferenced inode 311342
  [237459.966614] ext3_orphan_cleanup: deleting unreferenced inode 311340
  [237459.966621] ext3_orphan_cleanup: deleting unreferenced inode 311338
  [237459.967046] ext3_orphan_cleanup: deleting unreferenced inode 311298
  [237459.967056] EXT3-fs (dm-12): 5 orphan inodes deleted
  [237459.967662] EXT3-fs (dm-12): recovery complete
  [237460.037427] EXT3-fs (dm-12): mounted filesystem with ordered data mode
  [237712.565835] kjournald starting.  Commit interval 5 seconds
  [237712.596382] EXT3-fs (dm-12): using internal journal
  [237712.596995] EXT3-fs (dm-12): mounted filesystem with ordered data mode

Note that some of the timestamps in the kern.log and in dmesg correspond.

One could think that this is a result of the filesystem breaking, but that
is not the case. The above fsck's are from lvm snapshots that are (in 
theory) not interfering with the "real" FS.

Additionaly we are seing the same symptoms (gibberish in the logs) on 
another machine that doesn't do fscks on lvm snapshots, that is the 
gibberish occursare aparently random times there.

This problem has also been reported to the rsyslog mailing list:
  http://lists.adiscon.net/pipermail/rsyslog/2013-March/031832.html

We have also tested this with rsyslog 7.2.5-1 from experimental (as seen 
inthe ML thread above).

It's weird that this problem hasn't been reported elsewhere yet, since 
we're seeing it on two different, independently set up machines.
*t




Information forwarded to debian-bugs-dist@lists.debian.org, Michael Biebl <biebl@debian.org>:
Bug#702494; Package rsyslog. (Thu, 07 Mar 2013 11:09:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Michael Biebl <email@michaelbiebl.de>:
Extra info received and forwarded to list. Copy sent to Michael Biebl <biebl@debian.org>. (Thu, 07 Mar 2013 11:09:03 GMT) Full text and rfc822 format available.

Message #10 received at 702494@bugs.debian.org (full text, mbox):

From: Michael Biebl <email@michaelbiebl.de>
To: Tomas Pospisek <tpo_deb@sourcepole.ch>,702494@bugs.debian.org
Subject: Re: Bug#702494: malformed messages/gibberish in logs
Date: Thu, 07 Mar 2013 12:08:14 +0100
Is that on virtualised hardware? Does that only happen for kernel messages or also if you run logger foo?



Tomas Pospisek <tpo_deb@sourcepole.ch> schrieb:

>Package: rsyslog
>Version: 5.8.11-2
>Severity: important
>
>On two of our machines we are seeing that rsyslog is filling
>up the log with undecipherable gibberish:
>
>>From /var/log/kern.log (other logs are also affected such as
>*/syslog):
>
>Mar  7 03:29:02 brisi kernel: [236975.910574] EXT3-fs (dm-12): recovery
>complete
>Mar  7 03:37:06 brisi kernel: 9]X- m2 nnnjn<2596Xfd2mtfseiord
>e<6>[237459.93863] ET-s(m1) sn nenljra96]trnepeiufn d12>7.6
>3p_a:lnnecie1
>   Mar  7 03:37:06 brisi kernel: [492xr_ndirei3
>Mar  7 03:41:58 brisi kernel: 39]_nnegfed2>50X 2o st>56X 2c
>e<>[237712.63]jrlst.ottaes55E3sd1:ot lyewhrr tme15 ulti m
>elsn.34e3p_negfed9 rhnioe eee
>   Mar  7 03:45:07 brisi kernel: >3526Xs- eptam
>
>The corresponding output from dmesg:
>
>   [236975.910574] EXT3-fs (dm-12): recovery complete
>[236975.984204] EXT3-fs (dm-12): mounted filesystem with ordered data
>mode
>   [237050.760055] kjournald starting.  Commit interval 5 seconds
>   [237050.789205] EXT3-fs (dm-12): using internal journal
>[237050.790426] EXT3-fs (dm-12): mounted filesystem with ordered data
>mode
>   [237459.946661] kjournald starting.  Commit interval 5 seconds
>   [237459.953863] EXT3-fs (dm-12): using internal journal
>[237459.965601] ext3_orphan_cleanup: deleting unreferenced inode 311354
>[237459.966597] ext3_orphan_cleanup: deleting unreferenced inode 311342
>[237459.966614] ext3_orphan_cleanup: deleting unreferenced inode 311340
>[237459.966621] ext3_orphan_cleanup: deleting unreferenced inode 311338
>[237459.967046] ext3_orphan_cleanup: deleting unreferenced inode 311298
>   [237459.967056] EXT3-fs (dm-12): 5 orphan inodes deleted
>   [237459.967662] EXT3-fs (dm-12): recovery complete
>[237460.037427] EXT3-fs (dm-12): mounted filesystem with ordered data
>mode
>   [237712.565835] kjournald starting.  Commit interval 5 seconds
>   [237712.596382] EXT3-fs (dm-12): using internal journal
>[237712.596995] EXT3-fs (dm-12): mounted filesystem with ordered data
>mode
>
>Note that some of the timestamps in the kern.log and in dmesg
>correspond.
>
>One could think that this is a result of the filesystem breaking, but
>that
>is not the case. The above fsck's are from lvm snapshots that are (in 
>theory) not interfering with the "real" FS.
>
>Additionaly we are seing the same symptoms (gibberish in the logs) on 
>another machine that doesn't do fscks on lvm snapshots, that is the 
>gibberish occursare aparently random times there.
>
>This problem has also been reported to the rsyslog mailing list:
>   http://lists.adiscon.net/pipermail/rsyslog/2013-March/031832.html
>
>We have also tested this with rsyslog 7.2.5-1 from experimental (as
>seen 
>inthe ML thread above).
>
>It's weird that this problem hasn't been reported elsewhere yet, since 
>we're seeing it on two different, independently set up machines.
>*t

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.



Information forwarded to debian-bugs-dist@lists.debian.org, Michael Biebl <biebl@debian.org>:
Bug#702494; Package rsyslog. (Thu, 07 Mar 2013 11:42:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Michael Biebl <email@michaelbiebl.de>:
Extra info received and forwarded to list. Copy sent to Michael Biebl <biebl@debian.org>. (Thu, 07 Mar 2013 11:42:03 GMT) Full text and rfc822 format available.

Message #15 received at 702494@bugs.debian.org (full text, mbox):

From: Michael Biebl <email@michaelbiebl.de>
To: Tomas Pospisek <tpo_deb@sourcepole.ch>,702494@bugs.debian.org
Subject: Re: Bug#702494: malformed messages/gibberish in logs
Date: Thu, 07 Mar 2013 12:38:23 +0100
What kernel are you running? 



Tomas Pospisek <tpo_deb@sourcepole.ch> schrieb:

>Package: rsyslog
>Version: 5.8.11-2
>Severity: important
>
>On two of our machines we are seeing that rsyslog is filling
>up the log with undecipherable gibberish:
>
>>From /var/log/kern.log (other logs are also affected such as
>*/syslog):
>
>Mar  7 03:29:02 brisi kernel: [236975.910574] EXT3-fs (dm-12): recovery
>complete
>Mar  7 03:37:06 brisi kernel: 9]X- m2 nnnjn<2596Xfd2mtfseiord
>e<6>[237459.93863] ET-s(m1) sn nenljra96]trnepeiufn d12>7.6
>3p_a:lnnecie1
>   Mar  7 03:37:06 brisi kernel: [492xr_ndirei3
>Mar  7 03:41:58 brisi kernel: 39]_nnegfed2>50X 2o st>56X 2c
>e<>[237712.63]jrlst.ottaes55E3sd1:ot lyewhrr tme15 ulti m
>elsn.34e3p_negfed9 rhnioe eee
>   Mar  7 03:45:07 brisi kernel: >3526Xs- eptam
>
>The corresponding output from dmesg:
>
>   [236975.910574] EXT3-fs (dm-12): recovery complete
>[236975.984204] EXT3-fs (dm-12): mounted filesystem with ordered data
>mode
>   [237050.760055] kjournald starting.  Commit interval 5 seconds
>   [237050.789205] EXT3-fs (dm-12): using internal journal
>[237050.790426] EXT3-fs (dm-12): mounted filesystem with ordered data
>mode
>   [237459.946661] kjournald starting.  Commit interval 5 seconds
>   [237459.953863] EXT3-fs (dm-12): using internal journal
>[237459.965601] ext3_orphan_cleanup: deleting unreferenced inode 311354
>[237459.966597] ext3_orphan_cleanup: deleting unreferenced inode 311342
>[237459.966614] ext3_orphan_cleanup: deleting unreferenced inode 311340
>[237459.966621] ext3_orphan_cleanup: deleting unreferenced inode 311338
>[237459.967046] ext3_orphan_cleanup: deleting unreferenced inode 311298
>   [237459.967056] EXT3-fs (dm-12): 5 orphan inodes deleted
>   [237459.967662] EXT3-fs (dm-12): recovery complete
>[237460.037427] EXT3-fs (dm-12): mounted filesystem with ordered data
>mode
>   [237712.565835] kjournald starting.  Commit interval 5 seconds
>   [237712.596382] EXT3-fs (dm-12): using internal journal
>[237712.596995] EXT3-fs (dm-12): mounted filesystem with ordered data
>mode
>
>Note that some of the timestamps in the kern.log and in dmesg
>correspond.
>
>One could think that this is a result of the filesystem breaking, but
>that
>is not the case. The above fsck's are from lvm snapshots that are (in 
>theory) not interfering with the "real" FS.
>
>Additionaly we are seing the same symptoms (gibberish in the logs) on 
>another machine that doesn't do fscks on lvm snapshots, that is the 
>gibberish occursare aparently random times there.
>
>This problem has also been reported to the rsyslog mailing list:
>   http://lists.adiscon.net/pipermail/rsyslog/2013-March/031832.html
>
>We have also tested this with rsyslog 7.2.5-1 from experimental (as
>seen 
>inthe ML thread above).
>
>It's weird that this problem hasn't been reported elsewhere yet, since 
>we're seeing it on two different, independently set up machines.
>*t

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.



Information forwarded to debian-bugs-dist@lists.debian.org, Michael Biebl <biebl@debian.org>:
Bug#702494; Package rsyslog. (Thu, 07 Mar 2013 12:12:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Tomas Pospisek <tpo_deb2@sourcepole.ch>:
Extra info received and forwarded to list. Copy sent to Michael Biebl <biebl@debian.org>. (Thu, 07 Mar 2013 12:12:03 GMT) Full text and rfc822 format available.

Message #20 received at 702494@bugs.debian.org (full text, mbox):

From: Tomas Pospisek <tpo_deb2@sourcepole.ch>
To: Michael Biebl <email@michaelbiebl.de>
Cc: 702494@bugs.debian.org
Subject: Re: Bug#702494: malformed messages/gibberish in logs
Date: Thu, 7 Mar 2013 13:07:38 +0100 (CET)
On Thu, 7 Mar 2013, Michael Biebl wrote:

> Is that on virtualised hardware?

No, that's on the metal. However each lxc container is *also* running out 
of the box rsyslog (without any config adaptations), so it might be that 
there's a problem there somewhere with all those rsyslogs getting along. 
However of course each syslog is sitting on its own FS and thus on its 
own log files.

Ad hoc I couldn't tell what access rights those lxc containers have wrt to 
the kernel log feed and how that feed works.

> Does that only happen for kernel messages or also if you run logger foo?

I think I have only seen malformed log entries in connection with kernel 
messages, however it's not only kernel messages that are affected. I am 
getting the impression that rsyslog gets a hickup at some point and then 
is not able to issue coherent log messages until at some point it gets 
into step again and then log messages are coherent again (and then at some 
point they break again etc.).
*t

> Tomas Pospisek <tpo_deb@sourcepole.ch> schrieb:
>
>> Package: rsyslog
>> Version: 5.8.11-2
>> Severity: important
>>
>> On two of our machines we are seeing that rsyslog is filling
>> up the log with undecipherable gibberish:
>>
>>> From /var/log/kern.log (other logs are also affected such as
>> */syslog):
>>
>> Mar  7 03:29:02 brisi kernel: [236975.910574] EXT3-fs (dm-12): recovery
>> complete
>> Mar  7 03:37:06 brisi kernel: 9]X- m2 nnnjn<2596Xfd2mtfseiord
>> e<6>[237459.93863] ET-s(m1) sn nenljra96]trnepeiufn d12>7.6
>> 3p_a:lnnecie1
>>   Mar  7 03:37:06 brisi kernel: [492xr_ndirei3
>> Mar  7 03:41:58 brisi kernel: 39]_nnegfed2>50X 2o st>56X 2c
>> e<>[237712.63]jrlst.ottaes55E3sd1:ot lyewhrr tme15 ulti m
>> elsn.34e3p_negfed9 rhnioe eee
>>   Mar  7 03:45:07 brisi kernel: >3526Xs- eptam
>>
>> The corresponding output from dmesg:
>>
>>   [236975.910574] EXT3-fs (dm-12): recovery complete
>> [236975.984204] EXT3-fs (dm-12): mounted filesystem with ordered data
>> mode
>>   [237050.760055] kjournald starting.  Commit interval 5 seconds
>>   [237050.789205] EXT3-fs (dm-12): using internal journal
>> [237050.790426] EXT3-fs (dm-12): mounted filesystem with ordered data
>> mode
>>   [237459.946661] kjournald starting.  Commit interval 5 seconds
>>   [237459.953863] EXT3-fs (dm-12): using internal journal
>> [237459.965601] ext3_orphan_cleanup: deleting unreferenced inode 311354
>> [237459.966597] ext3_orphan_cleanup: deleting unreferenced inode 311342
>> [237459.966614] ext3_orphan_cleanup: deleting unreferenced inode 311340
>> [237459.966621] ext3_orphan_cleanup: deleting unreferenced inode 311338
>> [237459.967046] ext3_orphan_cleanup: deleting unreferenced inode 311298
>>   [237459.967056] EXT3-fs (dm-12): 5 orphan inodes deleted
>>   [237459.967662] EXT3-fs (dm-12): recovery complete
>> [237460.037427] EXT3-fs (dm-12): mounted filesystem with ordered data
>> mode
>>   [237712.565835] kjournald starting.  Commit interval 5 seconds
>>   [237712.596382] EXT3-fs (dm-12): using internal journal
>> [237712.596995] EXT3-fs (dm-12): mounted filesystem with ordered data
>> mode
>>
>> Note that some of the timestamps in the kern.log and in dmesg
>> correspond.
>>
>> One could think that this is a result of the filesystem breaking, but
>> that
>> is not the case. The above fsck's are from lvm snapshots that are (in
>> theory) not interfering with the "real" FS.
>>
>> Additionaly we are seing the same symptoms (gibberish in the logs) on
>> another machine that doesn't do fscks on lvm snapshots, that is the
>> gibberish occursare aparently random times there.
>>
>> This problem has also been reported to the rsyslog mailing list:
>>   http://lists.adiscon.net/pipermail/rsyslog/2013-March/031832.html
>>
>> We have also tested this with rsyslog 7.2.5-1 from experimental (as
>> seen
>> inthe ML thread above).
>>
>> It's weird that this problem hasn't been reported elsewhere yet, since
>> we're seeing it on two different, independently set up machines.
>> *t
>
> -- 
> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
>



Information forwarded to debian-bugs-dist@lists.debian.org, Michael Biebl <biebl@debian.org>:
Bug#702494; Package rsyslog. (Thu, 07 Mar 2013 12:12:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Tomas Pospisek <tpo_deb2@sourcepole.ch>:
Extra info received and forwarded to list. Copy sent to Michael Biebl <biebl@debian.org>. (Thu, 07 Mar 2013 12:12:05 GMT) Full text and rfc822 format available.

Message #25 received at 702494@bugs.debian.org (full text, mbox):

From: Tomas Pospisek <tpo_deb2@sourcepole.ch>
To: Michael Biebl <email@michaelbiebl.de>
Cc: 702494@bugs.debian.org
Subject: Re: Bug#702494: malformed messages/gibberish in logs
Date: Thu, 7 Mar 2013 13:09:19 +0100 (CET)
On Thu, 7 Mar 2013, Michael Biebl wrote:

> What kernel are you running?

vanilla Debian 3.2.0-3-amd64 and 3.2.0-4-amd64 kernels

> Tomas Pospisek <tpo_deb@sourcepole.ch> schrieb:
>
>> Package: rsyslog
>> Version: 5.8.11-2
>> Severity: important
>>
>> On two of our machines we are seeing that rsyslog is filling
>> up the log with undecipherable gibberish:
>>
>>> From /var/log/kern.log (other logs are also affected such as
>> */syslog):
>>
>> Mar  7 03:29:02 brisi kernel: [236975.910574] EXT3-fs (dm-12): recovery
>> complete
>> Mar  7 03:37:06 brisi kernel: 9]X- m2 nnnjn<2596Xfd2mtfseiord
>> e<6>[237459.93863] ET-s(m1) sn nenljra96]trnepeiufn d12>7.6
>> 3p_a:lnnecie1
>>   Mar  7 03:37:06 brisi kernel: [492xr_ndirei3
>> Mar  7 03:41:58 brisi kernel: 39]_nnegfed2>50X 2o st>56X 2c
>> e<>[237712.63]jrlst.ottaes55E3sd1:ot lyewhrr tme15 ulti m
>> elsn.34e3p_negfed9 rhnioe eee
>>   Mar  7 03:45:07 brisi kernel: >3526Xs- eptam
>>
>> The corresponding output from dmesg:
>>
>>   [236975.910574] EXT3-fs (dm-12): recovery complete
>> [236975.984204] EXT3-fs (dm-12): mounted filesystem with ordered data
>> mode
>>   [237050.760055] kjournald starting.  Commit interval 5 seconds
>>   [237050.789205] EXT3-fs (dm-12): using internal journal
>> [237050.790426] EXT3-fs (dm-12): mounted filesystem with ordered data
>> mode
>>   [237459.946661] kjournald starting.  Commit interval 5 seconds
>>   [237459.953863] EXT3-fs (dm-12): using internal journal
>> [237459.965601] ext3_orphan_cleanup: deleting unreferenced inode 311354
>> [237459.966597] ext3_orphan_cleanup: deleting unreferenced inode 311342
>> [237459.966614] ext3_orphan_cleanup: deleting unreferenced inode 311340
>> [237459.966621] ext3_orphan_cleanup: deleting unreferenced inode 311338
>> [237459.967046] ext3_orphan_cleanup: deleting unreferenced inode 311298
>>   [237459.967056] EXT3-fs (dm-12): 5 orphan inodes deleted
>>   [237459.967662] EXT3-fs (dm-12): recovery complete
>> [237460.037427] EXT3-fs (dm-12): mounted filesystem with ordered data
>> mode
>>   [237712.565835] kjournald starting.  Commit interval 5 seconds
>>   [237712.596382] EXT3-fs (dm-12): using internal journal
>> [237712.596995] EXT3-fs (dm-12): mounted filesystem with ordered data
>> mode
>>
>> Note that some of the timestamps in the kern.log and in dmesg
>> correspond.
>>
>> One could think that this is a result of the filesystem breaking, but
>> that
>> is not the case. The above fsck's are from lvm snapshots that are (in
>> theory) not interfering with the "real" FS.
>>
>> Additionaly we are seing the same symptoms (gibberish in the logs) on
>> another machine that doesn't do fscks on lvm snapshots, that is the
>> gibberish occursare aparently random times there.
>>
>> This problem has also been reported to the rsyslog mailing list:
>>   http://lists.adiscon.net/pipermail/rsyslog/2013-March/031832.html
>>
>> We have also tested this with rsyslog 7.2.5-1 from experimental (as
>> seen
>> inthe ML thread above).
>>
>> It's weird that this problem hasn't been reported elsewhere yet, since
>> we're seeing it on two different, independently set up machines.
>> *t
>
> -- 
> Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
>



Information forwarded to debian-bugs-dist@lists.debian.org, Michael Biebl <biebl@debian.org>:
Bug#702494; Package rsyslog. (Mon, 27 May 2013 20:06:17 GMT) Full text and rfc822 format available.

Acknowledgement sent to Florian Ernst <florian_ernst@gmx.net>:
Extra info received and forwarded to list. Copy sent to Michael Biebl <biebl@debian.org>. (Mon, 27 May 2013 20:06:17 GMT) Full text and rfc822 format available.

Message #30 received at 702494@bugs.debian.org (full text, mbox):

From: Florian Ernst <florian_ernst@gmx.net>
To: Tomas Pospisek <tpo_deb2@sourcepole.ch>, 702494@bugs.debian.org
Subject: Re: Bug#702494: malformed messages/gibberish in logs
Date: Mon, 27 May 2013 22:04:38 +0200
Hello there,

On Thu, Mar 07, 2013 at 01:07:38PM +0100, Tomas Pospisek wrote:
> On Thu, 7 Mar 2013, Michael Biebl wrote:
> 
> >Is that on virtualised hardware?
> 
> No, that's on the metal. However each lxc container is *also*
> running out of the box rsyslog (without any config adaptations), so
> it might be that there's a problem there somewhere with all those
> rsyslogs getting along. However of course each syslog is sitting on
> its own FS and thus on its own log files.
> 
> Ad hoc I couldn't tell what access rights those lxc containers have
> wrt to the kernel log feed and how that feed works.

FWIW, I experienced the same messed-up logs some time ago when running
rsyslog in an LXC guest. I circumvented that by disabling kernel logging
support in the guest as follows:

diff --git a/rsyslog.conf b/rsyslog.conf
index 2a7f9f9..94fff07 100644
--- a/rsyslog.conf
+++ b/rsyslog.conf
@@ -9,7 +9,7 @@
 #################
 
 $ModLoad imuxsock # provides support for local system logging
-$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
+#$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
 #$ModLoad immark  # provides --MARK-- message capability
 
 # provides UDP syslog reception


Cheers,
Flo



Information forwarded to debian-bugs-dist@lists.debian.org, Michael Biebl <biebl@debian.org>:
Bug#702494; Package rsyslog. (Wed, 05 Jun 2013 08:18:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Florian Ernst <florian_ernst@gmx.net>:
Extra info received and forwarded to list. Copy sent to Michael Biebl <biebl@debian.org>. (Wed, 05 Jun 2013 08:18:04 GMT) Full text and rfc822 format available.

Message #35 received at 702494@bugs.debian.org (full text, mbox):

From: Florian Ernst <florian_ernst@gmx.net>
To: Tomas Pospisek <tpo_deb@sourcepole.ch>
Cc: 702494@bugs.debian.org, Michael Biebl <email@michaelbiebl.de>, Michal Fiala <fiala@mfiala.net>, Daniel Baumann <mail@daniel-baumann.ch>
Subject: Re: Bug#702494: malformed messages/gibberish in logs
Date: Wed, 5 Jun 2013 10:15:48 +0200
Hello all,

On Wed, Jun 05, 2013 at 10:01:52AM +0200, Tomas Pospisek wrote:
> [...]
> The basic problem at hand here is that it seems that reading from
> the kernel log facility is *destructive*, so when multiple
> processes, no matter if inside a lxc container or on the host system
> read from the kernel log facility, then there's a race condition on
> the logs coming from the kernel log facility with an undefined
> outcome on who will get what data from the kernel log facility.
> 
> The result is gibberish in kern.log.
> 
> A solution to this problem is to disable reading from the kernel log
> facility inside the guest VMs.
> 
> Optimally this condition would be detected automatically by the VM
> guests which would automatically disable reading from the kernel log
> facility once they detect they are a VM.
> 
> Or we leave it as is and hope that the sysadmin is kowledgeable
> enough to disable it manually.
> 
> What to do?

IMHO per default no guest should have any access to the kernel log
facility of the host at all. I'm not aware, though, which LXC
capabilities are possible / normally set nowadays that could affect
this (the one guest where I experienced this oddity at hand was set up
quite obscurely and thus probably doesn't follow any standards or best
practices).

If the sysadmin desires to have such an access from within a guest they
should be allowed to do so and be given a pointer on how to configure
rsyslog to work as desired.

Cheers,
Flo



Bug 702494 cloned as bug 711170 Request was from Daniel Baumann <daniel.baumann@progress-technologies.net> to control@bugs.debian.org. (Wed, 05 Jun 2013 08:18:07 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Michael Biebl <biebl@debian.org>:
Bug#702494; Package rsyslog. (Wed, 05 Jun 2013 08:24:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Tomas Pospisek <tpo_deb@sourcepole.ch>:
Extra info received and forwarded to list. Copy sent to Michael Biebl <biebl@debian.org>. (Wed, 05 Jun 2013 08:24:04 GMT) Full text and rfc822 format available.

Message #42 received at 702494@bugs.debian.org (full text, mbox):

From: Tomas Pospisek <tpo_deb@sourcepole.ch>
To: Florian Ernst <florian_ernst@gmx.net>
Cc: 702494@bugs.debian.org, Michael Biebl <email@michaelbiebl.de>, Michal Fiala <fiala@mfiala.net>, Daniel Baumann <mail@daniel-baumann.ch>
Subject: Re: Bug#702494: malformed messages/gibberish in logs
Date: Wed, 5 Jun 2013 10:01:52 +0200 (CEST)
(Cc:ing Debian's lxc maintainer)

On Mon, 27 May 2013, Florian Ernst wrote:

> On Thu, Mar 07, 2013 at 01:07:38PM +0100, Tomas Pospisek wrote:
>> On Thu, 7 Mar 2013, Michael Biebl wrote:
>>
>>> Is that on virtualised hardware?
>>
>> No, that's on the metal. However each lxc container is *also*
>> running out of the box rsyslog (without any config adaptations), so
>> it might be that there's a problem there somewhere with all those
>> rsyslogs getting along. However of course each syslog is sitting on
>> its own FS and thus on its own log files.
>>
>> Ad hoc I couldn't tell what access rights those lxc containers have
>> wrt to the kernel log feed and how that feed works.
>
> FWIW, I experienced the same messed-up logs some time ago when running
> rsyslog in an LXC guest. I circumvented that by disabling kernel logging
> support in the guest as follows:
>
> diff --git a/rsyslog.conf b/rsyslog.conf
> index 2a7f9f9..94fff07 100644
> --- a/rsyslog.conf
> +++ b/rsyslog.conf
> @@ -9,7 +9,7 @@
> #################
>
> $ModLoad imuxsock # provides support for local system logging
> -$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
> +#$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
> #$ModLoad immark  # provides --MARK-- message capability
>
> # provides UDP syslog reception

I am not sure what to do about this bug report.

The basic problem at hand here is that it seems that reading from the 
kernel log facility is *destructive*, so when multiple processes, no 
matter if inside a lxc container or on the host system read from the 
kernel log facility, then there's a race condition on the logs coming 
from the kernel log facility with an undefined outcome on who will get 
what data from the kernel log facility.

The result is gibberish in kern.log.

A solution to this problem is to disable reading from the kernel log 
facility inside the guest VMs.

Optimally this condition would be detected automatically by the VM guests 
which would automatically disable reading from the kernel log facility 
once they detect they are a VM.

Or we leave it as is and hope that the sysadmin is kowledgeable enough to 
disable it manually.

What to do?

?

*t



Information forwarded to debian-bugs-dist@lists.debian.org:
Bug#702494; Package rsyslog. (Wed, 05 Jun 2013 08:30:21 GMT) Full text and rfc822 format available.

Acknowledgement sent to Michael Biebl <biebl@debian.org>:
Extra info received and forwarded to list. (Wed, 05 Jun 2013 08:30:21 GMT) Full text and rfc822 format available.

Message #47 received at 702494@bugs.debian.org (full text, mbox):

From: Michael Biebl <biebl@debian.org>
To: Tomas Pospisek <tpo_deb@sourcepole.ch>
Cc: Florian Ernst <florian_ernst@gmx.net>, 702494@bugs.debian.org, Michal Fiala <fiala@mfiala.net>, Daniel Baumann <mail@daniel-baumann.ch>
Subject: Re: Bug#702494: malformed messages/gibberish in logs
Date: Wed, 05 Jun 2013 10:26:01 +0200
[Message part 1 (text/plain, inline)]
Am 05.06.2013 10:01, schrieb Tomas Pospisek:
> (Cc:ing Debian's lxc maintainer)
> 
> On Mon, 27 May 2013, Florian Ernst wrote:
> 
>> On Thu, Mar 07, 2013 at 01:07:38PM +0100, Tomas Pospisek wrote:
>>> On Thu, 7 Mar 2013, Michael Biebl wrote:
>>>
>>>> Is that on virtualised hardware?
>>>
>>> No, that's on the metal. However each lxc container is *also*
>>> running out of the box rsyslog (without any config adaptations), so
>>> it might be that there's a problem there somewhere with all those
>>> rsyslogs getting along. However of course each syslog is sitting on
>>> its own FS and thus on its own log files.
>>>
>>> Ad hoc I couldn't tell what access rights those lxc containers have
>>> wrt to the kernel log feed and how that feed works.
>>
>> FWIW, I experienced the same messed-up logs some time ago when running
>> rsyslog in an LXC guest. I circumvented that by disabling kernel logging
>> support in the guest as follows:
>>
>> diff --git a/rsyslog.conf b/rsyslog.conf
>> index 2a7f9f9..94fff07 100644
>> --- a/rsyslog.conf
>> +++ b/rsyslog.conf
>> @@ -9,7 +9,7 @@
>> #################
>>
>> $ModLoad imuxsock # provides support for local system logging
>> -$ModLoad imklog   # provides kernel logging support (previously done
>> by rklogd)
>> +#$ModLoad imklog   # provides kernel logging support (previously done
>> by rklogd)
>> #$ModLoad immark  # provides --MARK-- message capability
>>
>> # provides UDP syslog reception
> 
> I am not sure what to do about this bug report.
> 
> The basic problem at hand here is that it seems that reading from the
> kernel log facility is *destructive*, so when multiple processes, no
> matter if inside a lxc container or on the host system read from the
> kernel log facility, then there's a race condition on the logs coming
> from the kernel log facility with an undefined outcome on who will get
> what data from the kernel log facility.
> 
> The result is gibberish in kern.log.
> 
> A solution to this problem is to disable reading from the kernel log
> facility inside the guest VMs.
> 
> Optimally this condition would be detected automatically by the VM
> guests which would automatically disable reading from the kernel log
> facility once they detect they are a VM.
> 
> Or we leave it as is and hope that the sysadmin is kowledgeable enough
> to disable it manually.
> 
> What to do?

To me this looks like a bug in lxc if reading from the kernel log inside
the lxc guest messes up the host kernel log.
Shouldn't the kernel log be namespaced/virtualised?

Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Michael Biebl <biebl@debian.org>:
Bug#702494; Package rsyslog. (Wed, 05 Jun 2013 08:30:24 GMT) Full text and rfc822 format available.

Acknowledgement sent to Tomas Pospisek <tpo_deb@sourcepole.ch>:
Extra info received and forwarded to list. Copy sent to Michael Biebl <biebl@debian.org>. (Wed, 05 Jun 2013 08:30:24 GMT) Full text and rfc822 format available.

Message #52 received at 702494@bugs.debian.org (full text, mbox):

From: Tomas Pospisek <tpo_deb@sourcepole.ch>
To: control@bugs.debian.org
Cc: 702494@bugs.debian.org, Michael Biebl <email@michaelbiebl.de>, Michal Fiala <fiala@mfiala.net>, Daniel Baumann <mail@daniel-baumann.ch>, Florian Ernst <florian_ernst@gmx.net>
Subject: lxc vms should by default not be able to read the kernel log facility
Date: Wed, 5 Jun 2013 10:28:32 +0200 (CEST)
reassign 702494 lxc

thanks

On Wed, 5 Jun 2013, Florian Ernst wrote:

> Hello all,
>
> On Wed, Jun 05, 2013 at 10:01:52AM +0200, Tomas Pospisek wrote:
>> [...]
>> The basic problem at hand here is that it seems that reading from
>> the kernel log facility is *destructive*, so when multiple
>> processes, no matter if inside a lxc container or on the host system
>> read from the kernel log facility, then there's a race condition on
>> the logs coming from the kernel log facility with an undefined
>> outcome on who will get what data from the kernel log facility.
>>
>> The result is gibberish in kern.log.
>>
>> A solution to this problem is to disable reading from the kernel log
>> facility inside the guest VMs.
>>
>> Optimally this condition would be detected automatically by the VM
>> guests which would automatically disable reading from the kernel log
>> facility once they detect they are a VM.
>>
>> Or we leave it as is and hope that the sysadmin is kowledgeable
>> enough to disable it manually.
>>
>> What to do?
>
> IMHO per default no guest should have any access to the kernel log
> facility of the host at all. I'm not aware, though, which LXC
> capabilities are possible / normally set nowadays that could affect
> this (the one guest where I experienced this oddity at hand was set up
> quite obscurely and thus probably doesn't follow any standards or best
> practices).
>
> If the sysadmin desires to have such an access from within a guest they
> should be allowed to do so and be given a pointer on how to configure
> rsyslog to work as desired.

I completely agree with you, and thus am reassigning this to the lxc 
package.
*t



Bug reassigned from package 'rsyslog' to 'lxc'. Request was from Tomas Pospisek <tpo_deb@sourcepole.ch> to control@bugs.debian.org. (Wed, 05 Jun 2013 08:54:07 GMT) Full text and rfc822 format available.

No longer marked as found in versions rsyslog/5.8.11-2. Request was from Tomas Pospisek <tpo_deb@sourcepole.ch> to control@bugs.debian.org. (Wed, 05 Jun 2013 08:54:07 GMT) Full text and rfc822 format available.

Bug reassigned from package 'lxc' to 'linux'. Request was from Daniel Baumann <daniel.baumann@progress-technologies.net> to control@bugs.debian.org. (Wed, 05 Jun 2013 09:03:21 GMT) Full text and rfc822 format available.

Severity set to 'wishlist' from 'important' Request was from Daniel Baumann <daniel.baumann@progress-technologies.net> to control@bugs.debian.org. (Wed, 05 Jun 2013 09:03:22 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Sun Apr 20 09:21:07 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.