Debian Bug report logs - #600031
updating kernel causes slow write speed to raid

version graph

Package: linux-2.6; Maintainer for linux-2.6 is Debian Kernel Team <debian-kernel@lists.debian.org>;

Reported by: Dave Clarke <dave.p.clarke@gmail.com>

Date: Tue, 12 Oct 2010 22:09:01 UTC

Severity: normal

Fixed in versions 2.6.36~rc6-1~experimental.1, 2.6.36-1

Done: Moritz Muehlenhoff <jmm@inutil.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, unknown-package@qa.debian.org:
Bug#600031; Package linux-image. (Tue, 12 Oct 2010 22:09:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Dave Clarke <dave.p.clarke@gmail.com>:
New Bug report received and forwarded. Copy sent to unknown-package@qa.debian.org. (Tue, 12 Oct 2010 22:09:04 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Dave Clarke <dave.p.clarke@gmail.com>
To: submit@bugs.debian.org
Subject: updating kernel causes slow write speed to raid
Date: Tue, 12 Oct 2010 23:07:46 +0100
Package: linux-image
Version: 2.6.32-5-686

After dist-upgrade from Lenny to Squeeze write speed is halved. Shown
with bonnie++. Before upgrade write speed was 21MB/s, after upgrade
10MB/s.

Hardware: Dell Powervault 745n with hardware raid Adaptec 2610SA.
Celeron(R) CPU 2.40GHz, 512MiB RAM.

Confirmed link to kernel with:

otherserver$ scp 4gbfile raid_server:
   100% 4000MB   6.6MB/s   10:06

reboot server with kernel 2.6.26-2-686 leftover from Lenny.
otherserver$ scp 4gbfile raid_server:
   100% 4000MB  11.3MB/s   05:55

Also of note that when copying a large file the system becomes heavily loaded.
   iowait: 80.1%wa
   load average: 3.58, 1.48, 0.66
   Mem:    514428k total,   508276k used,     6152k free,    10084k buffers




Bug reassigned from package 'linux-image' to 'linux-2.6'. Request was from Martin Michlmayr <tbm@cyrius.com> to control@bugs.debian.org. (Tue, 12 Oct 2010 22:15:05 GMT) Full text and rfc822 format available.

Bug No longer marked as found in versions 2.6.32-5-686. Request was from Martin Michlmayr <tbm@cyrius.com> to control@bugs.debian.org. (Tue, 12 Oct 2010 22:15:05 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#600031; Package linux-2.6. (Tue, 12 Oct 2010 23:18:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Tue, 12 Oct 2010 23:18:03 GMT) Full text and rfc822 format available.

Message #14 received at 600031@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: Dave Clarke <dave.p.clarke@gmail.com>
Cc: 600031@bugs.debian.org
Subject: Re: updating kernel causes slow write speed to raid
Date: Wed, 13 Oct 2010 00:15:08 +0100
[Message part 1 (text/plain, inline)]
On Tue, 2010-10-12 at 23:07 +0100, Dave Clarke wrote:
> Package: linux-image
> Version: 2.6.32-5-686
> 
> After dist-upgrade from Lenny to Squeeze write speed is halved. Shown
> with bonnie++. Before upgrade write speed was 21MB/s, after upgrade
> 10MB/s.

Let's see the output from bonnie++.  Also are you running this locally
or over NFS or other network filesystem?

> Hardware: Dell Powervault 745n with hardware raid Adaptec 2610SA.
> Celeron(R) CPU 2.40GHz, 512MiB RAM.
> 
> Confirmed link to kernel with:
> 
> otherserver$ scp 4gbfile raid_server:
>    100% 4000MB   6.6MB/s   10:06
> 
> reboot server with kernel 2.6.26-2-686 leftover from Lenny.
> otherserver$ scp 4gbfile raid_server:
>    100% 4000MB  11.3MB/s   05:55
> 
> Also of note that when copying a large file the system becomes heavily loaded.
>    iowait: 80.1%wa
>    load average: 3.58, 1.48, 0.66
>    Mem:    514428k total,   508276k used,     6152k free,    10084k buffers

That seems like too little memory for a file server.  Can you send the
contents of /proc/meminfo?

Also what filesystem is used on the RAID?

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
[signature.asc (application/pgp-signature, inline)]

Added tag(s) moreinfo. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Tue, 12 Oct 2010 23:18:04 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#600031; Package linux-2.6. (Wed, 13 Oct 2010 23:33:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Dave Clarke <dave.p.clarke@gmail.com>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Wed, 13 Oct 2010 23:33:06 GMT) Full text and rfc822 format available.

Message #21 received at 600031@bugs.debian.org (full text, mbox):

From: Dave Clarke <dave.p.clarke@gmail.com>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: 600031@bugs.debian.org
Subject: Re: updating kernel causes slow write speed to raid
Date: Thu, 14 Oct 2010 00:31:48 +0100
[Message part 1 (text/plain, inline)]
Running bonnie++ locally. Have formatted / as ext3 and /home/ as ext4.
using raid5. Also to note that when booted with windows server 2003
using Everest disk benchmark it does an average of 34MB/s (peaking to
50MB/s).

Attached are bonnie++ output for the two kernel versions and ext3 and
ext4 (ext4 won't mount with 2.6.26 kernel).
And meminfo for both kernels taken when the system is loaded and unloaded.

Dave

On 13 October 2010 00:15, Ben Hutchings <ben@decadent.org.uk> wrote:
> On Tue, 2010-10-12 at 23:07 +0100, Dave Clarke wrote:
>> Package: linux-image
>> Version: 2.6.32-5-686
>>
>> After dist-upgrade from Lenny to Squeeze write speed is halved. Shown
>> with bonnie++. Before upgrade write speed was 21MB/s, after upgrade
>> 10MB/s.
>
> Let's see the output from bonnie++.  Also are you running this locally
> or over NFS or other network filesystem?
>
>> Hardware: Dell Powervault 745n with hardware raid Adaptec 2610SA.
>> Celeron(R) CPU 2.40GHz, 512MiB RAM.
>>
>> Confirmed link to kernel with:
>>
>> otherserver$ scp 4gbfile raid_server:
>>    100% 4000MB   6.6MB/s   10:06
>>
>> reboot server with kernel 2.6.26-2-686 leftover from Lenny.
>> otherserver$ scp 4gbfile raid_server:
>>    100% 4000MB  11.3MB/s   05:55
>>
>> Also of note that when copying a large file the system becomes heavily loaded.
>>    iowait: 80.1%wa
>>    load average: 3.58, 1.48, 0.66
>>    Mem:    514428k total,   508276k used,     6152k free,    10084k buffers
>
> That seems like too little memory for a file server.  Can you send the
> contents of /proc/meminfo?
>
> Also what filesystem is used on the RAID?
>
> Ben.
>
> --
> Ben Hutchings
> Once a job is fouled up, anything done to improve it makes it worse.
>
[bonnie_2.6.26_ext3.txt (text/plain, attachment)]
[bonnie_2.6.32_ext3.txt (text/plain, attachment)]
[bonnie_2.6.32_ext4.txt (text/plain, attachment)]
[meminfo_2.6.26.txt (text/plain, attachment)]
[meminfo_2.6.32.txt (text/plain, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#600031; Package linux-2.6. (Thu, 14 Oct 2010 02:51:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Thu, 14 Oct 2010 02:51:03 GMT) Full text and rfc822 format available.

Message #26 received at 600031@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: Dave Clarke <dave.p.clarke@gmail.com>
Cc: 600031@bugs.debian.org
Subject: Re: updating kernel causes slow write speed to raid
Date: Thu, 14 Oct 2010 03:46:54 +0100
[Message part 1 (text/plain, inline)]
On Thu, 2010-10-14 at 00:31 +0100, Dave Clarke wrote:
> Running bonnie++ locally. Have formatted / as ext3 and /home/ as ext4.
> using raid5. Also to note that when booted with windows server 2003
> using Everest disk benchmark it does an average of 34MB/s (peaking to
> 50MB/s).
> 
> Attached are bonnie++ output for the two kernel versions and ext3 and
> ext4 (ext4 won't mount with 2.6.26 kernel).
> And meminfo for both kernels taken when the system is loaded and unloaded.

OK, it looks like it's not so short of memory.  And this is really quite
poor write performance.

Could you also test Linux 2.6.36-rc6, as packaged in experimental?

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#600031; Package linux-2.6. (Thu, 14 Oct 2010 14:18:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Dave Clarke <dave.p.clarke@gmail.com>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Thu, 14 Oct 2010 14:18:05 GMT) Full text and rfc822 format available.

Message #31 received at 600031@bugs.debian.org (full text, mbox):

From: Dave Clarke <dave.p.clarke@gmail.com>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: 600031@bugs.debian.org
Subject: Re: updating kernel causes slow write speed to raid
Date: Thu, 14 Oct 2010 15:16:36 +0100
[Message part 1 (text/plain, inline)]
Ben
That has solved it. Now getting around 27MB/s write on ext4 and 97MB/s
read with the experimental kernel. I'm happy with these speeds on this
hardware. It is still a little slower then with windows but I can live
with that.

May be able to get better by tuning read-ahead buffer and
vm.dirty_ratio. Do you think there is any merit in adjusting these?

Thanks
Dave

On 14 October 2010 03:46, Ben Hutchings <ben@decadent.org.uk> wrote:
> On Thu, 2010-10-14 at 00:31 +0100, Dave Clarke wrote:
>> Running bonnie++ locally. Have formatted / as ext3 and /home/ as ext4.
>> using raid5. Also to note that when booted with windows server 2003
>> using Everest disk benchmark it does an average of 34MB/s (peaking to
>> 50MB/s).
>>
>> Attached are bonnie++ output for the two kernel versions and ext3 and
>> ext4 (ext4 won't mount with 2.6.26 kernel).
>> And meminfo for both kernels taken when the system is loaded and unloaded.
>
> OK, it looks like it's not so short of memory.  And this is really quite
> poor write performance.
>
> Could you also test Linux 2.6.36-rc6, as packaged in experimental?
>
> Ben.
>
> --
> Ben Hutchings
> Once a job is fouled up, anything done to improve it makes it worse.
>
[meminfo_2.6.36.txt (text/plain, attachment)]
[bonnie_2.6.36rc6_ext3.txt (text/plain, attachment)]
[bonnie_2.6.36rc6_ext4.txt (text/plain, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#600031; Package linux-2.6. (Fri, 15 Oct 2010 01:36:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Fri, 15 Oct 2010 01:36:03 GMT) Full text and rfc822 format available.

Message #36 received at 600031@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: Dave Clarke <dave.p.clarke@gmail.com>
Cc: 600031@bugs.debian.org
Subject: Re: updating kernel causes slow write speed to raid
Date: Fri, 15 Oct 2010 02:33:13 +0100
[Message part 1 (text/plain, inline)]
On Thu, 2010-10-14 at 15:16 +0100, Dave Clarke wrote:
> Ben
> That has solved it. Now getting around 27MB/s write on ext4 and 97MB/s
> read with the experimental kernel. I'm happy with these speeds on this
> hardware. It is still a little slower then with windows but I can live
> with that.

Good, but I'm concerned as to why 2.6.32 is so much worse.  Since that
will be the kernel version in Debian 6.0 I want to make sure it does
perform well.

> May be able to get better by tuning read-ahead buffer and
> vm.dirty_ratio. Do you think there is any merit in adjusting these?

I don't know; this isn't my area of expertise.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
[signature.asc (application/pgp-signature, inline)]

Bug Marked as fixed in versions 2.6.36~rc6-1~experimental.1. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Fri, 15 Oct 2010 01:36:04 GMT) Full text and rfc822 format available.

Removed tag(s) moreinfo. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Fri, 15 Oct 2010 01:45:04 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#600031; Package linux-2.6. (Fri, 15 Oct 2010 11:42:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Dave Clarke <dave.p.clarke@gmail.com>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Fri, 15 Oct 2010 11:42:03 GMT) Full text and rfc822 format available.

Message #45 received at 600031@bugs.debian.org (full text, mbox):

From: Dave Clarke <dave.p.clarke@gmail.com>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: 600031@bugs.debian.org
Subject: Re: updating kernel causes slow write speed to raid
Date: Fri, 15 Oct 2010 12:37:54 +0100
On 15 October 2010 02:33, Ben Hutchings <ben@decadent.org.uk> wrote:
> On Thu, 2010-10-14 at 15:16 +0100, Dave Clarke wrote:
>> Ben
>> That has solved it. Now getting around 27MB/s write on ext4 and 97MB/s
>> read with the experimental kernel. I'm happy with these speeds on this
>> hardware. It is still a little slower then with windows but I can live
>> with that.
>
> Good, but I'm concerned as to why 2.6.32 is so much worse.  Since that
> will be the kernel version in Debian 6.0 I want to make sure it does
> perform well.

I'd be happy to test some fixes to 2.6.32 to find the source of the
problem if there is anything you want me to try.

>> May be able to get better by tuning read-ahead buffer and
>> vm.dirty_ratio. Do you think there is any merit in adjusting these?
>
> I don't know; this isn't my area of expertise.
>
> Ben.
>
> --
> Ben Hutchings
> Once a job is fouled up, anything done to improve it makes it worse.
>
Regards
Dave




Reply sent to Moritz Muehlenhoff <jmm@inutil.org>:
You have taken responsibility. (Fri, 07 Jun 2013 17:54:13 GMT) Full text and rfc822 format available.

Notification sent to Dave Clarke <dave.p.clarke@gmail.com>:
Bug acknowledged by developer. (Fri, 07 Jun 2013 17:54:13 GMT) Full text and rfc822 format available.

Message #50 received at 600031-done@bugs.debian.org (full text, mbox):

From: Moritz Muehlenhoff <jmm@inutil.org>
To: 600031-done@bugs.debian.org
Subject: Re: updating kernel causes slow write speed to raid
Date: Fri, 7 Jun 2013 19:49:13 +0200
Version: 2.6.36-1

On Thu, Oct 14, 2010 at 03:16:36PM +0100, Dave Clarke wrote:
> Ben
> That has solved it. Now getting around 27MB/s write on ext4 and 97MB/s
> read with the experimental kernel. I'm happy with these speeds on this
> hardware. It is still a little slower then with windows but I can live
> with that.

Marking 2.6.36 as fixed.

Cheers,
        Moritz



Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Sat, 06 Jul 2013 07:27:58 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Wed Apr 16 16:50:47 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.