Debian Bug report logs - #462617
debian-installer: need RAID bitmap support (mdadm -b internal)

Package: mdcfg; Maintainer for mdcfg is Debian Install System Team <debian-boot@lists.debian.org>; Source for mdcfg is src:mdcfg.

Reported by: Russell Coker <russell@coker.com.au>

Date: Sat, 26 Jan 2008 05:33:01 UTC

Severity: normal

Tags: d-i, wontfix

Merged with 616723

Done: Cyril Brulebois <kibi@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#462617; Package debian-installer. Full text and rfc822 format available.

Acknowledgement sent to Russell Coker <russell@coker.com.au>:
New Bug report received and forwarded. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Russell Coker <russell@coker.com.au>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: debian-installer: need RAID bitmap support (mdadm -b internal)
Date: Sat, 26 Jan 2008 16:19:16 +1100
Package: debian-installer
Severity: normal

A recent (some time in the last 5 years) feature of Linux software RAID is a
"write intent bitmap".  The purpose of this is that before writing to a
section of disk the bitmap is altered to mark it as dirty.  Then if the
machine experiences a power failure or other catastrophic event then when
rebooted it will know that the section referenced by that bit is dirty.

Thus on reboot after a serious failure small amounts of data will be
synchronised instead of large amounts.

This feature is not currently used by the Debian installer.  So if you
install to a system with multiple disks in a RAID-1 array (and probably
RAID-5 and RAID-6) then every time there is a power failure the disks in
the RAID will have to be completely read to ensure that they match.

If the mdadm command that was used for the Debian installer had "-b internal"
appended then the bitmap feature would be used and recovery from some failure
conditions would be much faster.  There does not seem to be any down-side to
using this.

I have not tested this with RAID-5 and RAID-6, but have tested the benefits
of this option for RAID-1 and found it to be very useful.

-- System Information:
Debian Release: 4.0
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-5-686
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to C)




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#462617; Package debian-installer. Full text and rfc822 format available.

Acknowledgement sent to Otavio Salvador <otavio@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #10 received at 462617@bugs.debian.org (full text, mbox):

From: Otavio Salvador <otavio@debian.org>
To: Russell Coker <russell@coker.com.au>
Cc: 462617@bugs.debian.org, control@bugs.debian.org, madduck@debian.org
Subject: Re: Bug#462617: debian-installer: need RAID bitmap support (mdadm -b internal)
Date: Sat, 26 Jan 2008 09:34:23 -0200
reassign 462617 mdcfg
thanks

Russell Coker <russell@coker.com.au> writes:

> Package: debian-installer
> Severity: normal
>
> A recent (some time in the last 5 years) feature of Linux software RAID is a
> "write intent bitmap".  The purpose of this is that before writing to a
> section of disk the bitmap is altered to mark it as dirty.  Then if the
> machine experiences a power failure or other catastrophic event then when
> rebooted it will know that the section referenced by that bit is dirty.
>
> Thus on reboot after a serious failure small amounts of data will be
> synchronised instead of large amounts.
>
> This feature is not currently used by the Debian installer.  So if you
> install to a system with multiple disks in a RAID-1 array (and probably
> RAID-5 and RAID-6) then every time there is a power failure the disks in
> the RAID will have to be completely read to ensure that they match.
>
> If the mdadm command that was used for the Debian installer had "-b internal"
> appended then the bitmap feature would be used and recovery from some failure
> conditions would be much faster.  There does not seem to be any down-side to
> using this.
>
> I have not tested this with RAID-5 and RAID-6, but have tested the benefits
> of this option for RAID-1 and found it to be very useful.

I've reassign this bug to mdcfg that's where it needs to be
change and added Martin (madduck) as Cc since he's the maintainer of
mdadm on Debian.

Please Martin, could you comment on that?

-- 
        O T A V I O    S A L V A D O R
---------------------------------------------
 E-mail: otavio@debian.org      UIN: 5906116
 GNU/Linux User: 239058     GPG ID: 49A5F855
 Home Page: http://otavio.ossystems.com.br
---------------------------------------------
"Microsoft sells you Windows ... Linux gives
 you the whole house."




Bug reassigned from package `debian-installer' to `mdcfg'. Request was from Otavio Salvador <otavio@debian.org> to control@bugs.debian.org. (Sat, 26 Jan 2008 11:39:05 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#462617; Package mdcfg. Full text and rfc822 format available.

Message #15 received at 462617@bugs.debian.org (full text, mbox):

From: martin f krafft <madduck@debian.org>
To: Otavio Salvador <otavio@debian.org>
Cc: 462617@bugs.debian.org
Subject: Re: Bug#462617: debian-installer: need RAID bitmap support (mdadm -b internal)
Date: Sat, 26 Jan 2008 18:44:59 +0700
[Message part 1 (text/plain, inline)]
also sprach Otavio Salvador <otavio@debian.org> [2008.01.26.1834 +0700]:
> Please Martin, could you comment on that?

I have: http://lists.debian.org/debian-devel/2008/01/msg00929.html

-- 
 .''`.   martin f. krafft <madduck@debian.org>
: :'  :  proud Debian developer, author, administrator, and user
`. `'`   http://people.debian.org/~madduck - http://debiansystem.info
  `-  Debian - when you have better things to do than fixing systems
 
save the plankton - eat a whale.
[digital_signature_gpg.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#462617; Package mdcfg. Full text and rfc822 format available.

Acknowledgement sent to Jérémy Bobbio <lunar@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #20 received at 462617@bugs.debian.org (full text, mbox):

From: Jérémy Bobbio <lunar@debian.org>
To: 462617@bugs.debian.org
Subject: Re: Bug#462617: debian-installer: need RAID bitmap support (mdadm -b internal)
Date: Tue, 5 Feb 2008 17:34:27 +0100
[Message part 1 (text/plain, inline)]
On Sat, Jan 26, 2008 at 06:44:59PM +0700, martin f krafft wrote:
> also sprach Otavio Salvador <otavio@debian.org> [2008.01.26.1834 +0700]:
> > Please Martin, could you comment on that?
> 
> I have: http://lists.debian.org/debian-devel/2008/01/msg00929.html

Joerg Jaspert did some tests to measure the impact of write
intent bitmaps on performances.  Quoting his blog entry: [1]  

   So, having turned them on I just waited to see what happens with the
   machines under normal load. Not doing any stresstests, as they don’t
   show normal load situations anyway. And a little performance impact
   was expected. A little, not the thing that happened…

   I’m impressed, haven’t seen that much IO wait ever before. Even the
   old packages.debian.org code, which was really bad IO-wise (thanks to
   multiple greps per search), didn’t have such a bad impact. I mean - a
   Dual Core Athlon 3800, running a few Xen domains, usually without any
   noticable load, now having a load of 20 and more, while IO wait is at
   some 90% (as shown by top for both cores), constantly. (2 RAID1
   devices).

   Or a Desktop System with a RAID5 (mad of 4 SATA discs), which
   normally unpacks a kernel tarball in some 10 seconds - now needs
   about 2 minutes to get it unpacked. Removing that tree now uses 5
   minutes.

   No, bitmap internal is nothing one ever wants to use. One possibly
   might want to use that feature with a file for the bitmaps which is
   stored on a different device. That might actually make sense and wont
   hurt as much as internal does.

   I have one positive thing to say about them: Resyncing an array is
   indeed much much faster. But I go with longer resync times and
   slightly less performance during that time, the cost of having the
   bitmaps all the time massively impacts, a recovery of even a day
   doesn’t cost as much in total. 

I would be in favor of trusting Joerg's comments and tagging this bug as
"wontfix" in order to document the issue.  Any other opinions?

[1] http://blog.ganneff.de/blog/2008/01/30#mdraid_bitmap_internal_bad

Cheers,
-- 
Jérémy Bobbio                        .''`. 
lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
                                    `. `'` 
                                      `-   
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#462617; Package mdcfg. Full text and rfc822 format available.

Acknowledgement sent to Frans Pop <elendil@planet.nl>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #25 received at 462617@bugs.debian.org (full text, mbox):

From: Frans Pop <elendil@planet.nl>
To: 462617@bugs.debian.org
Subject: Re: Bug#462617: debian-installer: need RAID bitmap support (mdadm -b internal)
Date: Tue, 5 Feb 2008 18:07:41 +0100
On Tuesday 05 February 2008, Jérémy Bobbio wrote:
> I would be in favor of trusting Joerg's comments and tagging this bug as
> "wontfix" in order to document the issue.  Any other opinions?

Sounds fine.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#462617; Package mdcfg. Full text and rfc822 format available.

Acknowledgement sent to Otavio Salvador <otavio@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #30 received at 462617@bugs.debian.org (full text, mbox):

From: Otavio Salvador <otavio@debian.org>
To: 462617@bugs.debian.org
Subject: Re: Bug#462617: debian-installer: need RAID bitmap support (mdadm -b internal)
Date: Tue, 05 Feb 2008 15:45:24 -0200
Jérémy Bobbio <lunar@debian.org> writes:

> I would be in favor of trusting Joerg's comments and tagging this bug as
> "wontfix" in order to document the issue.  Any other opinions?
>
> [1] http://blog.ganneff.de/blog/2008/01/30#mdraid_bitmap_internal_bad

It would be interesting to contact upstream and check with them if
it's the expect behaviour or not. I trust Joerg comments, for sure,
but I also think we ought listen upstream and see if it needs
something special in setup or something to get a great performance or
not.

Otherwise, I ack to tag it wontfix.

-- 
        O T A V I O    S A L V A D O R
---------------------------------------------
 E-mail: otavio@debian.org      UIN: 5906116
 GNU/Linux User: 239058     GPG ID: 49A5F855
 Home Page: http://otavio.ossystems.com.br
---------------------------------------------
"Microsoft sells you Windows ... Linux gives
 you the whole house."




Tags added: wontfix Request was from Jérémy Bobbio <lunar@debian.org> to control@bugs.debian.org. (Tue, 12 Feb 2008 15:30:11 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#462617; Package mdcfg. Full text and rfc822 format available.

Acknowledgement sent to martin f krafft <madduck@madduck.net>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #37 received at 462617@bugs.debian.org (full text, mbox):

From: martin f krafft <madduck@madduck.net>
To: Neil Brown <neilb@suse.de>
Cc: linux-raid mailing list <linux-raid@vger.kernel.org>, 462617@bugs.debian.org
Subject: Re: [patch 3/4] fastboot: make the raid autodetect code wait for all devices to init
Date: Tue, 22 Jul 2008 12:36:05 +0200
[Message part 1 (text/plain, inline)]
[462617@bugs.debian.org added to Cc]

also sprach Neil Brown <neilb@suse.de> [2008.07.22.0124 +0200]:
> > It was my understanding that write-intent bitmaps slow down all
> > operations and are not suggested on e.g. workstations. No?
> 
> Well, they don't slow down reads.
> If you have a separate root filesystem (i.e. /home and /var are
> elsewhere), it is likely to be read-mostly, so bitmaps probably won't
> hurt much.
> And an external bitmap on a dedicated device has minimal performance
> cost.
> 
> However I neither suggest having nor not-having bitmaps.  The choice to
> use them involves a trade-off which I cannot make for other people.

Ok, in the light of http://bugs.debian.org/462617, this answers my
question and Debian will not make them default. It would be nice to
have the option though, during installation.

Including the rest of your reply for the sake of the bug record,
which is on Cc:

> They would, however, be very useful to cover the gap when assembling
> arrays incrementally.
> If, for example, you have a 6 disk raid5 array and 5 disks have been
> found, what do you do?  
>   - wait for the 6th, that might never arrive
>   - start degraded and if a write happens before the 6th disk arrives,
>     have to rebuild the 6th disk completely.
> 
> Neither is a good option.
> An alternate is
>   - add an internal bitmap, and remove it after the 6th disk has
>     arrived, or after we are sure there are no more disks to find.
> 
> Doing this means that if a recovery is needed when the 6th disk
> arrives, it will be very fast.
> 
> It's not hard to notice that the bitmap proposed here does not need to
> be on stable storage.  It is not protecting against a crash, just
> against a window when the array is degraded.  So if we could support
> bitmaps on a tmpfs, we could use an external bitmap in /tmp instead of
> an internal bitmap.
> Or even - we could enhance the md code to always use a bitmap, but
> simply not write it to storage if no such was configured.

This would be very nice. In fact, everything that makes md better
and requires less complexity in the mdadm packaging is nice. :)

> (If a crash happens during that window between writing to the degraded
> array and recovering the few blocks needed on the final device, then
> you would be in an unfortunate position.  For raid1/10 you would just
> need a full resync, which you would have needed anyway, so no loss.
> For raid4/5/6, you have a potential for dataloss, so I probably would
> not make this behaviour the default for those levels...)

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
drink canada dry! you might not succeed, but it *is* fun trying.
 
spamtraps: madduck.bogus@madduck.net
[digital_signature_gpg.asc (application/pgp-signature, inline)]

Merged 462617 616723. Request was from Miguel Figueiredo <elmig@debianpt.org> to control@bugs.debian.org. (Mon, 14 Mar 2011 16:03:08 GMT) Full text and rfc822 format available.

Reply sent to Cyril Brulebois <kibi@debian.org>:
You have taken responsibility. (Sun, 02 Mar 2014 00:57:05 GMT) Full text and rfc822 format available.

Notification sent to Russell Coker <russell@coker.com.au>:
Bug acknowledged by developer. (Sun, 02 Mar 2014 00:57:05 GMT) Full text and rfc822 format available.

Message #44 received at 462617-done@bugs.debian.org (full text, mbox):

From: Cyril Brulebois <kibi@debian.org>
To: Russell Coker <russell@coker.com.au>, 462617-done@bugs.debian.org
Subject: Re: Bug#462617: debian-installer: need RAID bitmap support (mdadm -b internal)
Date: Sun, 2 Mar 2014 01:54:46 +0100
[Message part 1 (text/plain, inline)]
Russell Coker <russell@coker.com.au> (2008-01-26):
> A recent (some time in the last 5 years) feature of Linux software
> RAID is a "write intent bitmap".  The purpose of this is that before
> writing to a section of disk the bitmap is altered to mark it as
> dirty.  Then if the machine experiences a power failure or other
> catastrophic event then when rebooted it will know that the section
> referenced by that bit is dirty.
> 
> Thus on reboot after a serious failure small amounts of data will be
> synchronised instead of large amounts.
> 
> This feature is not currently used by the Debian installer.  So if you
> install to a system with multiple disks in a RAID-1 array (and
> probably RAID-5 and RAID-6) then every time there is a power failure
> the disks in the RAID will have to be completely read to ensure that
> they match.
> 
> If the mdadm command that was used for the Debian installer had "-b
> internal" appended then the bitmap feature would be used and recovery
> from some failure conditions would be much faster.  There does not
> seem to be any down-side to using this.
> 
> I have not tested this with RAID-5 and RAID-6, but have tested the
> benefits of this option for RAID-1 and found it to be very useful.

The subject reminded me of some digging I've done lately to answer
someone on #debian-boot. Pasting the IRC log, anonymizing the user
nickname just in case:

2014-02-14 22:11:41[ ....] do the d-i daily's automatically add write intent bitmaps to raid arrays created in the installer?
2014-02-14 22:12:11[ KiBi] I don't know what those are.
2014-02-14 22:12:18[ ....] I think they must, a system I just installed has them. if anyone knows more details I would love to discuss
2014-02-14 22:12:55[ ....] hmm, damn my notes on them aren't public
2014-02-14 22:13:17[ ....] they are a small bitmap file that corresponds to chunks of the raid array
2014-02-14 22:13:33[ ....] before the kernel goes to write to a chunk it sets the bit
2014-02-14 22:13:54[ KiBi] so maybe partman-auto-raid is the package you want to look at?
2014-02-14 22:14:12[ ....] so if a system loses power, when it comes back up it only needs to resync the chunks that were actively being written to
2014-02-14 22:14:15[ ....] really nice
2014-02-14 22:14:36[ KiBi] unless the code for it is elsewhere, maybe on the kernel side?
2014-02-14 22:14:59[ ....] you can store them on other disks too in order to prevent the heads of a HDD needing to seek a lot
2014-02-14 22:15:07[ KiBi] ah, mdadm apparently: https://raid.wiki.kernel.org/index.php/Write-intent_bitmap
2014-02-14 22:15:15[ ....] anyway it's super cool that d-i is doing it now
2014-02-14 22:15:44[ ....] it only did some of them, the larger ones, which is also exactly what you probably want
2014-02-14 22:16:29[ ....] when you set them up you get to pick the chunk size, it's a trade off of recovery time vs how often it needs to update the bitmap
2014-02-14 22:16:50[ ....] so I am curious as to the decisions made by whoever added it
2014-02-14 22:16:53[ KiBi] autoraid-cfg does that:
2014-02-14 22:16:54[ KiBi]                 mdadm --create /dev/md$MD_NUM --auto=yes --force -R -l raid$RAID_TYPE \
2014-02-14 22:16:54[ KiBi]                       -n $DEV_COUNT $MDADM_PARAMS
2014-02-14 22:17:07[ KiBi] (in the partman-auto-raid package)
2014-02-14 22:17:20[ KiBi] might just be some changed default on the mdadm side?
2014-02-14 22:17:28[ ....] hmm, could be
2014-02-14 22:17:29[ KiBi] mjt would probably know
2014-02-14 22:17:43[ ....] since that command line doesn't list it
2014-02-14 22:17:54[ ....] well unless it's in $MDADM_PARAMS
2014-02-14 22:18:12[ KiBi] auto-raidcfg:           MDADM_PARAMS="-x $SPARE_COUNT $RAID_DEVICES $MISSING_DEVICES $SPARE_DEVICES $MISSING_SPARES"
2014-02-14 22:18:12[ KiBi] auto-raidcfg:           MDADM_PARAMS="$RAID_DEVICES"
2014-02-14 22:18:21[ KiBi] doesn't look like it
2014-02-14 22:18:25[ ....] yeah
2014-02-14 22:21:01[ KiBi] not obvious from the mdadm 3.x announces, cloning the repo..
2014-02-14 22:21:22[ KiBi] Author: NeilBrown <neilb@suse.de>
2014-02-14 22:21:22[ KiBi] Date:   Tue Mar 5 10:36:21 2013 +1100
2014-02-14 22:21:22[ KiBi]     Create: default to bitmap=internal for large arrays.
2014-02-14 22:21:36[ KiBi] mdadm-3.2.5-210-g748952f
2014-02-14 22:21:48[ KiBi] sorry, mdadm-3.3-rc1~102
2014-02-14 22:22:05[ KiBi] so the 3.3 release has it, and we got that for free when building d-i against that version?
2014-02-14 22:22:22[ KiBi] [2013-10-25] mdadm 3.3-2 MIGRATED to testing (Britney)
2014-02-14 22:22:57[ KiBi] so since debian-installer 20131211

Closing this bug report accordingly.

Mraw,
KiBi.
[signature.asc (application/pgp-signature, inline)]

Reply sent to Cyril Brulebois <kibi@debian.org>:
You have taken responsibility. (Sun, 02 Mar 2014 00:57:06 GMT) Full text and rfc822 format available.

Notification sent to Pierre Bernhardt <pierre@starcumulus.owl.de>:
Bug acknowledged by developer. (Sun, 02 Mar 2014 00:57:06 GMT) Full text and rfc822 format available.

Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Sun, 30 Mar 2014 07:27:20 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 17 19:15:18 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.