Debian Bug report logs - #251905
Problems/workarounds for install to root on LVM on RAID

Package: lvmcfg; Maintainer for lvmcfg is Debian Install System Team <debian-boot@lists.debian.org>; Source for lvmcfg is src:lvmcfg.

Reported by: Charles Steinkuehler <charles@steinkuehler.net>

Date: Mon, 31 May 2004 16:48:02 UTC

Severity: normal

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#251905; Package lvmcfg. Full text and rfc822 format available.

Acknowledgement sent to Martin Michlmayr <tbm@cyrius.com>:
New Bug report received and forwarded. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Martin Michlmayr <tbm@cyrius.com>
To: submit@bugs.debian.org
Subject: Problems/workarounds for install to root on LVM on RAID
Date: Mon, 31 May 2004 13:37:44 -0300
Package: lvmcfg

----- Forwarded message from Charles Steinkuehler <charles@steinkuehler.net> -----

From: Charles Steinkuehler <charles@steinkuehler.net>
Subject: Problems/workarounds for install to root on LVM on RAID
Date: Thu, 27 May 2004 11:02:17 -0500
To: debian-boot@lists.debian.org
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5) Gecko/20031007
X-Accept-Language: en-us, en

I'm trying to install testing onto an x86 box with root on LVM on RAID1, 
and I ran into several issues with the latest daily image:
http://cdimage.debian.org/pub/cdimage-testing/sarge_d-i/i386/20040526/sarge-i386-netinst.iso

First, the problems:
--------------------
1) partman won't let you build an LVM on top of a RAID device

2) Kernel install portion of base install fails due to mkinitrd failure
2a) LVM tools missing
2b) Raidtools (actually mdadm) missing
2c) LVM entries in /dev missing on target
2d) initrd defaults to LVM1 instead of LVM2

3) Grub install fails when /boot is on a RAID1 device

Workarounds:
------------
1) partman won't let you build an LVM on top of a RAID device
This is the easiest to fix...just switch over to the console and 
manually run pvcreate, etc. to build the LVM VG(s) & LV(s).  Once 
created, the logical volums are properly recognized by partman, allowing 
them to be formatted, mount points to be set, etc.
------------
2) Kernel install portion of base install fails due to mkinitrd failure
2a) LVM tools missing
2b) Raidtools (actually mdadm) missing
Prior to installing the base system, switch to a console and edit 
/usr/lib/debootstrap/scripts/sarge, adding the following modules to the 
base="..." line near the top of the file:
LVM Packages: binutils libdevmapper1.00 lvm-common lvm2
RAID Packages: mdadm
------------
2c) LVM entries in /dev missing
Switch to a console and copy the entries from the install system to the 
target system:
cp -aR /dev/<vg> /target/dev/
cp -aR /dev/mapper/<vg>* /target/dev/mapper/
------------
2d) initrd defaults to LVM1 instead of LVM2
This is a nasty one, and oddly confusing.  If both LVM2 and LVM1 are 
present (as determined by the existance of appropriate kernel modules), 
LVM1 is selected.  This seems odd, since LVM2 can talk to LVM1 formatted 
volumes, but not the other way around.  Anyway, to fix this I did the 
following:
- Add initrd (plus dash and cramfsprogs dependencies) to base= line in 
sarge script (see 2a & 2b, above) prior to installing base system
- chroot to target system
- Prior to installing kernel, edit /etc/mkinitrd/mkinitrd.conf and set 
root=/dev/hda1 (a dummy entry that allows mkinitrd to complete 
successfully, letting the install continue)
- Install kernel 2.4.26-1-386
- Move (or delete) /lib/modules/2.4.16-1-386/kernel/drivers/md/lvm-mod.o 
so mkinitrd won't find it and default to LVM1
- mount -t proc proc /proc
- verify /etc/mkinitrd/mkinitrd.conf now lists root=probe (kernel 
install apparently causes config mods to be lost)
- run mkinitrd to replace currently broken root=/dev/hda1 image:
  mkinitrd -o /boot/initrd.img-2.4.26-1-386 2.4.26-1-386
- umount /proc
------------
3) Grub install fails when /boot is on a RAID1 device
Apparently, grub is trying to get 'smarter' about devices, but in the 
process is breaking functionality that used to work.  With /boot 
installed on a raid1 device (md0), grub-install and update-grub both 
fail dismally, reporting that /dev/md0 doesn't exist as a BIOS device 
(like that's a news flash!).
The stupid hack to get this to work is:
- Install grub from the debian installer menu (it will fail)
- Switch to a console and edit /boot/grub/device.map, replacing the 
(hd0) /dev/??? entry with (hd0) /dev/md0
- Re-run install grub from the debian installer menu (it will still 
fail, but get farther this time)
- Run update-grub from a console to create /boot/grub/menu.lst
- run grub and install grub to both HDD's:
  root (hd0,0)
  setup (hd0)
  setup (hd1)
- Continue w/o boot loader from the debian installer menu (you manually 
installed grub)

It's worth noting that grub from stable has no problems with /boot on a 
raid device (once installed).  I can see grub-install perhaps failing 
with /boot on a md device, but why in the world does update-grub even 
care where boot is?!?  All it needs to do is edit menu.lst...

NOTE: Your device numbers might be different.  Also, it's typically OK 
to install grub with root set to (hd0,0) on both hd0 and hd1, since if 
hd0 fails, hd1 will usually become hd0 with most BIOS's.

------------
Reboot and cross your fingers!!!!

I can provide additional information, if needed or desired.

-- 
Charles Steinkuehler
charles@steinkuehler.net


-- 
To UNSUBSCRIBE, email to debian-boot-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

----- End forwarded message -----

-- 
Martin Michlmayr
tbm@cyrius.com



Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#251905; Package lvmcfg. Full text and rfc822 format available.

Acknowledgement sent to Martin Michlmayr <tbm@cyrius.com>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #10 received at 251905@bugs.debian.org (full text, mbox):

From: Martin Michlmayr <tbm@cyrius.com>
To: 251905@bugs.debian.org
Cc: charles@steinkuehler.net
Subject: Re: Problems/workarounds for install to root on LVM on RAID
Date: Fri, 18 Jun 2004 19:40:26 +0100
* tbm <tbm@cyrius.com> [2004-05-31 13:37]:
> 1) partman won't let you build an LVM on top of a RAID device

Still there.  This needs changed to libparted - maybe you can look at
this.

> 2) Kernel install portion of base install fails due to mkinitrd failure
> 2a) LVM tools missing
> 2b) Raidtools (actually mdadm) missing
> 2c) LVM entries in /dev missing on target
> 2d) initrd defaults to LVM1 instead of LVM2

Fixed.

> 3) Grub install fails when /boot is on a RAID1 device

Still there.
-- 
Martin Michlmayr
tbm@cyrius.com



Changed Bug submitter from Martin Michlmayr <tbm@cyrius.com> to Charles Steinkuehler <charles@steinkuehler.net>. Request was from Martin Michlmayr <tbm@cyrius.com> to control@bugs.debian.org. Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#251905; Package lvmcfg. Full text and rfc822 format available.

Acknowledgement sent to Charles Steinkuehler <charles@steinkuehler.net>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #17 received at 251905@bugs.debian.org (full text, mbox):

From: Charles Steinkuehler <charles@steinkuehler.net>
To: Martin Michlmayr <tbm@cyrius.com>, 251905@bugs.debian.org, debian-boot@lists.debian.org
Subject: Re: Bug#251905: Problems/workarounds for install to root on LVM on RAID
Date: Mon, 21 Jun 2004 15:28:29 -0500
[Message part 1 (text/plain, inline)]
Martin Michlmayr wrote:

> * tbm <tbm@cyrius.com> [2004-05-31 13:37]:
>> 1) partman won't let you build an LVM on top of a RAID device
> 
> Still there.  This needs changed to libparted - maybe you can look at
> this.

I haven't looked at this.  If it doesn't involve lots of coding, I may 
be able to take a stab at it.

>> 3) Grub install fails when /boot is on a RAID1 device
> 
> Still there.

Attached are two patches to address this (one for grub-install, and one 
for update-grub).  Patches are against the version I have installed, 
which according to the changelog is 0.94+cvs20040511-1

update-grub:
------------

This one is pretty easy.  The convert routine (which fails when passed 
an md device) is only called to get a default grub root device to use if 
one is not provided in the menu.lst file.  It's not really necessary to 
have anything provided by this routine (in fact, if device.map doesn't 
exist, the default behavior is simply set the default to (hd0,0)).

I just created a convert_default 'wrapper', which calls convert() and 
returns the result or (hd0,0) if anything goes wrong (ie: convert exits 
with non-zero status).

grub-install:
-------------

This is a bit more complicated, as we actually have to figure out what 
device(s) to install grub on.  My solution was to add a test for an md 
device prior to any of the other processing in the convert() procedure. 
If we're trying to convert /dev/md*, the new procedure getraid_mdadm() 
is called to identify a BIOS device we can then pass to the rest of the 
convert() code.

The getraid_mdadm procedure is pretty much lifted from the Debian 
mkinitrd scripts (at least the part that converts an md device into a 
list of physical drives).  I modified the routine to return a single 
device, and to return the highest priority BIOS device that's also part 
of the RAID array.  If no RAID members are BIOS devices (according to 
the device.map file), the first device listed by mdadm is returned.

NOTES:
------

- I have not implemented support for raidtools extraction of the RAID 
device info (ie: mdadm is required), although the code is in the 
mkinitrd script.  If needed, this is left as an exercise for the reader :)

- I did not modify grub-install to install itself on the MBR of *ALL* 
devices in the array (which I usually do manually, in case the 'primary' 
HDD is the one that dies).  This may or may not be desired behavior, but 
it seems somewhat unsafe to simply assume this behavior is wanted.  You 
can do this manually by:
  grub-install '(hd0)'
  grub-install '(hd1)'
  ...

- I have not yet tested these mods on a 'bare-metal' install, but my 
system still boots after running 'grub-install' and 'update-grub' with 
the included mods.  I need to do another re-install, so hopefully I'll 
be able to report on the success (or failure) of these patches on a 
ground-up install.

-- 
Charles Steinkuehler
charles@steinkuehler.net
[grub-install.diff (text/plain, inline)]
--- grub-install.was	2004-06-21 14:36:27.000000000 -0500
+++ grub-install	2004-06-21 14:34:05.000000000 -0500
@@ -80,6 +80,50 @@
 EOF
 }
 
+# Usage: getraid_mdadm mddevice
+# Routine to find a physical device from an md device
+# If found, the first grub BIOS device (from device.map) is returned 
+# If no BIOS drives match the RAID devices, the first device returned
+# from mdadm -D is returned
+getraid_mdadm() {
+	device=$1
+	mdadm=$(mdadm -D "$device") || {
+		echo "$PROG: mdadm -D $device failed" >&2
+		exit 1
+	}
+	eval "$(
+		echo "$mdadm" | awk '
+			$1 == "Number" && $2 == "Major" { start = 1; next }
+			$1 == "UUID" { print "uuid=" $3; start = 0; next }
+			!start { next }
+			$2 == 0 && $3 == 0 { next }
+			{ devices = devices "\n" $NF }
+			END { print "devices='\''" devices "'\''" }
+		'
+	)"
+
+	# Convert RAID devices list into a list of disks
+	tmp_disks=`echo "$devices" | sed -e 's%\([sh]d[a-z]\)[0-9]*$%\1%' \
+					 -e 's%\(d[0-9]*\)p[0-9]*$%\1%' \
+					 -e 's%\(fd[0-9]*\)$%\1%' \
+					 -e 's%/part[0-9]*$%/disc%' \
+					 -e 's%\(c[0-7]d[0-9]*\).*$%\1%' \
+					 -e '/^$/d' |
+				     sed -n '1h;2,$H;${g;s/\n/|/g;p}'`
+
+	# Find first BIOS disk that's a member of the RAID array
+	# Default to first RAID member if no tmp_disks are BIOS devices
+	set -- `egrep $tmp_disks $device_map | \
+		sort | \
+		sed -n 1p `
+	device=${2:-${tmp_disks%%|*}}
+
+	# Return first partition on BIOS disk that's part of the RAID
+	echo "$devices" | \
+		sed -n "\:${device}:p" | \
+		sed -n 1p
+}
+
 # Usage: convert os_device
 # Convert an OS device to the corresponding GRUB drive.
 # This part is OS-specific.
@@ -95,6 +139,10 @@
     # Break the device name into the disk part and the partition part.
     case "$host_os" in
     linux*)
+	# Find an actual physical device if we're passed a RAID device
+	case $1 in
+		/dev/md*)  set -- `getraid_mdadm $1`
+	esac
 	tmp_disk=`echo "$1" | sed -e 's%\([sh]d[a-z]\)[0-9]*$%\1%' \
 				  -e 's%\(d[0-9]*\)p[0-9]*$%\1%' \
 				  -e 's%\(fd[0-9]*\)$%\1%' \
[update-grub.diff (text/plain, inline)]
--- update-grub.was	2004-06-21 14:37:35.000000000 -0500
+++ update-grub	2004-06-21 14:46:58.000000000 -0500
@@ -196,6 +196,18 @@
     fi
 }
 
+# Usage: convert_default os_device
+# Convert an OS device to the corresponding GRUB drive.
+# Calls OS-specific convert, and returns a default of
+# (hd0,0) if anything goes wrong
+convert_default () {
+	local tmp
+	if tmp=$(convert $1 2>/dev/null) ; then
+		echo $tmp
+	else
+		echo "(hd0,0)"
+	fi
+}
 
 ## Configuration Options
 # directory's to look for the grub installation and the menu file
@@ -224,9 +236,9 @@
 # if we don't have a device.map then we can't use the convert function.
 if test -f "$device_map" ; then
 	if test -z "$boot_device" ; then
-		grub_root_device=$(convert "$root_device")
+		grub_root_device=$(convert_default "$root_device")
 	else
-		grub_root_device=$(convert "$boot_device")
+		grub_root_device=$(convert_default "$boot_device")
 	fi
 else
 	grub_root_device="(hd0,0)"

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#251905; Package lvmcfg. Full text and rfc822 format available.

Acknowledgement sent to Philipp Kolmann <philipp@kolmann.at>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #22 received at 251905@bugs.debian.org (full text, mbox):

From: Philipp Kolmann <philipp@kolmann.at>
To: Charles Steinkuehler <charles@steinkuehler.net>, Martin Michlmayr <tbm@cyrius.com>, 251905@bugs.debian.org, debian-boot@lists.debian.org
Subject: Re: Bug#251905: Problems/workarounds for install to root on LVM on RAID
Date: Tue, 22 Jun 2004 00:05:28 +0200
Hi!

On Mon, Jun 21, 2004 at 03:28:29PM -0500, Charles Steinkuehler wrote:
> Martin Michlmayr wrote:
> >>3) Grub install fails when /boot is on a RAID1 device
> >
> >Still there.
> 
> Attached are two patches to address this (one for grub-install, and one 
> for update-grub).  Patches are against the version I have installed, 
> which according to the changelog is 0.94+cvs20040511-1

Just tried these 2 patches on a fresh install with a raid1 set and grub
installed really nicely.

System booted afterwards nice and seems to work.

I have disabled the primary mirror and tried booting from the secondary
disk. But this failed, and I found the old lilo install on this MBR.
Seems that there is the grub-install on the other disk missing.

Very good work.
Philipp

-- 
Life is not fair,
but the root password helps!



Bug 251905 cloned as bug 259238. Request was from Martin Michlmayr <tbm@cyrius.com> to control@bugs.debian.org. Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#251905; Package lvmcfg. Full text and rfc822 format available.

Acknowledgement sent to The Anarcat <anarcat@anarcat.ath.cx>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #29 received at 251905@bugs.debian.org (full text, mbox):

From: The Anarcat <anarcat@anarcat.ath.cx>
To: Debian Bug Tracking System <251905@bugs.debian.org>
Subject: lvmcfg: works here
Date: Wed, 14 Dec 2005 15:52:06 -0500
Package: lvmcfg
Followup-For: Bug #251905

I just want to mention that I was able to setup a RAID-1 system with /
on a RAID array and the rest (/usr, /var, /tmp and /home) on LVM using
the sarge amd64 installer images.

Worked like a charm. The only problem I saw was that grub was installed
only on one drive, but that grub magic fixed it:

# grub
Probing devices to guess BIOS drives. This may take a long time.
[...]
grub> device (hd0) /dev/sdb
grub> root (hd0,1)
 Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  16 sectors are
embedded.
succeeded
 Running "install /boot/grub/stage1 (hd0) (hd0)1+16 p
(hd0,1)/boot/grub/stage2 /boot/grub/menu.lst"... succeeded
Done.

note that (hd0,1) is because I reserved a non-raid and unused partition
at the beginning of the disk, so the root is actually on the second
partition.

a.



Information forwarded to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>:
Bug#251905; Package lvmcfg. Full text and rfc822 format available.

Acknowledgement sent to Sam Morris <sam@robots.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>. Full text and rfc822 format available.

Message #34 received at 251905@bugs.debian.org (full text, mbox):

From: Sam Morris <sam@robots.org.uk>
To: 251905@bugs.debian.org
Subject: Re: Bug#251905: Problems/workarounds for install to root on LVM on RAID
Date: Fri, 22 Sep 2006 11:59:20 +0100
[Message part 1 (text/plain, inline)]
I just tried a root-on-RAID1 setup and the only issue was that GRUB was
only installed to the MBR of the first hard disk, so when I simulated
the failure of the first disk, the system would not boot.

-- 
Sam Morris
http://robots.org.uk/

PGP key id 1024D/5EA01078
3412 EA18 1277 354B 991B  C869 B219 7FDB 5EA0 1078
[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Sun Apr 20 17:00:39 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.