Debian Bug report logs - #672959
startpar triggers kfreebsd panic: vm_fault_copy_wired

version graph

Package: src:sysvinit; Maintainer for src:sysvinit is Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>;

Reported by: Steven Chamberlain <steven@pyro.eu.org>

Date: Mon, 14 May 2012 23:00:01 UTC

Severity: serious

Tags: help, moreinfo, patch, sid, wheezy

Found in versions sysvinit/2.88dsf-31, sysvinit/2.88dsf-22.1

Fixed in version sysvinit/2.88dsf-32

Done: Roger Leigh <rleigh@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, debian-bsd@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Mon, 14 May 2012 23:00:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steven Chamberlain <steven@pyro.eu.org>:
New Bug report received and forwarded. Copy sent to debian-bsd@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Mon, 14 May 2012 23:00:04 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Steven Chamberlain <steven@pyro.eu.org>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: kfreebsd-*: panic: vm_fault_copy_wired
Date: Mon, 14 May 2012 23:57:40 +0100
Package: src:sysvinit
Version: 2.88dsf-22.1
Severity: important
Tags: sid wheezy
User: debian-bsd@lists.debian.org
Usertags: kfreebsd
X-Debbugs-Cc: debian-bsd@lists.debian.org

Hi,

Not exactly sure yet where to file this bug, perhaps initscripts, a
kernel (kfreebsd-8 and -9) bug or something else but I shall file it
here for now.

If a GNU/kFreeBSD ufs root filesystem is dirty (from unclean shutdown),
the panic below is triggered during the early init scripts.  The only
way to fix it is to use some rescue system (e.g. d-i) to fsck the
filesystem, marking it clean.

First reported 2012-05-07, and confirmed by two others on
debian-bsd@lists.d.o;  see
http://lists.debian.org/4FA80DBD.3000200@pyro.eu.org

> Trying to mount root from ufs:/dev/ad0s1
> WARNING: / was not properly dismounted
> mount option <nodev> is unknown
> mount option <nodev> is unknown
> mount option <nodev> is unknown
> panic: vm_fault_copy_wired: page missing
> cpuid = 1
> KDB: stack backtrace:
> Uptime: 4s
> Cannot dump. Device not defined or unavailable.
> Automatic reboot in 15 seconds - press a key on the console to abort
> --> Press a key on the console to reboot,
> --> or switch off the system now.

Thanks!
Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org




Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Wed, 16 May 2012 06:39:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Robert Millan <rmh@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Wed, 16 May 2012 06:39:03 GMT) Full text and rfc822 format available.

Message #10 received at 672959@bugs.debian.org (full text, mbox):

From: Robert Millan <rmh@debian.org>
To: Steven Chamberlain <steven@pyro.eu.org>, 672959@bugs.debian.org
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Wed, 16 May 2012 08:34:30 +0200
2012/5/15 Steven Chamberlain <steven@pyro.eu.org>:
> If a GNU/kFreeBSD ufs root filesystem is dirty (from unclean shutdown),
> the panic below is triggered during the early init scripts.  The only
> way to fix it is to use some rescue system (e.g. d-i) to fsck the
> filesystem, marking it clean.

I don't think there's anything userland can do at this point (I
suggest reassigning to kernel). Later on it'd have to ensure that
fsck.ufs is run before remounting read-write, but AFAICS this point
isn't reached.

Can you verify that the file system is being mounted read-only?

Is it reproducible with upstream kernel? (kfreebsd-downloader)

-- 
Robert Millan




Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Wed, 16 May 2012 06:39:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Robert Millan <rmh@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Wed, 16 May 2012 06:39:05 GMT) Full text and rfc822 format available.

Message #15 received at 672959@bugs.debian.org (full text, mbox):

From: Robert Millan <rmh@debian.org>
To: Steven Chamberlain <steven@pyro.eu.org>, 672959@bugs.debian.org
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Wed, 16 May 2012 08:37:45 +0200
2012/5/16 Robert Millan <rmh@debian.org>:
> I don't think there's anything userland can do at this point (I
> suggest reassigning to kernel). Later on it'd have to ensure that
> fsck.ufs is run before remounting read-write, but AFAICS this point
> isn't reached.
>
> Can you verify that the file system is being mounted read-only?
>
> Is it reproducible with upstream kernel? (kfreebsd-downloader)

Erm sorry, disregard that.  I didn't notice that userland output isn't
being included in your log.

Does this problem happen when botting in single mode? (second option
in GRUB menu)

-- 
Robert Millan




Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Wed, 16 May 2012 12:03:53 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steven Chamberlain <steven@pyro.eu.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Wed, 16 May 2012 12:04:23 GMT) Full text and rfc822 format available.

Message #20 received at 672959@bugs.debian.org (full text, mbox):

From: Steven Chamberlain <steven@pyro.eu.org>
To: Robert Millan <rmh@debian.org>
Cc: 672959@bugs.debian.org, "debian-bsd@lists.debian.org" <debian-bsd@lists.debian.org>
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Wed, 16 May 2012 13:00:40 +0100
On 16/05/12 07:37, Robert Millan wrote:
> Erm sorry, disregard that.  I didn't notice that userland output isn't
> being included in your log.

Oooh I didn't realise that either, the serial console only shows the
kernel output.  I saw the nodev errors which are triggered from
mountkernfs.sh, but forgot that those messages were actually coming from
the kernel.

> Does this problem happen when botting in single mode? (second option
> in GRUB menu)

Yes it still happens in single user mode, on 8.3 or 9.0 kernels.  And in
fact, the filesystem doesn't have to be marked dirty -- I managed to
trigger this just by creating a /forcefsck file:

> Activating swap:.
> Will now check root file system:fsck from util-linux 2.20.1
> [/sbin/fsck.ufs (1) -- /] fsck.ufs -f -a /dev/ad0s1
> /dev/ad0s1: 15710 files, 260322 used, 682949 free (11855 frags, 83883 blocks, 1.3% fragmentation)
> .
> panic: vm_fault_copy_wired: page missing
> cpuid = 0
> KDB: stack backtrace:
> Uptime: 4s
> Cannot dump. Device not defined or unavailable.
> Automatic reboot in 15 seconds - press a key on the console to abort
> --> Press a key on the console to reboot,
> --> or switch off the system now.


I tried setting kFreeBSD.init_path to a shell but as mentioned on the
FAQ that doesn't work:

http://wiki.debian.org/Debian_GNU/kFreeBSD_FAQ#Q._How_do_I_make_the_kFreeBSD_kernel_launch_another_binary_instead_of_.2BAC8-sbin.2BAC8-init.3F

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org




Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Wed, 16 May 2012 21:39:11 GMT) Full text and rfc822 format available.

Acknowledgement sent to Robert Millan <rmh@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Wed, 16 May 2012 21:39:12 GMT) Full text and rfc822 format available.

Message #25 received at 672959@bugs.debian.org (full text, mbox):

From: Robert Millan <rmh@debian.org>
To: Steven Chamberlain <steven@pyro.eu.org>
Cc: 672959@bugs.debian.org, "debian-bsd@lists.debian.org" <debian-bsd@lists.debian.org>
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Wed, 16 May 2012 23:37:43 +0200
2012/5/16 Steven Chamberlain <steven@pyro.eu.org>:
> I tried setting kFreeBSD.init_path to a shell but as mentioned on the
> FAQ that doesn't work:
>
> http://wiki.debian.org/Debian_GNU/kFreeBSD_FAQ#Q._How_do_I_make_the_kFreeBSD_kernel_launch_another_binary_instead_of_.2BAC8-sbin.2BAC8-init.3F

Init is expected to setup console all by itself. You can use a shell
script as init:

#!/bin/sh
exec bash < /dev/console > /dev/console 2> /dev/console

-- 
Robert Millan




Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Sat, 16 Jun 2012 02:57:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steven Chamberlain <steven@pyro.eu.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Sat, 16 Jun 2012 02:57:03 GMT) Full text and rfc822 format available.

Message #30 received at 672959@bugs.debian.org (full text, mbox):

From: Steven Chamberlain <steven@pyro.eu.org>
To: Robert Millan <rmh@debian.org>
Cc: 672959@bugs.debian.org, "debian-bsd@lists.debian.org" <debian-bsd@lists.debian.org>
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Sat, 16 Jun 2012 03:52:29 +0100
[Message part 1 (text/plain, inline)]
On 16/05/12 22:37, Robert Millan wrote:
> #!/bin/sh
> exec bash < /dev/console > /dev/console 2> /dev/console

If I do that and start each script in /etc/rcS.d/S* manually one at a
time, there is no problem.  The rootfs is mounted read-only to begin
with, it gets fixed/marked clean during the checkfs step, later gets
mounted read-write and all is fine.

This has to do with parallelism of "/etc/init.d/rc S".  The default
"makefile-style" concurrency can trigger this fault, whereas "touch
/etc/init.d/.legacy-bootordering" makes it no longer reproducible.

Making this particularly awkward, startpar buffers the output of scripts
until they complete, so the console output is probably hiding the
responsible script, which may have already started before fsck
completes;  and fsck cannot be triggering the fault directly.

Trying to analyse /etc/init.d/.depend.boot (somewhat messy graphviz.dot
file attached), I think kldutils -> {geli freebsdutils}, rsyslog ->
powerd and zfs are possible candidates because they seem to be able to
start before checkroot.sh completes.

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org
[vm_fault_copy_wired.png (image/png, attachment)]
[graphviz.dot (application/msword, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Sat, 21 Jul 2012 20:57:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Carlos Alberto Lopez Perez <clopez@igalia.com>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Sat, 21 Jul 2012 20:57:03 GMT) Full text and rfc822 format available.

Message #35 received at submit@bugs.debian.org (full text, mbox):

From: Carlos Alberto Lopez Perez <clopez@igalia.com>
To: 672959@bugs.debian.org, Debian Bug Tracking System <submit@bugs.debian.org>
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Sat, 21 Jul 2012 22:55:46 +0200
[Message part 1 (text/plain, inline)]
severity 672959 serious
thanks
justification: unacceptable behavior on a production/stable system. renders the system unable to boot.


Hello,

This is a me-to report. Running Debian/sid kFreeBSD ADM64 on a virtual machine with qemu/kvm I unclean closed the machine (killed the qemu process when the machine was idle) and now the machine not longer starts.

When the init system reachs to "Checking root filesystems..." it gives a fatal error and the machine restarts itself (attached screen-shoot)

The machine keeps restarting itself in an infinite loop and there is no way to fix it.

Neither booting single-use mode (recovery mode) nor normal mode changes anything. As also neither booting kFreeBSD kernel version 9.0-1-amd64 nor 8.3-1-amd64 changes anything.

The machine has the following schema for its unique disk:

sysresccd ~ # fdisk -l /dev/sda

Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00042061

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    15988735     7993344   a5  FreeBSD
/dev/sda2        15990782    16775167      392193    5  Extended
/dev/sda5        15990784    16775167      392192   82  Linux swap / Solaris

sysresccd ~ # file -s /dev/sda1
/dev/sda1: Unix Fast File system [v2] (little-endian) last mounted on /, last written at Sat Jul 21 20:36:14 2012, clean flag 0, readonly flag 0, number of blocks 3996672, number of data blocks 3869351, number of cylinder groups 43, block size 16384, fragment size 2048, average file size 16384, average number of files in dir 64, pending blocks to free 0, pending inodes to free 0, system-wide uuid 0, minimum percentage of free blocks 8, TIME optimization


I tried fixing it booting the system with systemrescuecd but seems that the tool
fsck.ufs is not included on it.

Finally I was able to fix it by booting with the Debian/kFreeBSD AMD64 netinstall,
pressing next-next until reaching the partitioning dialog and then after
getting a shell with alt+f2 and running fsck.ufs /dev/ad0s1 the error goes
away.


-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Carlos Alberto Lopez Perez                           http://neutrino.es
Igalia - Free Software Engineering                http://www.igalia.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[kfreebsdcrash.png (image/png, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Sat, 21 Jul 2012 21:09:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Carlos Alberto Lopez Perez <clopez@igalia.com>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Sat, 21 Jul 2012 21:09:03 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Sat, 21 Jul 2012 21:36:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Roger Leigh <rleigh@codelibre.net>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Sat, 21 Jul 2012 21:36:05 GMT) Full text and rfc822 format available.

Message #45 received at 672959@bugs.debian.org (full text, mbox):

From: Roger Leigh <rleigh@codelibre.net>
To: Carlos Alberto Lopez Perez <clopez@igalia.com>, 672959@bugs.debian.org
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Sat, 21 Jul 2012 22:32:36 +0100
On Sat, Jul 21, 2012 at 10:55:46PM +0200, Carlos Alberto Lopez Perez wrote:
> severity 672959 serious
> thanks
> justification: unacceptable behavior on a production/stable system. renders the system unable to boot.
> 
> When the init system reachs to "Checking root filesystems..." it gives a fatal error and the machine restarts itself (attached screen-shoot)
> 
> The machine keeps restarting itself in an infinite loop and there is no way to fix it.
> 
> Neither booting single-use mode (recovery mode) nor normal mode changes anything. As also neither booting kFreeBSD kernel version 9.0-1-amd64 nor 8.3-1-amd64 changes anything.

So from what the other submitters reported, it sounds like this is
the result of paralellism in the rcS scripts.  This would imply
that there's a missing dependency relating to checkroot.sh.  So
this should be fairly simple to fix once we know what's running
in parallel inappropriately.

/usr/share/insserv/make-testsuite and
/etc/init.d/.depend.boot
will show you the dependency graph.

Looking at the screenshot you attached, I can't see any obvious
reason for fsck to make the kernel panic.  There's no indication
of odd scripts (other than geli) running in parallel here.  This
really needs investigation from a kfreebsd person; I'm not
sufficiently knowledgeable to look into it myself.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux    http://people.debian.org/~rleigh/
 `. `'   schroot and sbuild  http://alioth.debian.org/projects/buildd-tools
   `-    GPG Public Key      F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Sat, 21 Jul 2012 21:51:09 GMT) Full text and rfc822 format available.

Acknowledgement sent to Carlos Alberto Lopez Perez <clopez@igalia.com>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Sat, 21 Jul 2012 21:51:09 GMT) Full text and rfc822 format available.

Message #50 received at 672959@bugs.debian.org (full text, mbox):

From: Carlos Alberto Lopez Perez <clopez@igalia.com>
To: Roger Leigh <rleigh@codelibre.net>
Cc: 672959@bugs.debian.org
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Sat, 21 Jul 2012 23:47:24 +0200
[Message part 1 (text/plain, inline)]
On 21/07/12 23:32, Roger Leigh wrote:
> So from what the other submitters reported, it sounds like this is
> the result of paralellism in the rcS scripts.  This would imply
> that there's a missing dependency relating to checkroot.sh.  So
> this should be fairly simple to fix once we know what's running
> in parallel inappropriately.
> 
> /usr/share/insserv/make-testsuite and

attached make-testsuite.output.txt


> /etc/init.d/.depend.boot
> will show you the dependency graph.

# cat /etc/init.d/.depend.boot
TARGETS = kldutils mountkernfs.sh hostname.sh freebsd-utils keyboard-setup geli mountdevsubfs.sh checkroot.sh mtab.sh checkroot-bootclean.sh checkfs.sh mountall.sh urandom devd mountall-bootclean.sh networking rpcbind mountnfs.sh mountnfs-bootclean.sh console-setup rpc.statd nfsiod powerd bootmisc.sh kbdcontrol rpc.lockd
INTERACTIVE = freebsd-utils keyboard-setup checkroot.sh checkfs.sh console-setup
freebsd-utils: kldutils
keyboard-setup: mountkernfs.sh
geli: kldutils
mountdevsubfs.sh: mountkernfs.sh
checkroot.sh: geli mountdevsubfs.sh hostname.sh keyboard-setup
mtab.sh: freebsd-utils checkroot.sh
checkroot-bootclean.sh: checkroot.sh
checkfs.sh: checkroot.sh mtab.sh
mountall.sh: checkfs.sh kldutils
urandom: mountall.sh
devd: mountall.sh kldutils
mountall-bootclean.sh: mountall.sh
networking: devd mountkernfs.sh mountall.sh
rpcbind: networking mountall.sh
mountnfs.sh: mountall.sh networking rpcbind
mountnfs-bootclean.sh: mountall.sh mountnfs.sh
console-setup: mountall.sh mountnfs.sh mountnfs-bootclean.sh
rpc.statd: networking mountall.sh mountnfs.sh mountnfs-bootclean.sh rsyslog
nfsiod: networking mountall.sh rsyslog
powerd: mountall.sh mountnfs.sh mountnfs-bootclean.sh rsyslog
bootmisc.sh: mountall.sh mountnfs.sh mountnfs-bootclean.sh
kbdcontrol: mountall.sh mountnfs.sh mountnfs-bootclean.sh
rpc.lockd: rpc.statd mountall.sh mountnfs.sh mountnfs-bootclean.sh

> 
> Looking at the screenshot you attached, I can't see any obvious
> reason for fsck to make the kernel panic.  There's no indication
> of odd scripts (other than geli) running in parallel here.  This
> really needs investigation from a kfreebsd person; I'm not
> sufficiently knowledgeable to look into it myself.
> 
> 

I guess this is reproducible by simply installing Debian/kFreeBSD AMD64
on a virtual machine and forcing and unclean shutdown (killing the qemu
process when the virtual machine is running).

Regards!
[make-testsuite.output.txt (text/plain, attachment)]
[signature.asc (application/pgp-signature, attachment)]

Severity set to 'serious' from 'important' Request was from Carlos Alberto Lopez Perez <clopez@igalia.com> to control@bugs.debian.org. (Sat, 21 Jul 2012 22:03:03 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Sun, 22 Jul 2012 13:09:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Roger Leigh <rleigh@codelibre.net>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Sun, 22 Jul 2012 13:09:03 GMT) Full text and rfc822 format available.

Message #57 received at 672959@bugs.debian.org (full text, mbox):

From: Roger Leigh <rleigh@codelibre.net>
To: Carlos Alberto Lopez Perez <clopez@igalia.com>
Cc: 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Sun, 22 Jul 2012 14:06:11 +0100
On Sat, Jul 21, 2012 at 11:47:24PM +0200, Carlos Alberto Lopez Perez wrote:
> On 21/07/12 23:32, Roger Leigh wrote:
> > Looking at the screenshot you attached, I can't see any obvious
> > reason for fsck to make the kernel panic.  There's no indication
> > of odd scripts (other than geli) running in parallel here.  This
> > really needs investigation from a kfreebsd person; I'm not
> > sufficiently knowledgeable to look into it myself.
> 
> I guess this is reproducible by simply installing Debian/kFreeBSD AMD64
> on a virtual machine and forcing and unclean shutdown (killing the qemu
> process when the virtual machine is running).

I'm sure it is, but it needs someone with kfreebsd familiarity to
do this, and that's not me.  I don't have the time or skills to fix
kfreebsd issues, but I can test, review and apply patches that any
kfreebsd people wish to provide.


Thanks,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux    http://people.debian.org/~rleigh/
 `. `'   schroot and sbuild  http://alioth.debian.org/projects/buildd-tools
   `-    GPG Public Key      F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800



Added tag(s) help and moreinfo. Request was from Julien Cristau <jcristau@debian.org> to control@bugs.debian.org. (Fri, 03 Aug 2012 08:24:06 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Sat, 11 Aug 2012 16:09:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Roger Leigh <rleigh@codelibre.net>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Sat, 11 Aug 2012 16:09:06 GMT) Full text and rfc822 format available.

Message #64 received at 672959@bugs.debian.org (full text, mbox):

From: Roger Leigh <rleigh@codelibre.net>
To: 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Sat, 11 Aug 2012 17:04:54 +0100
On Sun, Jul 22, 2012 at 02:06:11PM +0100, Roger Leigh wrote:
> On Sat, Jul 21, 2012 at 11:47:24PM +0200, Carlos Alberto Lopez Perez wrote:
> > On 21/07/12 23:32, Roger Leigh wrote:
> > > Looking at the screenshot you attached, I can't see any obvious
> > > reason for fsck to make the kernel panic.  There's no indication
> > > of odd scripts (other than geli) running in parallel here.  This
> > > really needs investigation from a kfreebsd person; I'm not
> > > sufficiently knowledgeable to look into it myself.
> > 
> > I guess this is reproducible by simply installing Debian/kFreeBSD AMD64
> > on a virtual machine and forcing and unclean shutdown (killing the qemu
> > process when the virtual machine is running).
> 
> I'm sure it is, but it needs someone with kfreebsd familiarity to
> do this, and that's not me.  I don't have the time or skills to fix
> kfreebsd issues, but I can test, review and apply patches that any
> kfreebsd people wish to provide.

Hi debian-bsd people,

This bug is still RC for wheezy.  It really needs looking at by a
kfreebsd expert, if someone could take a look at this.

Given that this is only being experienced on kfreebsd, I would
suggest starting with any kfreebsd-specific init scripts which
could be running ahead of time if the dependencies aren't
correct.


Many thanks,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux    http://people.debian.org/~rleigh/
 `. `'   schroot and sbuild  http://alioth.debian.org/projects/buildd-tools
   `-    GPG Public Key      F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Mon, 13 Aug 2012 14:36:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Axel Beckert <abe@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Mon, 13 Aug 2012 14:36:03 GMT) Full text and rfc822 format available.

Message #69 received at 672959@bugs.debian.org (full text, mbox):

From: Axel Beckert <abe@debian.org>
To: 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Mon, 13 Aug 2012 16:33:43 +0200
Hi,

I ran into that on Saturday, too.

Roger Leigh wrote:
> > > I guess this is reproducible by simply installing Debian/kFreeBSD AMD64
> > > on a virtual machine and forcing and unclean shutdown (killing the qemu
> > > process when the virtual machine is running).

It seems so, yes. It seems triggered by an unclean FS, yes. Was able
to reproduce it with 8.2, 8.3 and 9.0 kernels. (Couldn't get the 10.0
kernel to boot at all, but that's another story, not yet reported
though.)

> This bug is still RC for wheezy.

And justifiably so.

> It really needs looking at by a kfreebsd expert, if someone could
> take a look at this.

Will do.

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Sat, 18 Aug 2012 11:30:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Axel Beckert <abe@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Sat, 18 Aug 2012 11:30:03 GMT) Full text and rfc822 format available.

Message #74 received at 672959@bugs.debian.org (full text, mbox):

From: Axel Beckert <abe@debian.org>
To: 672959@bugs.debian.org
Cc: debian-bsd@lists.debian.org
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Sat, 18 Aug 2012 13:28:28 +0200
Hi,

here are my observations so far after several hours and evenings of
rebooting my kFreeBSD EeeBox again and again.

I've unfortunately not yet found the real cause or an solution, but
maybe some of my observations spark ideas at others. I'll continue
debugging the issue on Monday evening as I'm not at home for the
weekend.

Roger Leigh wrote:
> Looking at the screenshot you attached, I can't see any obvious
> reason for fsck to make the kernel panic.  There's no indication
> of odd scripts (other than geli) running in parallel here.

Same here. But compared to
http://bugs.debian.org/cgi-bin/bugreport.cgi?msg=30;filename=vm_fault_copy_wired.png;att=1;bug=672959
and
http://bugs.debian.org/cgi-bin/bugreport.cgi?msg=35;filename=kfreebsdcrash.png;att=1;bug=672959
where crash happens after 4 and 6 seconds, my fsck runs around 7
_minutes_ before it crashes. So I doubt that after 7 minutes
uptime a job running parallel _to_ checkroot causes this.

One of the screenshots also hints that the crash is after checkroot.sh
and my experience confirms that, also because my fsck always runs more
than 6 minutes and I think that this means it's likely caused by
something which comes after the checkroot.sh.

Interestingly pressing Ctrl-C during the fsck gives you (after
entering the root password) a working shell and starting
"/etc/init.d/checkroot.sh start" manually there works without issues.
Exiting the shell reboots (cleanly) and then the system comes up
cleanly again. (Should work as workaround without any preparations
like creating shell scripts to run be instead of init or a live CD to
run the fsck manually. Just be quick enough if your fsck just takes a
few seconds. :-)

So I added "set -x" to all init.d scripts and (unfortunately) the
crash seems to always happen after the last line of checkroot.sh which
is ":", i.e. the last line shown always was "+ :" and the second last
line was "+ rm -f /dev/rootdev".

So I commented out the removal of /dev/rootdev, but that didn't change
anything either.

I also played around with additional dependencies based on guessing
(e.g. added dependencies on devd or urandom) to at least all the
scripts which have checkroot as direct dependencies to reduce the
parallelism after checkroot.sh, but that didn't change anything
either so far.

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Mon, 20 Aug 2012 14:33:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to martin-eric.racine@iki.fi:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Mon, 20 Aug 2012 14:33:03 GMT) Full text and rfc822 format available.

Message #79 received at 672959@bugs.debian.org (full text, mbox):

From: Martin-Éric Racine <martin-eric.racine@iki.fi>
To: 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Mon, 20 Aug 2012 17:28:52 +0300
Something that popped into mind:

Has the exact sysvinit version where this issue crept in been
positively identified?

While the bug report was filed against version 2.88dsf-22.1, I'm
wondering if that was the exact sysvinit version where fsck.ufs
started crashing the boot process in such a systematic way. Could
someone from the debian-bsd list check this using earlier binaries
from ( http://snapshot.debian.org/package/sysvinit/ ) and report
whether any of them magically fixes the issue? If this is the case, it
would allow us to backtrack the source of the bug.

Martin-Éric



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Mon, 20 Aug 2012 15:09:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Axel Beckert <abe@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Mon, 20 Aug 2012 15:09:05 GMT) Full text and rfc822 format available.

Message #84 received at 672959@bugs.debian.org (full text, mbox):

From: Axel Beckert <abe@debian.org>
To: martin-eric.racine@iki.fi, 672959@bugs.debian.org
Cc: debian-bsd@lists.debian.org
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Mon, 20 Aug 2012 17:07:56 +0200
Hi,

Martin-Éric Racine wrote:
> Something that popped into mind:
> 
> Has the exact sysvinit version where this issue crept in been
> positively identified?

AFAIK not yet, because nobody tried.

> While the bug report was filed against version 2.88dsf-22.1, I'm
> wondering if that was the exact sysvinit version where fsck.ufs
> started crashing the boot process in such a systematic way. Could
> someone from the debian-bsd list check this using earlier binaries
> from ( http://snapshot.debian.org/package/sysvinit/ ) and report
> whether any of them magically fixes the issue? If this is the case, it
> would allow us to backtrack the source of the bug.

Will do! Thanks for the hint!

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Mon, 20 Aug 2012 18:21:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steven Chamberlain <steven@pyro.eu.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Mon, 20 Aug 2012 18:21:03 GMT) Full text and rfc822 format available.

Message #89 received at 672959@bugs.debian.org (full text, mbox):

From: Steven Chamberlain <steven@pyro.eu.org>
To: martin-eric.racine@iki.fi, 672959@bugs.debian.org, debian-bsd@lists.debian.org, Axel Beckert <abe@debian.org>
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Mon, 20 Aug 2012 19:16:52 +0100
On 20/08/12 16:07, Axel Beckert wrote:
>> Has the exact sysvinit version where this issue crept in been
>> positively identified?
> 
> AFAIK not yet, because nobody tried.

2.88dsf-22.1 is simply what I had installed when I filed the bug.  It
most likely affects earlier versions too.  It would be a good idea to
check this I think.

Axel, have you tested any more recent versions than this?  In particular
2.88dsf-23 and 2.88dsf-28 made changes relevant to kFreeBSD.

When I disabled parallel init as explained in [1], that avoided the
crash.  So that narrows down the affected versions at least a /little/
bit;  concurrent boot became default in 2.88dsf-3.

[1] http://bugs.debian.org/672959#20

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Mon, 20 Aug 2012 18:30:12 GMT) Full text and rfc822 format available.

Acknowledgement sent to Petter Reinholdtsen <pere@hungry.com>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Mon, 20 Aug 2012 18:30:12 GMT) Full text and rfc822 format available.

Message #94 received at 672959@bugs.debian.org (full text, mbox):

From: Petter Reinholdtsen <pere@hungry.com>
To: Steven Chamberlain <steven@pyro.eu.org>, 672959@bugs.debian.org
Cc: martin-eric.racine@iki.fi, debian-bsd@lists.debian.org, Axel Beckert <abe@debian.org>
Subject: Re: [Pkg-sysvinit-devel] Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Mon, 20 Aug 2012 20:26:31 +0200
Maybe I am stupit or something, but if fsck called by some init script
causes the kernel to crash, why are you trying to figure out how the
init script changed to cause this, instead of looking at the kernel
and fsck?

If the kernel can crash, it is a bug in the kernel.  If fsck can cause
the kernel, this might be a bug in fsck.  But I fail to see how
calling fsck and causing a crash in the kernel can be a bug in the
script calling fsck.

why isn't this bug reassigned to the freebsd kernel?

-- 
Happy hacking
Petter Reinholdtsen



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Mon, 20 Aug 2012 18:42:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steven Chamberlain <steven@pyro.eu.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Mon, 20 Aug 2012 18:42:06 GMT) Full text and rfc822 format available.

Message #99 received at 672959@bugs.debian.org (full text, mbox):

From: Steven Chamberlain <steven@pyro.eu.org>
To: Petter Reinholdtsen <pere@hungry.com>
Cc: 672959@bugs.debian.org, martin-eric.racine@iki.fi, debian-bsd@lists.debian.org, Axel Beckert <abe@debian.org>
Subject: Re: [Pkg-sysvinit-devel] Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Mon, 20 Aug 2012 19:41:17 +0100
On 20/08/12 19:26, Petter Reinholdtsen wrote:
> If the kernel can crash, it is a bug in the kernel.

Well, something in the initscripts could be telling the kernel to do
something bad.  Maybe to do with the rootfs, or swap.  Then it might be
acceptable for the kernel to panic and that wouldn't be a kernel bug.

Probably some initscript is at fault here, but since I don't know which
one yet, I filed against sysvinit until we know where to reassign it.

> If fsck can cause the kernel [to crash], this might be a bug in fsck.

I'm not sure that fsck itself causes this.  Because if I boot a shell
instead of /sbin/init, and run initscripts manually (including fsck), it
doesn't panic.

I think when sysvinit runs initscripts in parallel, it does some
buffering of console output, so that makes it hard to know what was
running at the time of the panic.  I think fsck must have already
finished running when the panic happens (or else we wouldn't see it
mentioned?).

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Mon, 20 Aug 2012 19:21:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Axel Beckert <abe@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Mon, 20 Aug 2012 19:21:03 GMT) Full text and rfc822 format available.

Message #104 received at 672959@bugs.debian.org (full text, mbox):

From: Axel Beckert <abe@debian.org>
To: Petter Reinholdtsen <pere@hungry.com>
Cc: 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: [Pkg-sysvinit-devel] Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Mon, 20 Aug 2012 21:16:10 +0200
Petter Reinholdtsen wrote:
> Maybe I am stupit or something, but if fsck called by some init script
> causes the kernel to crash,

No, it doesn't. As I wrote, clearly something _after_ the fsck does
that.

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Mon, 20 Aug 2012 19:24:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Axel Beckert <abe@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Mon, 20 Aug 2012 19:24:03 GMT) Full text and rfc822 format available.

Message #109 received at 672959@bugs.debian.org (full text, mbox):

From: Axel Beckert <abe@debian.org>
To: Steven Chamberlain <steven@pyro.eu.org>
Cc: 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Mon, 20 Aug 2012 21:21:39 +0200
Hi Steven,

Steven Chamberlain wrote:
> 2.88dsf-22.1 is simply what I had installed when I filed the bug.  It
> most likely affects earlier versions too.  It would be a good idea to
> check this I think.
> 
> Axel, have you tested any more recent versions than this?

So far all my tests where with 2.88dsf-31 which is the most up-to-date
version as far as I can see. Not sure about the version when I ran
into it initially, but it was probably not older than a few weeks.

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5



Marked as found in versions sysvinit/2.88dsf-31. Request was from Steven Chamberlain <steven@pyro.eu.org> to control@bugs.debian.org. (Mon, 20 Aug 2012 19:27:05 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Mon, 20 Aug 2012 19:54:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steven Chamberlain <steven@pyro.eu.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Mon, 20 Aug 2012 19:54:02 GMT) Full text and rfc822 format available.

Message #116 received at 672959@bugs.debian.org (full text, mbox):

From: Steven Chamberlain <steven@pyro.eu.org>
To: Axel Beckert <abe@debian.org>, 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: [Pkg-sysvinit-devel] Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Mon, 20 Aug 2012 20:50:33 +0100
On 20/08/12 20:16, Axel Beckert wrote:
> [...] clearly something _after_ the fsck does that.

Looking back at my 'dot' graph of initscript dependencies, it seems like
freebsdutils is the only thing that is free to run before, after, or at
the same time as checkroot.sh and thus could be affected by how long it
takes to run.

Then mtab.sh would be the next thing to run after both checkroot.sh and
freebsdutils are finished.


freebsdutils might try to rm and recreate /etc/mtab as a symlink while
fsck is running, if it is not already there.  Or more likely that would
fail because I think it would be still read-only at that point.

Or instead it may try to mount things on /dev/fd, /proc or /sys while
fsck is running.  I don't know what implications that could have.

Do you think freebsdutils should really wait for checkroot?

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Tue, 21 Aug 2012 02:24:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steven Chamberlain <steven@pyro.eu.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Tue, 21 Aug 2012 02:24:03 GMT) Full text and rfc822 format available.

Message #121 received at 672959@bugs.debian.org (full text, mbox):

From: Steven Chamberlain <steven@pyro.eu.org>
To: Axel Beckert <abe@debian.org>, 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: [Pkg-sysvinit-devel] Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Tue, 21 Aug 2012 03:21:18 +0100
Hi,

I tried commenting out the code in checkroot.sh that remounts the rootfs
as writable after fsck is finished.  It still panics, but then the
filesystem isn't marked as dirty on next boot (because fsck succeeded,
marked it as clean, and the fs was still mounted read-only at the time
of the panic).

Adding a sleep atop of mtab.sh doesn't seem to delay the panic.  I
figure that would be the next script to run after checkroot.sh.


On 20/08/12 20:50, Steven Chamberlain wrote:
> freebsdutils is [...] free to run before, after, or at
> the same time as checkroot.sh

I've been looking into that.  freebsd-utils provides "udev" and
checkroot depends on that.  That seems to affect the ordering of
/etc/rcS.d/S* symlinks at least.

But the /etc/init.d/.depend.boot file as used by startpar, doesn't seem
to reflect that dependency at all.

Either way, I didn't manage to fix anything yet by changing the
dependencies of the initscripts around.


I can successfully boot a system with a dirty rootfs by starting a shell
instead of /sbin/init, and then:

cd /etc/rcS.d && for i in S* ; do sh "$i" start ; done

Whereas invoking this instead, would trigger the kernel panic, whether I
mount linprocfs beforehand or not:

/sbin/startpar -p 4 -t 20 -T 3 -M boot -P N -R S

And the same happens even with -p 0.  This is a single-CPU VM running
kfreebsd-i386.


I'm beginning to think that startpar is malfunctioning in some way
(after checkroot.sh returns, but before it runs the next script).

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Tue, 21 Aug 2012 06:45:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to martin-eric.racine@iki.fi:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Tue, 21 Aug 2012 06:45:06 GMT) Full text and rfc822 format available.

Message #126 received at 672959@bugs.debian.org (full text, mbox):

From: Martin-Éric Racine <martin-eric.racine@iki.fi>
To: Steven Chamberlain <steven@pyro.eu.org>
Cc: Axel Beckert <abe@debian.org>, 672959@bugs.debian.org, debian-bsd@lists.debian.org, Petter Reinholdtsen <pere@hungry.com>
Subject: Re: [Pkg-sysvinit-devel] Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Tue, 21 Aug 2012 09:42:00 +0300
2012/8/20 Steven Chamberlain <steven@pyro.eu.org>:
> On 20/08/12 20:16, Axel Beckert wrote:
>> [...] clearly something _after_ the fsck does that.
>
> Looking back at my 'dot' graph of initscript dependencies, it seems like
> freebsdutils is the only thing that is free to run before, after, or at
> the same time as checkroot.sh and thus could be affected by how long it
> takes to run.

Given how this issue only seems to affect kFreeBSD ports anyhow, I'm
begining to suspect that it's due to freebsdutils having incorrect LSB
header information in its init scripts and thus extremely tempted to
reassign this bug to freebsdutils.

Martin-Éric



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Tue, 21 Aug 2012 08:00:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Petr Salinger <Petr.Salinger@seznam.cz>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Tue, 21 Aug 2012 08:00:03 GMT) Full text and rfc822 format available.

Message #131 received at 672959@bugs.debian.org (full text, mbox):

From: Petr Salinger <Petr.Salinger@seznam.cz>
To: 672959@bugs.debian.org
Cc: control@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: [patch] Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Tue, 21 Aug 2012 10:16:20 +0200 (CEST)
tags 672959 +patch
--

Hi.

> /sbin/startpar -p 4 -t 20 -T 3 -M boot -P N -R S
>
> And the same happens even with -p 0.  This is a single-CPU VM running
> kfreebsd-i386.
>
>
> I'm beginning to think that startpar is malfunctioning in some way
> (after checkroot.sh returns, but before it runs the next script).

Thanks to Steven for excelent hint.
The patch bellow fixes it for me.
Please could also other people verify it.

Petr


--- sysvinit-2.88dsf.orig/startpar/startpar.c
+++ sysvinit-2.88dsf/startpar/startpar.c
@@ -1121,10 +1121,11 @@ int main(int argc, char **argv)
       exit(1);
     }
 #endif
-
+#ifdef __linux__
   /* lock us into memory */
   if (geteuid() == 0)
     mlockall(MCL_CURRENT|MCL_FUTURE);
+#endif
   errno = 0;

   gettimeofday(&glastio, 0);




Added tag(s) patch. Request was from Petr Salinger <Petr.Salinger@seznam.cz> to control@bugs.debian.org. (Tue, 21 Aug 2012 08:00:07 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Tue, 21 Aug 2012 08:51:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Axel Beckert <abe@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Tue, 21 Aug 2012 08:51:02 GMT) Full text and rfc822 format available.

Message #138 received at 672959@bugs.debian.org (full text, mbox):

From: Axel Beckert <abe@debian.org>
To: 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: Bug#672959: [patch] Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Tue, 21 Aug 2012 10:47:57 +0200
Hi,

Petr Salinger wrote:
> >I'm beginning to think that startpar is malfunctioning in some way
> >(after checkroot.sh returns, but before it runs the next script).
> 
> Thanks to Steven for excelent hint.

Indeed. That fits perfectly with my observation that always the last
thing I saw before the crash was the ":" from the last line of
checkroot.sh. No trace of another init.d script being started.

> The patch bellow fixes it for me.
> Please could also other people verify it.

Will do this evening. Thanks Petr!

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Tue, 21 Aug 2012 12:30:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steven Chamberlain <steven@pyro.eu.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Tue, 21 Aug 2012 12:30:03 GMT) Full text and rfc822 format available.

Message #143 received at 672959@bugs.debian.org (full text, mbox):

From: Steven Chamberlain <steven@pyro.eu.org>
To: Petr Salinger <Petr.Salinger@seznam.cz>
Cc: 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: Bug#672959: startpar triggers kfreebsd panic: vm_fault_copy_wired
Date: Tue, 21 Aug 2012 13:27:22 +0100
retitle 672959 startpar triggers kfreebsd panic: vm_fault_copy_wired
thanks

On 21/08/12 09:16, Petr Salinger wrote:
>> I'm beginning to think that startpar is malfunctioning in some way
>> (after checkroot.sh returns, but before it runs the next script).
> 
> Thanks to Steven for excelent hint.

I'm just happy my mail was coherent at that hour.

Thank you for a brilliant patch.  With it I'm no longer seeing panics on
kfreebsd-i386

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org



Changed Bug title to 'startpar triggers kfreebsd panic: vm_fault_copy_wired' from 'kfreebsd-*: panic: vm_fault_copy_wired' Request was from Steven Chamberlain <steven@pyro.eu.org> to control@bugs.debian.org. (Tue, 21 Aug 2012 12:30:05 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Tue, 21 Aug 2012 21:45:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Roger Leigh <rleigh@codelibre.net>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Tue, 21 Aug 2012 21:45:03 GMT) Full text and rfc822 format available.

Message #150 received at 672959@bugs.debian.org (full text, mbox):

From: Roger Leigh <rleigh@codelibre.net>
To: 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: Bug#672959: [patch] Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Tue, 21 Aug 2012 22:43:55 +0100
On Tue, Aug 21, 2012 at 10:47:57AM +0200, Axel Beckert wrote:
> Hi,
> 
> Petr Salinger wrote:
> > >I'm beginning to think that startpar is malfunctioning in some way
> > >(after checkroot.sh returns, but before it runs the next script).
> > 
> > Thanks to Steven for excelent hint.
> 
> Indeed. That fits perfectly with my observation that always the last
> thing I saw before the crash was the ":" from the last line of
> checkroot.sh. No trace of another init.d script being started.
> 
> > The patch bellow fixes it for me.
> > Please could also other people verify it.
> 
> Will do this evening. Thanks Petr!

Many thanks Petr for the patch, and everyone else for testing and
investigating.

I've put a test package here:
  http://people.debian.org/~rleigh/sysvinit/sysvinit_2.88dsf-33.dsc

I'd be grateful if anyone could build this and double-check that this
is correct, and fixes the bug.  I'll upload this as soon as that's
done.


Thanks again,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux    http://people.debian.org/~rleigh/
 `. `'   schroot and sbuild  http://alioth.debian.org/projects/buildd-tools
   `-    GPG Public Key      F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Tue, 21 Aug 2012 22:48:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Axel Beckert <abe@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Tue, 21 Aug 2012 22:48:04 GMT) Full text and rfc822 format available.

Message #155 received at 672959@bugs.debian.org (full text, mbox):

From: Axel Beckert <abe@debian.org>
To: 672959@bugs.debian.org
Cc: debian-bsd@lists.debian.org
Subject: Re: Bug#672959: [patch] Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Wed, 22 Aug 2012 00:46:37 +0200
Hi Roger,

Roger Leigh wrote:
> I've put a test package here:
>   http://people.debian.org/~rleigh/sysvinit/sysvinit_2.88dsf-33.dsc
> 
> I'd be grateful if anyone could build this and double-check that this
> is correct, and fixes the bug.  I'll upload this as soon as that's
> done.

Works for me on kfreebsd-i386 after a "touch /forcefsck". No more
crash.

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5



Information forwarded to debian-bugs-dist@lists.debian.org, Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>:
Bug#672959; Package src:sysvinit. (Tue, 21 Aug 2012 23:30:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Steven Chamberlain <steven@pyro.eu.org>:
Extra info received and forwarded to list. Copy sent to Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>. (Tue, 21 Aug 2012 23:30:03 GMT) Full text and rfc822 format available.

Message #160 received at 672959@bugs.debian.org (full text, mbox):

From: Steven Chamberlain <steven@pyro.eu.org>
To: Roger Leigh <rleigh@codelibre.net>
Cc: 672959@bugs.debian.org, debian-bsd@lists.debian.org
Subject: Re: Bug#672959: [patch] Bug#672959: kfreebsd-*: panic: vm_fault_copy_wired
Date: Wed, 22 Aug 2012 00:26:39 +0100
Hi!

On 21/08/12 22:43, Roger Leigh wrote:
> I've put a test package here:
>   http://people.debian.org/~rleigh/sysvinit/sysvinit_2.88dsf-33.dsc
> 
> I'd be grateful if anyone could build this [...]

That works okay, even with a genuinely dirty rootfs where fsck carries
out a repair.  I'm using kfreebsd-i386 as well.

Thanks!
Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org



Reply sent to Roger Leigh <rleigh@debian.org>:
You have taken responsibility. (Fri, 31 Aug 2012 22:36:04 GMT) Full text and rfc822 format available.

Notification sent to Steven Chamberlain <steven@pyro.eu.org>:
Bug acknowledged by developer. (Fri, 31 Aug 2012 22:36:04 GMT) Full text and rfc822 format available.

Message #165 received at 672959-close@bugs.debian.org (full text, mbox):

From: Roger Leigh <rleigh@debian.org>
To: 672959-close@bugs.debian.org
Subject: Bug#672959: fixed in sysvinit 2.88dsf-32
Date: Fri, 31 Aug 2012 22:32:43 +0000
Source: sysvinit
Source-Version: 2.88dsf-32

We believe that the bug you reported is fixed in the latest version of
sysvinit, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 672959@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Roger Leigh <rleigh@debian.org> (supplier of updated sysvinit package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Fri, 31 Aug 2012 23:03:14 +0100
Source: sysvinit
Binary: sysvinit sysvinit-utils sysv-rc initscripts bootlogd
Architecture: source amd64 all
Version: 2.88dsf-32
Distribution: unstable
Urgency: low
Maintainer: Debian sysvinit maintainers <pkg-sysvinit-devel@lists.alioth.debian.org>
Changed-By: Roger Leigh <rleigh@debian.org>
Description: 
 bootlogd   - daemon to log boot messages
 initscripts - scripts for initializing and shutting down the system
 sysv-rc    - System-V-like runlevel change mechanism
 sysvinit   - System-V-like init utilities
 sysvinit-utils - System-V-like utilities
Closes: 672959 685212
Changes: 
 sysvinit (2.88dsf-32) unstable; urgency=low
 .
   [ Roger Leigh ]
   * Apply patch to startpar to prevent the use of mlockall on
     kFreeBSD, which was causing a kernel panic at boot.  Thanks to
     Petr Salinger for this patch.  Closes: #672959.
   * Correctly detect service availability with service(8).  Thanks
     to Alexander Golov for this patch.  Closes: #685212.
Checksums-Sha1: 
 a28587046ec48371fcb1edf87e5cbf8754b051fc 2342 sysvinit_2.88dsf-32.dsc
 e7f1e2d13d0160041b27553fe6acc64f9b495378 208284 sysvinit_2.88dsf-32.debian.tar.gz
 ee2fcdc80f3d9c6aded126464014437a75db7564 132028 sysvinit_2.88dsf-32_amd64.deb
 c753bd3e83a402dc38a4035354efcc362e91f2f6 98384 sysvinit-utils_2.88dsf-32_amd64.deb
 50f66fba3faee043e07971580baa25914e6b8def 79964 sysv-rc_2.88dsf-32_all.deb
 f94db4a59cae668e6b97175f6ce9ddf364e3c66d 92226 initscripts_2.88dsf-32_amd64.deb
 bc37490905d1c061c93b540a01c3a7d866da1e33 54094 bootlogd_2.88dsf-32_amd64.deb
Checksums-Sha256: 
 116f98e5b09b74ca77524869411f59942ad6f0a4780ee676b47b24c2a0411124 2342 sysvinit_2.88dsf-32.dsc
 d08abe9070aad2ef78162bce392e97eb8b51e77551a4ca7c18e5ce85fd061488 208284 sysvinit_2.88dsf-32.debian.tar.gz
 a0e667fab59f4b200a88b72063ded3dafc15f78e768e09a177709eb19ce440e2 132028 sysvinit_2.88dsf-32_amd64.deb
 2989adf99067a54652eddde217a3fb6bce132e4e036323be5c97e2fb0eeb529c 98384 sysvinit-utils_2.88dsf-32_amd64.deb
 b668bef03c790b7eb472d981b829849f833e315d0154a5c36959ecd79dda589d 79964 sysv-rc_2.88dsf-32_all.deb
 4ac3809a02873763b0b5e7d04b908160dc5d4602004599c52434468d668945ef 92226 initscripts_2.88dsf-32_amd64.deb
 45d63787a94be57d5aeaafdd6c8f05cc5b00900a4e774f48226191e6ec85a684 54094 bootlogd_2.88dsf-32_amd64.deb
Files: 
 d69f13289fd805214506d63403ec4ab6 2342 admin required sysvinit_2.88dsf-32.dsc
 c070088279da3497c0ca378e4e03c465 208284 admin required sysvinit_2.88dsf-32.debian.tar.gz
 55a42184c90af6f1d408d66ac3dff067 132028 admin required sysvinit_2.88dsf-32_amd64.deb
 1a18e487864d8aafb64679437fd25ee4 98384 admin required sysvinit-utils_2.88dsf-32_amd64.deb
 f44cb033fb0318d1ca3e961d8eac7306 79964 admin required sysv-rc_2.88dsf-32_all.deb
 19bbac6254a716cdeb88a4c53eeb11b8 92226 admin required initscripts_2.88dsf-32_amd64.deb
 b807a570f325812ec32d9f3b51532631 54094 admin optional bootlogd_2.88dsf-32_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBCgAGBQJQQTcoAAoJEOJSSsUKn1xZr/sQAJuiqp6bsDUbd0i/ox0Wx7Ox
aYJx92VhUy9YySVxGv83OJ1Q6SqMgo9lr8+acYrrYVb7w3vHB7zSao3gMxwhD7K4
cujZOOWe3Y9w4HGp8Gj9JNyEAH102hSGDKL+NhconV0CfBu4WwGFRyB0a00cB4SC
v337woVSQZU6MXb3dKo87RAQs9VWazq5w3mi5+FXYzXH/paV2cYG3L7ynQumzmcU
VrjzfB7JvjhUw5j9PgBhfUutllcUo9F0j+lbxq5g+IMSAhZ+gT3EjKXB+kF1aiMo
rHgutj2O8uimdwFy0gTBSMPm/KFQv1mVvCGhRbYr/E2V59vZKq+Cfur+t3wbc0dC
Ms8DSrFOSQviRm/h/O1Rn7XDyEXzFQEhUljuEl4koR7reEpSyOi1MInRd3SI56vK
9c2k1rfxXcK31o0qljF9CJ6bSpIiqOTXSj4OLUkAFU33P0euaK7EZvxUxgNQ7tOR
3wSEGbOpcb2A9ejE8GR7kUr6PioLX4oYcBWYkmyGmNAXkX5CKOp0n8y+u+efXOx4
PkZTJgcX5VVNkbQ8bxZpXabtfnK/O4KecElY6FQKIwWK4QpHZTv9yNW17aLTycoU
jbU3BZI/JXtvg+nfYgn1nvia6OaYq71cNWebqtrSg4N9V3J5nJ0PkQTLckg3rJn5
i0EL/ztpa95d+xw3hvnT
=VzFp
-----END PGP SIGNATURE-----




Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Tue, 09 Oct 2012 07:28:25 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Sun Apr 20 06:06:34 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.