Debian Bug report logs - #684661
Xen panic on boot

version graph

Package: xen-hypervisor-4.1-amd64; Maintainer for xen-hypervisor-4.1-amd64 is Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>; Source for xen-hypervisor-4.1-amd64 is src:xen.

Reported by: Stephan Beyer <s-beyer@gmx.net>

Date: Sun, 12 Aug 2012 15:27:04 UTC

Severity: important

Tags: moreinfo

Found in version xen/4.1.3~rc1+hg-20120614.a9c0a89c08f2-5

Fixed in version 4.1.4-1

Done: Bastian Blank <waldi@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#684661; Package xen-hypervisor-4.1-amd64. (Sun, 12 Aug 2012 15:27:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to Stephan Beyer <s-beyer@gmx.net>:
New Bug report received and forwarded. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Sun, 12 Aug 2012 15:27:07 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Stephan Beyer <s-beyer@gmx.net>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: Xen panic on boot
Date: Sun, 12 Aug 2012 17:25:50 +0200
Package: xen-hypervisor-4.1-amd64
Version: 4.1.3~rc1+hg-20120614.a9c0a89c08f2-5
Severity: important

I updated from xen-hypervisor-4.0-amd64 to xen-hypervisor-4.1-amd64
and the system did not boot anymore (Xen panic).
A downgrade made the system work again.
I cannot paste the boot error messages but the main message was:

	Xen BUG at pci_amd_iommu.c:33

The bug appears for Linux images 3.2.0.2 from squeeze and for 3.4-trunk
and 3.5-trunk from experimental.

Best
  Stephan

PS: Furthermore note that on Linux 3.2.0.2, xen-hypervisor-4.0-amd64
floods the console/logs with powernow-k8 transition error messages, see

 * http://bugzilla.xen.org/bugzilla/show_bug.cgi?id=1789
 * http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=373947

However, Linux image 3.5-trunk and xen-hypervisor-4.0-amd64 seem to
work nicely.



Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#684661; Package xen-hypervisor-4.1-amd64. (Fri, 17 Aug 2012 11:51:14 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Campbell <ijc@hellion.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Fri, 17 Aug 2012 11:51:14 GMT) Full text and rfc822 format available.

Message #10 received at 684661@bugs.debian.org (full text, mbox):

From: Ian Campbell <ijc@hellion.org.uk>
To: Stephan Beyer <s-beyer@gmx.net>, 684661@bugs.debian.org
Subject: Re: [Pkg-xen-devel] Bug#684661: Xen panic on boot
Date: Fri, 17 Aug 2012 12:50:19 +0100
On Sun, 2012-08-12 at 17:25 +0200, Stephan Beyer wrote:
> Package: xen-hypervisor-4.1-amd64
> Version: 4.1.3~rc1+hg-20120614.a9c0a89c08f2-5
> Severity: important
> 
> I updated from xen-hypervisor-4.0-amd64 to xen-hypervisor-4.1-amd64
> and the system did not boot anymore (Xen panic).
> A downgrade made the system work again.
> I cannot paste the boot error messages but the main message was:
> 
> 	Xen BUG at pci_amd_iommu.c:33

The BUG is "BUG_ON ( bdf >= ivrs_bdf_entries );" and ivrs_bdf_entries
has come from ACPI -- do you have the latest firmware in this machine?

I'm afraid that without some more information (e.g. a stack trace) I
think this is going to be pretty hard to track down.

Can you either configure a serial console
(http://wiki.xen.org/wiki/Xen_Serial_Console) or use the "noreboot"
option on the hypervisor command line to capture a digital picture of
the error.

Thanks,
Ian.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#684661; Package xen-hypervisor-4.1-amd64. (Fri, 17 Aug 2012 11:57:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Campbell <Ian.Campbell@citrix.com>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Fri, 17 Aug 2012 11:57:06 GMT) Full text and rfc822 format available.

Message #15 received at 684661@bugs.debian.org (full text, mbox):

From: Ian Campbell <Ian.Campbell@citrix.com>
To: Wei Wang <wei.wang2@amd.com>, xen-devel <xen-devel@lists.xen.org>
Cc: <684661@bugs.debian.org>
Subject: Xen BUG at pci_amd_iommu.c:33
Date: Fri, 17 Aug 2012 12:46:22 +0100
Hi Wei,

A Debian user has hit this message and reported it in
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=684661

I think it is the BUG_ON in:
        struct amd_iommu *find_iommu_for_device(int bdf)
        {
            BUG_ON ( bdf >= ivrs_bdf_entries );
            return ivrs_mappings[bdf].iommu;
        }

It looks like ivrs_bdf_entries comes from ACPI. Unfortunately the bug
report is a bit vague about things like stack traces etc so it's hard to
say where BDF came from.

Any ideas? I'll also follow up to the submitter to try and get some more
details.

Ian.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#684661; Package xen-hypervisor-4.1-amd64. (Fri, 17 Aug 2012 12:27:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Wei Wang <wei.wang2@amd.com>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Fri, 17 Aug 2012 12:27:03 GMT) Full text and rfc822 format available.

Message #20 received at 684661@bugs.debian.org (full text, mbox):

From: Wei Wang <wei.wang2@amd.com>
To: Ian Campbell <Ian.Campbell@citrix.com>
Cc: xen-devel <xen-devel@lists.xen.org>, <684661@bugs.debian.org>
Subject: Re: [Xen-devel] Xen BUG at pci_amd_iommu.c:33
Date: Fri, 17 Aug 2012 14:10:22 +0200
On 08/17/2012 01:46 PM, Ian Campbell wrote:
> Hi Wei,
>
> A Debian user has hit this message and reported it in
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=684661
>
> I think it is the BUG_ON in:
>          struct amd_iommu *find_iommu_for_device(int bdf)
>          {
>              BUG_ON ( bdf>= ivrs_bdf_entries );
>              return ivrs_mappings[bdf].iommu;
>          }
>
> It looks like ivrs_bdf_entries comes from ACPI. Unfortunately the bug
> report is a bit vague about things like stack traces etc so it's hard to
> say where BDF came from.
>
> Any ideas? I'll also follow up to the submitter to try and get some more
> details.

It difficult to identify the issue from the description and I think a 
full serial boot log would be great. Also using iommu=debug as boot 
option is very helpful to get more information.

Thanks,
Wei

> Ian.
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#684661; Package xen-hypervisor-4.1-amd64. (Fri, 17 Aug 2012 12:51:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Campbell <ijc@hellion.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Fri, 17 Aug 2012 12:51:03 GMT) Full text and rfc822 format available.

Message #25 received at 684661@bugs.debian.org (full text, mbox):

From: Ian Campbell <ijc@hellion.org.uk>
To: 684661@bugs.debian.org
Cc: Stephan Beyer <s-beyer@gmx.net>
Subject: Re: [Pkg-xen-devel] Bug#684661: Bug#684661: Xen panic on boot
Date: Fri, 17 Aug 2012 13:49:52 +0100
On Fri, 2012-08-17 at 12:50 +0100, Ian Campbell wrote:
> On Sun, 2012-08-12 at 17:25 +0200, Stephan Beyer wrote:
> > Package: xen-hypervisor-4.1-amd64
> > Version: 4.1.3~rc1+hg-20120614.a9c0a89c08f2-5
> > Severity: important
> > 
> > I updated from xen-hypervisor-4.0-amd64 to xen-hypervisor-4.1-amd64
> > and the system did not boot anymore (Xen panic).
> > A downgrade made the system work again.
> > I cannot paste the boot error messages but the main message was:
> > 
> > 	Xen BUG at pci_amd_iommu.c:33
> 
> The BUG is "BUG_ON ( bdf >= ivrs_bdf_entries );" and ivrs_bdf_entries
> has come from ACPI -- do you have the latest firmware in this machine?
> 
> I'm afraid that without some more information (e.g. a stack trace) I
> think this is going to be pretty hard to track down.
> 
> Can you either configure a serial console
> (http://wiki.xen.org/wiki/Xen_Serial_Console) or use the "noreboot"
> option on the hypervisor command line to capture a digital picture of
> the error.

Please add iommu=debug to the hypervisor command line too.

Ian.





Added tag(s) moreinfo. Request was from Ian Campbell <ijc@hellion.org.uk> to control@bugs.debian.org. (Tue, 28 Aug 2012 08:15:06 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#684661; Package xen-hypervisor-4.1-amd64. (Tue, 04 Sep 2012 00:27:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Joe Julian <joej@edwyse.com>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Tue, 04 Sep 2012 00:27:03 GMT) Full text and rfc822 format available.

Message #32 received at 684661@bugs.debian.org (full text, mbox):

From: Joe Julian <joej@edwyse.com>
To: 684661@bugs.debian.org
Subject: Xen panic on boot
Date: Mon, 03 Sep 2012 16:57:56 -0700
[Message part 1 (text/plain, inline)]
I'm also getting the same error. I don't have anything with serial 
ports, but hopefully this will be of some use.

[IMG_20120903_165005.jpg (image/jpeg, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#684661; Package xen-hypervisor-4.1-amd64. (Tue, 04 Sep 2012 08:27:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Campbell <ijc@hellion.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Tue, 04 Sep 2012 08:27:03 GMT) Full text and rfc822 format available.

Message #37 received at 684661@bugs.debian.org (full text, mbox):

From: Ian Campbell <ijc@hellion.org.uk>
To: Joe Julian <joej@edwyse.com>, 684661@bugs.debian.org
Cc: xen-devel <xen-devel@lists.xen.org>, Wei Wang <wei.wang2@amd.com>
Subject: Re: [Pkg-xen-devel] Bug#684661: Xen panic on boot
Date: Tue, 04 Sep 2012 09:24:24 +0100
[Message part 1 (text/plain, inline)]
On Mon, 2012-09-03 at 16:57 -0700, Joe Julian wrote:
> I'm also getting the same error. I don't have anything with serial 
> ports, but hopefully this will be of some use.

Thanks.

Wei does this give any hints to this bug? This is the Xen BUG at
pci_amd_iommu.c:33 one I CCd you on before http://bugs.debian.org/684661

Joe, are you able to press Shift-PgUp to scroll up to the top part of
the trace? (I must confess I'm not sure that works for Xen, perhaps its
a Linux-ism).

Looks like you are already using the vga= option to increase the number
of lines displayed.

Ian.

-- 
Ian Campbell
Current Noise: High On Fire - Speak In Tongues

QOTD:
	How can I miss you if you won't go away?
[IMG_20120903_165005.jpg (image/jpeg, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#684661; Package xen-hypervisor-4.1-amd64. (Tue, 04 Sep 2012 09:12:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Joe Julian <joej@edwyse.com>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Tue, 04 Sep 2012 09:12:03 GMT) Full text and rfc822 format available.

Message #42 received at 684661@bugs.debian.org (full text, mbox):

From: Joe Julian <joej@edwyse.com>
To: Ian Campbell <ijc@hellion.org.uk>
Cc: 684661@bugs.debian.org, xen-devel <xen-devel@lists.xen.org>, Wei Wang <wei.wang2@amd.com>
Subject: Re: [Pkg-xen-devel] Bug#684661: Xen panic on boot
Date: Tue, 04 Sep 2012 02:09:46 -0700
On 09/04/2012 01:24 AM, Ian Campbell wrote:
> On Mon, 2012-09-03 at 16:57 -0700, Joe Julian wrote:
>> I'm also getting the same error. I don't have anything with serial
>> ports, but hopefully this will be of some use.
> Thanks.
>
> Wei does this give any hints to this bug? This is the Xen BUG at
> pci_amd_iommu.c:33 one I CCd you on before http://bugs.debian.org/684661
>
> Joe, are you able to press Shift-PgUp to scroll up to the top part of
> the trace? (I must confess I'm not sure that works for Xen, perhaps its
> a Linux-ism).
>
> Looks like you are already using the vga= option to increase the number
> of lines displayed.
>
> Ian.
>
No, once it's hung there it's stuck (doesn't even reboot in 5 seconds 
like it claims).

This was actually the Fedora 17 build, 4.1.3-2 and coincided with me 
having to remove 2 gig (bringing it down to 2 remaining gig) from an 
Asus K8N-DL due to memory failure. I tried rolling back to 4.1.2-15 with 
the same result. I found an old reference to this motherboard suggesting 
that with less than 3 gig, there's an iommu problem: 
http://www.gentoo-wiki.info/Asus_K8N-DL

I'll get this memory replaced and see if that has any effect.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#684661; Package xen-hypervisor-4.1-amd64. (Sun, 23 Sep 2012 13:39:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to jagi2mail <jagi2mail@gmail.com>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Sun, 23 Sep 2012 13:39:03 GMT) Full text and rfc822 format available.

Message #47 received at 684661@bugs.debian.org (full text, mbox):

From: jagi2mail <jagi2mail@gmail.com>
To: 684661@bugs.debian.org
Subject: Xen panic on boot
Date: Sun, 23 Sep 2012 21:25:38 +0800
also get this message

machine locks and have to reboot

prior to putting xen had been running squeeze with no problems.

installed wheezy from new and got error.

upgraded to sid and error still occurs

has 4GB so not the memory as mentioned by another

on a AMD opteron 275
hp proliant dl385

xen 4.1

to do more checking




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#684661; Package xen-hypervisor-4.1-amd64. (Sun, 23 Sep 2012 15:06:29 GMT) Full text and rfc822 format available.

Acknowledgement sent to Thomas Goirand <thomas@goirand.fr>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Sun, 23 Sep 2012 15:06:29 GMT) Full text and rfc822 format available.

Message #52 received at 684661@bugs.debian.org (full text, mbox):

From: Thomas Goirand <thomas@goirand.fr>
To: jagi2mail <jagi2mail@gmail.com>, 684661@bugs.debian.org
Subject: Re: [Pkg-xen-devel] Bug#684661: Xen panic on boot
Date: Sun, 23 Sep 2012 23:03:15 +0800
On 09/23/2012 09:25 PM, jagi2mail wrote:
> also get this message
>
> machine locks and have to reboot
>
> prior to putting xen had been running squeeze with no problems.
>
> installed wheezy from new and got error.
>
> upgraded to sid and error still occurs
>
> has 4GB so not the memory as mentioned by another
>
> on a AMD opteron 275
> hp proliant dl385
>
> xen 4.1
>
> to do more checking

Hi,

This type of server has a serial port. Could you hook the serial of 
another server and do a serial dump of the crash?

It's explained here under "Enable Serial Console" how to configure Xen 
under Debian with a serial console:
http://wiki.debian.org/Xen

We are particularly interested to know in which line & file the server 
crashes, so we can locate where the issue is (xen hypervisor or, more 
likely, linux kernel). Without this, it's impossible to know what the 
issue is about.

Cheers,

Thomas



Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#684661; Package xen-hypervisor-4.1-amd64. (Mon, 25 Feb 2013 18:09:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Christian Sünkenberg <christian.suenkenberg@student.kit.edu>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Mon, 25 Feb 2013 18:09:03 GMT) Full text and rfc822 format available.

Message #57 received at 684661@bugs.debian.org (full text, mbox):

From: Christian Sünkenberg <christian.suenkenberg@student.kit.edu>
To: <684661@bugs.debian.org>
Subject: Workaround, patch and full trace
Date: Mon, 25 Feb 2013 18:44:20 +0100
[Message part 1 (text/plain, inline)]
Hi,

I've also recently hit this bug und it should be fixed by [1],
also see the original analysis at [2].

Passing "allow_unsafe" as a Hypervisor option circumvents the crash.

A serial boot log is attached.

[1]: http://lists.xen.org/archives/html/xen-devel/2012-10/msg01856.html
[2]: http://lists.xen.org/archives/html/xen-devel/2012-10/msg01396.html

Regards,
Christian
[bootlog-xen-4.1-opteron-175-iommu-crash.log (text/plain, attachment)]
[smime.p7s (application/pkcs7-signature, attachment)]

Reply sent to Bastian Blank <waldi@debian.org>:
You have taken responsibility. (Mon, 25 Feb 2013 18:27:09 GMT) Full text and rfc822 format available.

Notification sent to Stephan Beyer <s-beyer@gmx.net>:
Bug acknowledged by developer. (Mon, 25 Feb 2013 18:27:09 GMT) Full text and rfc822 format available.

Message #62 received at 684661-done@bugs.debian.org (full text, mbox):

From: Bastian Blank <waldi@debian.org>
To: 684661-done@bugs.debian.org
Subject: Re: [Pkg-xen-devel] Bug#684661: Xen panic on boot
Date: Mon, 25 Feb 2013 19:22:57 +0100
Version: 4.1.4-1

On Sun, Aug 12, 2012 at 05:25:50PM +0200, Stephan Beyer wrote:
> I updated from xen-hypervisor-4.0-amd64 to xen-hypervisor-4.1-amd64
> and the system did not boot anymore (Xen panic).
> A downgrade made the system work again.
> I cannot paste the boot error messages but the main message was:
> 
> 	Xen BUG at pci_amd_iommu.c:33

But you could have read it. This is "fixed" differently in 4.1.4. It
will not longer refuse to boot, but to start additional domains.

Bastian

-- 
Spock: The odds of surviving another attack are 13562190123 to 1, Captain.



Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Tue, 26 Mar 2013 07:29:29 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Wed Apr 23 14:29:44 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.