Debian Bug report logs - #620216
linux-image-2.6.38-1-686: kernel hangs with XFS

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Thomas Prokosch <tprokos+debbugs@gmail.com>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: linux-image-2.6.38-1-686: kernel hangs with XFS

Date: Thu, 31 Mar 2011 08:25:05 +0200

[Message part 1 (text/plain, inline)]

Package: linux-2.6
Version: 2.6.38-1
Severity: important

Since the big XFS refacturing, kernels tend to hang on one particular machine. It is not a hardware problem, since the very same machine works flawlessly with the very same configuration, by just using an older (<=2.6.26) kernel.

Especially bad in this respect is the latest 2.6.38 as given in the email subject: The problem appears right after the reboot. I am filing this bug report on linux-image-2.6.37-1-686 which also shows hangs but only 1-2 times per week.

In order to isolate this bug, I am attending the kernel log for two distinct hangs (with a reboot in between) and disk information. I suspect that the problem occurs only on special types of disks, as I run almost the same configuration on multiple machines (same kernel, XFS formatted disks with the same XFS format version, same partitioning, etc.) without having any problems.

If you need any more info, I will be more than glad to help.

Best,
Thomas

-- Package-specific info:
** Kernel log: boot messages should be attached

** Model information
not available

** PCI devices:
00:00.0 Host bridge [0600]: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express Memory Controller Hub [8086:27a0] (rev 03)
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
	Latency: 0
	Capabilities: <access denied>
	Kernel driver in use: agpgart-intel

00:02.0 VGA compatible controller [0300]: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller [8086:27a2] (rev 03) (prog-if 00 [VGA controller])
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at d4100000 (32-bit, non-prefetchable) [size=512K]
	Region 1: I/O ports at 1800 [size=8]
	Region 2: Memory at c0000000 (32-bit, prefetchable) [size=256M]
	Region 3: Memory at d4200000 (32-bit, non-prefetchable) [size=256K]
	Expansion ROM at <unassigned> [disabled]
	Capabilities: <access denied>
	Kernel driver in use: i915

00:02.1 Display controller [0380]: Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller [8086:27a6] (rev 03)
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Region 0: Memory at d4180000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: <access denied>

00:1b.0 Audio device [0403]: Intel Corporation N10/ICH 7 Family High Definition Audio Controller [8086:27d8] (rev 02)
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 41
	Region 0: Memory at d4240000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: HDA Intel

00:1c.0 PCI bridge [0604]: Intel Corporation N10/ICH 7 Family PCI Express Port 1 [8086:27d0] (rev 02) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=02, subordinate=05, sec-latency=0
	I/O behind bridge: 00002000-00002fff
	Memory behind bridge: d2000000-d3ffffff
	Prefetchable memory behind bridge: 00000000d0000000-00000000d1ffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>

00:1d.0 USB Controller [0c03]: Intel Corporation N10/ICH 7 Family USB UHCI Controller #1 [8086:27c8] (rev 02) (prog-if 00 [UHCI])
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 17
	Region 4: I/O ports at 1820 [size=32]
	Kernel driver in use: uhci_hcd

00:1d.1 USB Controller [0c03]: Intel Corporation N10/ICH 7 Family USB UHCI Controller #2 [8086:27c9] (rev 02) (prog-if 00 [UHCI])
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 19
	Region 4: I/O ports at 1840 [size=32]
	Kernel driver in use: uhci_hcd

00:1d.2 USB Controller [0c03]: Intel Corporation N10/ICH 7 Family USB UHCI Controller #3 [8086:27ca] (rev 02) (prog-if 00 [UHCI])
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin C routed to IRQ 21
	Region 4: I/O ports at 1860 [size=32]
	Kernel driver in use: uhci_hcd

00:1d.3 USB Controller [0c03]: Intel Corporation N10/ICH 7 Family USB UHCI Controller #4 [8086:27cb] (rev 02) (prog-if 00 [UHCI])
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 17
	Region 4: I/O ports at 1880 [size=32]
	Kernel driver in use: uhci_hcd

00:1d.7 USB Controller [0c03]: Intel Corporation N10/ICH 7 Family USB2 EHCI Controller [8086:27cc] (rev 02) (prog-if 20 [EHCI])
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin D routed to IRQ 23
	Region 0: Memory at d4444000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: <access denied>
	Kernel driver in use: ehci_hcd

00:1e.0 PCI bridge [0604]: Intel Corporation 82801 Mobile PCI Bridge [8086:2448] (rev e2) (prog-if 01 [Subtractive decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Bus: primary=00, secondary=06, subordinate=0a, sec-latency=64
	I/O behind bridge: 00003000-00003fff
	Memory behind bridge: d4000000-d40fffff
	Prefetchable memory behind bridge: 0000000040000000-0000000043ffffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>

00:1f.0 ISA bridge [0601]: Intel Corporation 82801GBM (ICH7-M) LPC Interface Bridge [8086:27b9] (rev 02)
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Capabilities: <access denied>

00:1f.1 IDE interface [0101]: Intel Corporation 82801G (ICH7 Family) IDE Controller [8086:27df] (rev 02) (prog-if 8a [Master SecP PriP])
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin B routed to IRQ 22
	Region 0: I/O ports at 01f0 [size=8]
	Region 1: I/O ports at 03f4 [size=1]
	Region 2: I/O ports at 0170 [size=8]
	Region 3: I/O ports at 0374 [size=1]
	Region 4: I/O ports at 1810 [size=16]
	Kernel driver in use: ata_piix

00:1f.3 SMBus [0c05]: Intel Corporation N10/ICH 7 Family SMBus Controller [8086:27da] (rev 02)
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Region 4: I/O ports at 18a0 [size=32]

02:00.0 Network controller [0280]: Intel Corporation PRO/Wireless 3945ABG [Golan] Network Connection [8086:4222] (rev 02)
	Subsystem: Intel Corporation Device [8086:1051]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 40
	Region 0: Memory at d2000000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: iwl3945

06:04.0 CardBus bridge [0607]: Texas Instruments PCIxx12 Cardbus Controller [104c:8039]
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 168, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 20
	Region 0: Memory at d4007000 (32-bit, non-prefetchable) [size=4K]
	Bus: primary=06, secondary=07, subordinate=0a, sec-latency=176
	Memory window 0: 40000000-43fff000 (prefetchable)
	Memory window 1: 44000000-47fff000
	I/O window 0: 00003400-000034ff
	I/O window 1: 00003800-000038ff
	BridgeCtl: Parity- SERR- ISA- VGA- MAbort- >Reset+ 16bInt+ PostWrite+
	16-bit legacy interface ports at 0001
	Kernel driver in use: yenta_cardbus

06:04.1 FireWire (IEEE 1394) [0c00]: Texas Instruments PCIxx12 OHCI Compliant IEEE 1394 Host Controller [104c:803a] (prog-if 10 [OHCI])
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32 (750ns min, 1000ns max), Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 21
	Region 0: Memory at d4006000 (32-bit, non-prefetchable) [size=2K]
	Region 1: Memory at d4000000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: firewire_ohci

06:04.2 Mass storage controller [0180]: Texas Instruments 5-in-1 Multimedia Card Reader (SD/MMC/MS/MS PRO/xD) [104c:803b]
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 57 (1750ns min, 1000ns max), Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 22
	Region 0: Memory at d4004000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: tifm_7xx1

06:08.0 Ethernet controller [0200]: Intel Corporation PRO/100 VM Network Connection [8086:1093] (rev 02)
	Subsystem: Sony Corporation Device [104d:8207]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 66 (2000ns min, 14000ns max), Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 20
	Region 0: Memory at d4005000 (32-bit, non-prefetchable) [size=4K]
	Region 1: I/O ports at 3000 [size=64]
	Capabilities: <access denied>
	Kernel driver in use: e100


** USB devices:
Bus 005 Device 002: ID 044e:300c Alps Electric Co., Ltd Bluetooth Controller (ALPS/UGPZ6)
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 002: ID 0483:2016 SGS Thomson Microelectronics Fingerprint Reader
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub


-- System Information:
Debian Release: 6.0.1
  APT prefers stable
  APT policy: (990, 'stable'), (500, 'squeeze-updates'), (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.37-1-686 (SMP w/1 CPU core)
Locale: LANG=en_US.ISO-8859-15, LC_CTYPE=en_US.ISO-8859-15 (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to C)
Shell: /bin/sh linked to /bin/dash

Versions of packages linux-image-2.6.38-1-686 depends on:
ii  debconf [debconf-2.0]         1.5.36.1   Debian configuration management sy
ii  initramfs-tools [linux-initra 0.98.8     tools for generating an initramfs
ii  linux-base                    3.1        Linux image base package
ii  module-init-tools             3.12-1     tools for managing Linux kernel mo

Versions of packages linux-image-2.6.38-1-686 recommends:
ii  firmware-linux-free           2.6.32-31  Binary firmware for various driver
ii  libc6-i686                    2.11.2-10  Embedded GNU C Library: Shared lib

Versions of packages linux-image-2.6.38-1-686 suggests:
ii  grub-pc                 1.98+20100804-14 GRand Unified Bootloader, version 
pn  linux-doc-2.6.38        <none>           (no description available)

Versions of packages linux-image-2.6.38-1-686 is related to:
pn  firmware-bnx2                 <none>     (no description available)
pn  firmware-bnx2x                <none>     (no description available)
pn  firmware-ipw2x00              <none>     (no description available)
pn  firmware-ivtv                 <none>     (no description available)
ii  firmware-iwlwifi              0.28       Binary firmware for Intel Wireless
pn  firmware-linux                <none>     (no description available)
ii  firmware-linux-nonfree        0.28       Binary firmware for various driver
pn  firmware-qlogic               <none>     (no description available)
pn  firmware-ralink               <none>     (no description available)
pn  xen-hypervisor                <none>     (no description available)

-- debconf information excluded

[hang1.log (text/plain, attachment)]

[hang2.log (text/plain, attachment)]

[disk-info.log (text/plain, attachment)]

Message #10 received at 620216@bugs.debian.org (full text, mbox, reply):

From: maximilian attems <max@stro.at>

To: Thomas Prokosch <tprokos+debbugs@gmail.com>, 620216@bugs.debian.org

Subject: Re: Bug#620216: linux-image-2.6.38-1-686: kernel hangs with XFS

Date: Thu, 31 Mar 2011 08:26:51 +0000

On Thu, Mar 31, 2011 at 08:25:05AM +0200, Thomas Prokosch wrote:
> Since the big XFS refacturing, kernels tend to hang on one particular machine. It is not a hardware problem, since the very same machine works flawlessly with the very same configuration, by just using an older (<=2.6.26) kernel.
> 
> Especially bad in this respect is the latest 2.6.38 as given in the email subject: The problem appears right after the reboot. I am filing this bug report on linux-image-2.6.37-1-686 which also shows hangs but only 1-2 times per week.
> 
> In order to isolate this bug, I am attending the kernel log for two distinct hangs (with a reboot in between) and disk information. I suspect that the problem occurs only on special types of disks, as I run almost the same configuration on multiple machines (same kernel, XFS formatted disks with the same XFS format version, same partitioning, etc.) without having any problems.
> 

I remember seeing a quite similar hang on 2.6.38 on a xftest run. please submit upstream on bugzilla.kernel.org
and let us know the bug nr.

thank you.

-- 
maks

Message #15 received at 620216@bugs.debian.org (full text, mbox, reply):

From: Thomas Prokosch <tprokos+debbugs@gmail.com>

To: 620216@bugs.debian.org

Subject: bugzilla.kernel.org

Date: Thu, 31 Mar 2011 15:46:26 +0200

Kernel bug number is 32342
https://bugzilla.kernel.org/show_bug.cgi?id=32342

Message #24 received at 620216@bugs.debian.org (full text, mbox, reply):

From: Jonathan Nieder <jrnieder@gmail.com>

To: Thomas Prokosch <tprokos+debbugs@gmail.com>

Cc: 620216@bugs.debian.org

Subject: Re: kernel hangs with XFS

Date: Mon, 6 Feb 2012 15:56:14 -0600

reassign 620216 src:linux-2.6 2.6.38-1
severity 655353 important
forwarded 655353 https://bugzilla.kernel.org/show_bug.cgi?id=32342
forcemerge 655353 620216
quit

Hi Thomas,

Thomas Prokosch wrote[1]:

> For reference purposes, the problem seems to be fixed:
> - http://comments.gmane.org/gmane.comp.file-systems.xfs.general/41907
> - http://bugs.debian.org/655353
> - https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=17b3847
>
> I am currently testing the new kernel, but so far the bug did not emerge.
[...]
> It seems that the above patch really fixed this bug - even under heavy load,
> the kernel does not stall any more.
 
Nice.  Thanks much!

It would be nice to get this fixed in the 2.6.32.y series, too.  Would
you be able to test the patch from <http://bugs.debian.org/655353#10>?
It works like this:

 0. Prerequisites:

	apt-get install git build-essential

 1. Grab a copy of the stable kernel series:

	git clone -o stable \
	 git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git \
	 linux
	cd linux

    Or, if you already have a clone of the kernel repository:

	cd linux
	git remote add -f stable \
	 git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

 2. Check out the 2.6.32.y series:

	git checkout stable/linux-2.6.32.y
	cp /boot/config-$(uname -r) .config; # use current configuration
	make localmodconfig; # optional: minimize configuration
	make; # optionally with -j<num> for parallel build
	fakeroot -u make deb-pkg
	dpkg -i ../<name of package>
	reboot

    Hopefully it reproduces the problem.

 3. Apply and test patch:

	git apply --index thepatch
	make; # maybe with -j4
	fakeroot -u make deb-pkg
	dpkg -i ../<name of package>
	reboot

If it works, we can apply the patch in squeeze and send the patch to
Greg for application to upstream 2.6.32.y kernels, so everyone
benefits.

If you have any questions, just ask.

Jonathan

Send a report that this bug log contains spam.