Debian Bug report logs - #584881
Lockups under heavy disk IO; md (RAID) resync/check implicated

version graph

Package: linux-2.6; Maintainer for linux-2.6 is Debian Kernel Team <debian-kernel@lists.debian.org>;

Reported by: Ian Jackson <ijackson@chiark.greenend.org.uk>

Date: Mon, 7 Jun 2010 09:57:02 UTC

Severity: normal

Tags: moreinfo, patch, pending

Found in version 2.6.26-21lenny4

Fixed in version linux-2.6/3.2.14-1

Done: Ben Hutchings <ben@decadent.org.uk>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-image-2.6.26-2-686-bigmem. (Mon, 07 Jun 2010 09:57:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
New Bug report received and forwarded. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Mon, 07 Jun 2010 09:57:05 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>
To: submit@bugs.debian.org
Subject: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Mon, 7 Jun 2010 10:53:17 +0100
[Message part 1 (text/plain, inline)]
Package: linux-image-2.6.26-2-686-bigmem
Version: 2.6.26-21lenny4

I keep getting soft lockups; the symptoms appear to be that user
processes become deadlocked when they try to access the disk, but the
kernel doesn't notice that anything is wrong.

It appears that the lockups happen when:
 (a) my software (mdadm) RAID mirror is resyncing or checking
 (b) some other program does heavy disk IO

I'm currently trying to work around the problem by disabling the
system's backups (which are a prime trigger in category (b)) until the
post-crash mirror resync is complete, combined with setting the sysctl
dev.raid.speed_limit_max to 1000.  Just disabling the backups wasn't
sufficient because, it seems, various recovery tasks in the most
recent crash are sufficient to trigger the bug.

I observe that the message from the md driver, and the presence of the
sysctls dev.raid.speed_limit_{min,max}, suggest that the md driver is
using a fundamentally incorrect algorithm which depends on guessing
the available IO bandwidth.


Actual hard data:

* chiark locked up at 0355 on the 2nd of May.  It did so shortly after
  the backups start.  chiark's backups are done with LVM snapshots and
  rsync.  I don't any more have information about exactly at what
  stage the backups were at.

* chiark locked up again at 0355 on Sunday morning the 6th of June.
  In this case I was able to see that the system had successfully
  created and mounted the LVM snapshot as the rsync process on the
  backup master system was still present.

* Note that both these two crashes happened on the 1st Sunday of the
  month.  Investigating the kernel logs for the 2nd crash show this:
    Jun  6 00:57:01 chiark kernel: [3250794.079478] md: data-check of RAID array md127
    Jun  6 00:57:01 chiark kernel: [3250794.079479] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
    Jun  6 00:57:01 chiark kernel: [3250794.079479] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
    Jun  6 00:57:01 chiark kernel: [3250794.079479] md: using 128k window, over a total of 222297980 blocks.
  However there is no sign of it completing.

* Shortly after I power cycled the system to get it back up, it locked
  up again.  At that point, the RAID was doing a post-crash resync,
  and the crash happened at the point where the backup master
  attempted to start on the next partition.

* Some research dug up some reports of first-Sunday-of-the-month
  crashes associated with RAID integrity checks.[1] So I tried setting
  the max IO bandwidth sysctl to the lowest reasonable value (by hand)
  and left the system running.  However, it locked up again as the
  backups started.

* I power cycled the machine again and disabled the backups and made
  sure they were truly stopped.  However chiark locked up again mere
  minutes after coming back up.

* I have now power cycled chiark once more and this time I set the
  max IO bandwidth sysctl.  So far it seems to have stayed up for
  about half an hour.

If nothing more goes wrong I intend to leave the backups disabled
until the RAID finishes its resync, which we can expect to be some
time on Wednesday night.


chiark is running LVM-on-MD-on-LVM.  So the two "disks" which form the
mirror which is resyncing are actually LVM LVs, and that mirror is
itself an LVM PV.  The underlying hardware is a Dell R210 with two
WD SATA disks (WD7500AYPS-0).


[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/212684
  seems relevant for example.


Ian.


lvs:

  LV                VG               Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  mirror-base       chiark-early-a   -wi-ao 400.00G
  stripe-base       chiark-early-a   -wi-ao  52.00G
  swap              chiark-early-a   -wi-ao   4.00G
  mirror-base       chiark-early-b   -wi-ao 400.00G
  stripe-base       chiark-early-b   -wi-ao  52.00G
  swap              chiark-early-b   -wi-ao   4.00G
  lv-quicksand      vg-chiark-stripe -wi-ao  40.00G
  twomack-relations vg-chiark-stripe -wi-ao  60.00G
  lv-log            vg-main          -wi-ao   3.00G
  lv-mail           vg-main          -wi-ao   3.00G
  lv-news           vg-main          -wi-ao  15.94G
  lv-root           vg-main          -wi-ao   1.00G
  lv-u              vg-main          -wi-ao   9.75G
  lv-u2             vg-main          -wi-ao  12.25G
  lv-u3             vg-main          -wi-ao   7.06G
  lv-usr            vg-main          -wi-ao   4.12G
  lv-var            vg-main          -wi-ao   5.94G
  lv-volt           vg-main          -wi-ao  77.44G


pvs:

  PV                    VG               Fmt  Attr PSize   PFree 
  /dev/md_chiark-mirror vg-main          lvm2 a-   211.94G 72.44G
  /dev/md_chiark-stripe vg-chiark-stripe lvm2 a-   103.94G  3.94G
  /dev/sdb2             chiark-early-a   lvm2 a-   464.00G  8.00G
  /dev/sdc2             chiark-early-b   lvm2 a-   464.00G  8.00G


/proc/mdstat:

Personalities : [raid0] [raid1] 
md126 : active raid0 dm-2[0] dm-5[1]
      109051776 blocks super 1.1 64k chunks
      
md127 : active raid1 dm-0[3] dm-3[2]
      222297980 blocks super 1.1 [2/2] [UU]
      [>....................]  resync =  0.9% (2014848/222297980) finish=3659.9min speed=1000K/sec


mdadm -D -vv --scan:

/dev/md127:
        Version : 01.01
  Creation Time : Sat Mar  6 20:06:57 2010
     Raid Level : raid1
     Array Size : 222297980 (212.00 GiB 227.63 GB)
  Used Dev Size : 444595960 (424.00 GiB 455.27 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Mon Jun  7 10:13:05 2010
          State : active, resyncing
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

 Rebuild Status : 0% complete

           Name : chiark:chiark-mirror  (local to host chiark)
           UUID : [***]
         Events : 9033

    Number   Major   Minor   RaidDevice State
       3     254        0        0      active sync   /dev/dm-0
       2     254        3        1      active sync   /dev/dm-3
/dev/md126:
        Version : 01.01
  Creation Time : Thu Mar 11 13:52:05 2010
     Raid Level : raid0
     Array Size : 109051776 (104.00 GiB 111.67 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 126
    Persistence : Superblock is persistent

    Update Time : Mon Jun  7 10:47:15 2010
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

     Chunk Size : 64K

           Name : chiark:chiark-stripe  (local to host chiark)
           UUID : [***]
         Events : 5

    Number   Major   Minor   RaidDevice State
       0     254        2        0      active sync   /dev/dm-2
       1     254        5        1      active sync   /dev/dm-5

brw-r----- 1 root disk 254, 0 Jun  7 10:13 /dev/dm-0
brw-r----- 1 root disk 254, 1 Jun  7 10:13 /dev/dm-1
brw-r----- 1 root disk 254, 2 Jun  7 10:13 /dev/dm-2
brw-r----- 1 root disk 254, 3 Jun  7 10:13 /dev/dm-3
brw-r----- 1 root disk 254, 4 Jun  7 10:13 /dev/dm-4
brw-r----- 1 root disk 254, 5 Jun  7 10:13 /dev/dm-5
brw-r----- 1 root disk 254, 6 Jun  7 10:13 /dev/dm-6
brw-r----- 1 root disk 254, 7 Jun  7 10:13 /dev/dm-7
brw-r----- 1 root disk 254, 8 Jun  7 10:13 /dev/dm-8

/dev/mapper/:
total 0
drwxr-xr-x  2 root root     420 Jun  7 10:43 ./
drwxr-xr-x 19 root root    3840 Jun  7 10:43 ../
brw-rw----  1 root disk 254,  0 Jun  7 10:13 chiark--early--a-mirror--base
brw-rw----  1 root disk 254,  2 Jun  7 10:13 chiark--early--a-stripe--base
brw-rw----  1 root disk 254,  1 Jun  7 10:13 chiark--early--a-swap
brw-rw----  1 root disk 254,  3 Jun  7 10:13 chiark--early--b-mirror--base
brw-rw----  1 root disk 254,  5 Jun  7 10:13 chiark--early--b-stripe--base
brw-rw----  1 root disk 254,  4 Jun  7 10:13 chiark--early--b-swap
crw-rw----  1 root root  10, 60 Jun  7 10:13 control
brw-rw----  1 root disk 254, 20 Jun  7 10:13 vg--chiark--stripe-lv--quicksand
brw-rw----  1 root disk 254, 19 Jun  7 10:13 vg--chiark--stripe-twomack--relations
brw-rw----  1 root disk 254,  6 Jun  7 10:13 vg--main-lv--log
brw-rw----  1 root disk 254,  7 Jun  7 10:13 vg--main-lv--mail
brw-rw----  1 root disk 254,  8 Jun  7 10:13 vg--main-lv--news
brw-rw----  1 root disk 254, 18 Jun  7 10:13 vg--main-lv--root
brw-rw----  1 root disk 254, 10 Jun  7 10:13 vg--main-lv--u
brw-rw----  1 root disk 254, 13 Jun  7 10:13 vg--main-lv--u2
brw-rw----  1 root disk 254, 14 Jun  7 10:13 vg--main-lv--u3
brw-rw----  1 root disk 254, 15 Jun  7 10:13 vg--main-lv--usr
brw-rw----  1 root disk 254, 16 Jun  7 10:13 vg--main-lv--var
brw-rw----  1 root disk 254, 17 Jun  7 10:13 vg--main-lv--volt

[dmesg (text/plain, inline)]
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.26-2-686-bigmem (Debian 2.6.26-21lenny4) (dannf@debian.org) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Tue Mar 9 18:01:52 UTC 2010
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009d000 (usable)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000bf699000 (usable)
[    0.000000]  BIOS-e820: 00000000bf699000 - 00000000bf6af000 (reserved)
[    0.000000]  BIOS-e820: 00000000bf6af000 - 00000000bf6ce000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000bf6ce000 - 00000000c0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
[    0.000000] 4224MB HIGHMEM available.
[    0.000000] 896MB LOWMEM available.
[    0.000000] found SMP MP-table at [c00fe710] 000fe710
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] Entering add_active_range(0, 0, 1310720) 0 entries of 256 used
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA             0 ->     4096
[    0.000000]   Normal       4096 ->   229376
[    0.000000]   HighMem    229376 ->  1310720
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[1] active PFN ranges
[    0.000000]     0:        0 ->  1310720
[    0.000000] On node 0 totalpages: 1310720
[    0.000000]   DMA zone: 32 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 4064 pages, LIFO batch:0
[    0.000000]   Normal zone: 1760 pages used for memmap
[    0.000000]   Normal zone: 223520 pages, LIFO batch:31
[    0.000000]   HighMem zone: 8448 pages used for memmap
[    0.000000]   HighMem zone: 1072896 pages, LIFO batch:31
[    0.000000]   Movable zone: 0 pages used for memmap
[    0.000000] DMI 2.6 present.
[    0.000000] Using APIC driver default
[    0.000000] ACPI: RSDP 000F0B40, 0024 (r2 DELL  )
[    0.000000] ACPI: XSDT 000F0C40, 0094 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: FACP BF6C3BB4, 00F4 (r3 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: DSDT BF6AF000, 398C (r1 DELL   PE_SC3          1 INTL 20050624)
[    0.000000] ACPI: FACS BF6C6000, 0040
[    0.000000] ACPI: APIC BF6C3478, 0152 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: SPCR BF6C35CC, 0050 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: HPET BF6C3620, 0038 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: DM__ BF6C365C, 00A8 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: MCFG BF6C3850, 003C (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: WD__ BF6C3890, 0134 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: SLIC BF6C39C8, 0024 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: ERST BF6B2B0C, 0270 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: HEST BF6B2D7C, 03A8 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: BERT BF6B298C, 0030 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: EINJ BF6B29BC, 0150 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: TCPA BF6C3B4C, 0064 (r2 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: SSDT BF6C7000, 2164 (r1  INTEL PPM RCM  80000001 INTL 20061109)
[    0.000000] ACPI: PM-Timer IO Port: 0x808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x28] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x29] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x2b] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2c] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2d] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x2e] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x2f] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x30] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x31] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x32] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x33] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x34] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x35] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x36] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x37] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0x38] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x39] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x3a] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x3b] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x3c] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x3d] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x3e] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0x3f] disabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
[    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] Allocating PCI resources starting at c2000000 (gap: c0000000:20000000)
[    0.000000] PM: Registered nosave memory: 000000000009d000 - 0000000000100000
[    0.000000] SMP: Allowing 32 CPUs, 24 hotplug CPUs
[    0.000000] PERCPU: Allocating 42184 bytes of per cpu data
[    0.000000] NR_CPUS: 32, nr_cpu_ids: 32
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 1300480
[    0.000000] Kernel command line: ro root=/dev/vg-main/lv-root console=tty0 console=ttyS0,9600n8 TERM=dumb reboot=bios,smp,warm 
[    0.000000] mapped APIC to ffffb000 (fee00000)
[    0.000000] mapped IOAPIC to ffffa000 (fec00000)
[    0.000000] Enabling fast FPU save and restore... done.
[    0.000000] Enabling unmasked SIMD FPU exception support... done.
[    0.000000] Initializing CPU#0
[    0.000000] PID hash table entries: 4096 (order: 12, 16384 bytes)
[    0.000000] Extended CMOS year: 2000
[    0.000000] Detected 1862.111 MHz processor.
[    0.004000] Console: colour VGA+ 80x25
[    0.004000] console [tty0] enabled
[    0.004000] console [ttyS0] enabled
[    0.004000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.004000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.004000] Memory: 4130168k/5242880k available (1836k kernel code, 53248k reserved, 790k data, 256k init, 3267172k highmem)
[    0.004000] virtual kernel memory layout:
[    0.004000]     fixmap  : 0xffe13000 - 0xfffff000   (1968 kB)
[    0.004000]     pkmap   : 0xffa00000 - 0xffc00000   (2048 kB)
[    0.004000]     vmalloc : 0xf8800000 - 0xff9fe000   ( 113 MB)
[    0.004000]     lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
[    0.004000]       .init : 0xc0399000 - 0xc03d9000   ( 256 kB)
[    0.004000]       .data : 0xc02cb06d - 0xc0390ac0   ( 790 kB)
[    0.004000]       .text : 0xc0100000 - 0xc02cb06d   (1836 kB)
[    0.004000] Checking if this processor honours the WP bit even in supervisor mode...Ok.
[    0.004000] CPA: page pool initialized 1 of 1 pages preallocated
[    0.004000] hpet clockevent registered
[    0.083864] Calibrating delay using timer specific routine.. 3727.45 BogoMIPS (lpj=7454918)
[    0.092005] Security Framework initialized
[    0.096006] SELinux:  Disabled at boot.
[    0.100006] Capability LSM initialized
[    0.104006] Mount-cache hash table entries: 512
[    0.108006] Initializing cgroup subsys ns
[    0.112007] Initializing cgroup subsys cpuacct
[    0.116007] Initializing cgroup subsys devices
[    0.120007] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.128008] CPU: L2 cache: 256K
[    0.132008] CPU: L3 cache: 8192K
[    0.136008] CPU: Physical Processor ID: 0
[    0.140008] CPU: Processor Core ID: 0
[    0.144009] Intel machine check architecture supported.
[    0.148009] Intel machine check reporting enabled on CPU#0.
[    0.152009] using mwait in idle threads.
[    0.156009] Checking 'hlt' instruction... OK.
[    0.180011] ACPI: Core revision 20080321
[    0.188011] ENABLING IO-APIC IRQs
[    0.192012] ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.235662] CPU0: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.248015] Booting processor 1/2 ip 6000
[    0.260016] Initializing CPU#1
[    0.260016] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447620)
[    0.260016] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.260016] CPU: L2 cache: 256K
[    0.260016] CPU: L3 cache: 8192K
[    0.260016] CPU: Physical Processor ID: 0
[    0.260016] CPU: Processor Core ID: 1
[    0.260016] Intel machine check architecture supported.
[    0.260016] Intel machine check reporting enabled on CPU#1.
[    0.340021] CPU1: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.352022] checking TSC synchronization [CPU#0 -> CPU#1]: passed.
[    0.356022] Booting processor 2/4 ip 6000
[    0.372023] Initializing CPU#2
[    0.372023] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447631)
[    0.372023] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.372023] CPU: L2 cache: 256K
[    0.372023] CPU: L3 cache: 8192K
[    0.372023] CPU: Physical Processor ID: 0
[    0.372023] CPU: Processor Core ID: 2
[    0.372023] Intel machine check architecture supported.
[    0.372023] Intel machine check reporting enabled on CPU#2.
[    0.448857] CPU2: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.464029] checking TSC synchronization [CPU#0 -> CPU#2]: passed.
[    0.468029] Booting processor 3/6 ip 6000
[    0.484030] Initializing CPU#3
[    0.484030] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447636)
[    0.484030] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.484030] CPU: L2 cache: 256K
[    0.484030] CPU: L3 cache: 8192K
[    0.484030] CPU: Physical Processor ID: 0
[    0.484030] CPU: Processor Core ID: 3
[    0.484030] Intel machine check architecture supported.
[    0.484030] Intel machine check reporting enabled on CPU#3.
[    0.560779] CPU3: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.576031] checking TSC synchronization [CPU#0 -> CPU#3]: passed.
[    0.580031] Booting processor 4/1 ip 6000
[    0.596037] Initializing CPU#4
[    0.596037] Calibrating delay using timer specific routine.. 3723.80 BogoMIPS (lpj=7447615)
[    0.596037] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.596037] CPU: L2 cache: 256K
[    0.596037] CPU: L3 cache: 8192K
[    0.596037] CPU: Physical Processor ID: 0
[    0.596037] CPU: Processor Core ID: 0
[    0.596037] Intel machine check architecture supported.
[    0.596037] Intel machine check reporting enabled on CPU#4.
[    0.672947] CPU4: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.688031] checking TSC synchronization [CPU#0 -> CPU#4]: passed.
[    0.692031] Booting processor 5/3 ip 6000
[    0.708044] Initializing CPU#5
[    0.708044] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447627)
[    0.708044] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.708044] CPU: L2 cache: 256K
[    0.708044] CPU: L3 cache: 8192K
[    0.708044] CPU: Physical Processor ID: 0
[    0.708044] CPU: Processor Core ID: 1
[    0.708044] Intel machine check architecture supported.
[    0.708044] Intel machine check reporting enabled on CPU#5.
[    0.784980] CPU5: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.800038] checking TSC synchronization [CPU#0 -> CPU#5]: passed.
[    0.804038] Booting processor 6/5 ip 6000
[    0.820051] Initializing CPU#6
[    0.820051] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447636)
[    0.820051] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.820051] CPU: L2 cache: 256K
[    0.820051] CPU: L3 cache: 8192K
[    0.820051] CPU: Physical Processor ID: 0
[    0.820051] CPU: Processor Core ID: 2
[    0.820051] Intel machine check architecture supported.
[    0.820051] Intel machine check reporting enabled on CPU#6.
[    0.896887] CPU6: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.912044] checking TSC synchronization [CPU#0 -> CPU#6]: passed.
[    0.916044] Booting processor 7/7 ip 6000
[    0.932058] Initializing CPU#7
[    0.932058] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447631)
[    0.932058] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.932058] CPU: L2 cache: 256K
[    0.932058] CPU: L3 cache: 8192K
[    0.932058] CPU: Physical Processor ID: 0
[    0.932058] CPU: Processor Core ID: 3
[    0.932058] Intel machine check architecture supported.
[    0.932058] Intel machine check reporting enabled on CPU#7.
[    1.008795] CPU7: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    1.024052] checking TSC synchronization [CPU#0 -> CPU#7]: passed.
[    1.028052] Brought up 8 CPUs
[    1.032052] Total of 8 processors activated (29794.15 BogoMIPS).
[    1.036052] CPU0 attaching sched-domain:
[    1.036052]  domain 0: span 0,4
[    1.036052]   groups: 0 4
[    1.036052]   domain 1: span 0-7
[    1.036052]    groups: 0,4 1,5 2,6 3,7
[    1.036052] CPU1 attaching sched-domain:
[    1.036052]  domain 0: span 1,5
[    1.036052]   groups: 1 5
[    1.036052]   domain 1: span 0-7
[    1.036052]    groups: 1,5 2,6 3,7 0,4
[    1.036052] CPU2 attaching sched-domain:
[    1.036052]  domain 0: span 2,6
[    1.036052]   groups: 2 6
[    1.036052]   domain 1: span 0-7
[    1.036052]    groups: 2,6 3,7 0,4 1,5
[    1.036052] CPU3 attaching sched-domain:
[    1.036052]  domain 0: span 3,7
[    1.036052]   groups: 3 7
[    1.036052]   domain 1: span 0-7
[    1.036052]    groups: 3,7 0,4 1,5 2,6
[    1.036052] CPU4 attaching sched-domain:
[    1.036052]  domain 0: span 0,4
[    1.036052]   groups: 4 0
[    1.036052]   domain 1: span 0-7
[    1.036052]    groups: 0,4 1,5 2,6 3,7
[    1.036052] CPU5 attaching sched-domain:
[    1.036052]  domain 0: span 1,5
[    1.036052]   groups: 5 1
[    1.036052]   domain 1: span 0-7
[    1.036052]    groups: 1,5 2,6 3,7 0,4
[    1.036052] CPU6 attaching sched-domain:
[    1.036052]  domain 0: span 2,6
[    1.036052]   groups: 6 2
[    1.036052]   domain 1: span 0-7
[    1.036052]    groups: 2,6 3,7 0,4 1,5
[    1.036052] CPU7 attaching sched-domain:
[    1.036052]  domain 0: span 3,7
[    1.036052]   groups: 7 3
[    1.036052]   domain 1: span 0-7
[    1.036052]    groups: 3,7 0,4 1,5 2,6
[    1.036052] net_namespace: 660 bytes
[    1.040060] Booting paravirtualized kernel on bare hardware
[    1.044060] NET: Registered protocol family 16
[    1.048060] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    1.055393] ACPI: bus type pci registered
[    1.059393] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
[    1.063393] PCI: MCFG area at e0000000 reserved in E820
[    1.067393] PCI: Using MMCONFIG for extended config space
[    1.071393] PCI: Using configuration type 1 for base access
[    1.075393] Setting up standard PCI resources
[    1.079393] ACPI: EC: Look up EC in DSDT
[    1.084794] ACPI: BIOS _OSI(Linux) query ignored
[    1.091030] ACPI: DMI System Vendor: Dell Inc.
[    1.095030] ACPI: DMI Product Name: PowerEdge R210
[    1.099030] ACPI: DMI Product Version: 
[    1.103030] ACPI: DMI Board Name: 0M877N
[    1.107030] ACPI: DMI BIOS Vendor: Dell Inc.
[    1.111032] ACPI: DMI BIOS Date: 10/30/2009
[    1.115032] ACPI: Please send DMI info above to linux-acpi@vger.kernel.org
[    1.119032] ACPI: If "acpi_osi=Linux" works better, please notify linux-acpi@vger.kernel.org
[    1.123033] ACPI: Interpreter enabled
[    1.127033] ACPI: (supports S0 S4 S5)
[    1.139034] ACPI: Using IOAPIC for interrupt routing
[    1.150844] ACPI: PCI Root Bridge [PCI0] (0000:00)
[    1.158836] PCI: Transparent bridge - 0000:00:1e.0
[    1.166131] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    1.166131] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.LYD0._PRT]
[    1.166131] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX0._PRT]
[    1.166131] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.COMP._PRT]
[    1.170131] ACPI: PCI Interrupt Link [LK00] (IRQs 3 4 5 6 7 10 11 14 *15)
[    1.200049] ACPI: PCI Interrupt Link [LK01] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
[    1.239241] ACPI: PCI Interrupt Link [LK02] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
[    1.281507] ACPI: PCI Interrupt Link [LK03] (IRQs 3 4 5 6 7 *10 11 14 15)
[    1.313679] ACPI: PCI Interrupt Link [LK04] (IRQs 3 4 5 6 7 10 *11 14 15)
[    1.345937] ACPI: PCI Interrupt Link [LK05] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
[    1.384679] ACPI: PCI Interrupt Link [LK06] (IRQs 3 4 5 6 7 10 11 *14 15)
[    1.420582] ACPI: PCI Interrupt Link [LK07] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
[    1.463236] Linux Plug and Play Support v0.97 (c) Adam Belay
[    1.467236] pnp: PnP ACPI init
[    1.471236] ACPI: bus type pnp registered
[    1.481343] pnp: PnP ACPI: found 13 devices
[    1.485344] ACPI: ACPI bus type pnp unregistered
[    1.489344] PnPBIOS: Disabled by ACPI PNP
[    1.493344] PCI: Using ACPI for IRQ routing
[    1.505352] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    1.581606] hpet0: 8 64-bit timers, 14318180 Hz
[    1.636850] ACPI: RTC can wake from S4
[    1.637508] Switched to high resolution mode on CPU 0
[    1.639335] Switched to high resolution mode on CPU 7
[    1.641058] Switched to high resolution mode on CPU 3
[    1.641096] Switched to high resolution mode on CPU 1
[    1.641096] Switched to high resolution mode on CPU 2
[    1.644899] Switched to high resolution mode on CPU 5
[    1.646400] Switched to high resolution mode on CPU 6
[    1.686592] Switched to high resolution mode on CPU 4
[    1.690619] system 00:07: ioport range 0x800-0x87f has been reserved
[    1.766671] system 00:07: ioport range 0x880-0x8ff has been reserved
[    1.842794] system 00:07: ioport range 0x900-0x91f has been reserved
[    1.918917] system 00:07: ioport range 0x920-0x923 has been reserved
[    1.995040] system 00:07: ioport range 0x924-0x924 has been reserved
[    2.071165] system 00:07: ioport range 0xca0-0xca7 has been reserved
[    2.147289] system 00:07: ioport range 0xca9-0xcab has been reserved
[    2.223411] system 00:07: ioport range 0xcad-0xcaf has been reserved
[    2.299538] system 00:08: ioport range 0xca8-0xca8 has been reserved
[    2.375656] system 00:08: ioport range 0xcac-0xcac has been reserved
[    2.451787] system 00:0a: iomem range 0xe0000000-0xefffffff could not be reserved
[    2.541442] system 00:0c: iomem range 0xfed50000-0xfed53fff could not be reserved
[    2.749565] PCI: Bridge: 0000:00:03.0
[    2.793561]   IO window: f000-ffff
[    2.837572]   MEM window: 0xdf900000-0xdf9fffff
[    2.892592]   PREFETCH window: disabled.
[    2.945573] PCI: Bridge: 0000:00:1c.0
[    2.989382]   IO window: disabled.
[    3.030080]   MEM window: disabled.
[    3.071815]   PREFETCH window: disabled.
[    3.118746] PCI: Bridge: 0000:00:1e.0
[    3.162554]   IO window: disabled.
[    3.203254]   MEM window: 0xde800000-0xdf7fffff
[    3.258049]   PREFETCH window: 0x00000000de000000-0x00000000de7fffff
[    3.334064] ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 16 (level, low) -> IRQ 16
[    3.437572] PCI: Setting latency timer of device 0000:00:03.0 to 64
[    3.437590] PCI: Setting latency timer of device 0000:00:1c.0 to 64
[    3.437600] PCI: Setting latency timer of device 0000:00:1e.0 to 64
[    3.437653] NET: Registered protocol family 2
[    3.529659] IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
[    3.615173] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[    3.704445] TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
[    3.783971] TCP: Hash tables configured (established 131072 bind 65536)
[    3.863107] TCP reno registered
[    3.918800] NET: Registered protocol family 1
[    3.970943] checking if image is initramfs... it is
[    4.596694] Freeing initrd memory: 6602k freed
[    5.358462] audit: initializing netlink socket (disabled)
[    5.658791] type=2000 audit(1275901948.528:1): initialized
[    5.728510] highmem bounce pool size: 64 pages
[    5.780517] Total HugeTLB memory allocated, 0
[    5.834268] VFS: Disk quotas dquot_6.5.1
[    5.905506] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    5.999711] msgmni has been set to 1700
[    6.045670] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    6.237671] io scheduler noop registered
[    6.285670] io scheduler anticipatory registered
[    6.341670] io scheduler deadline registered
[    7.306212] io scheduler cfq registered (default)
[    7.377656] pci 0000:03:03.0: Boot video device
[    7.378704] PCI: Setting latency timer of device 0000:00:03.0 to 64
[    7.378704] assign_interrupt_mode Found MSI capability
[    7.438735] Allocate Port Service[0000:00:03.0:pcie00]
[    7.440386] Allocate Port Service[0000:00:03.0:pcie01]
[    7.440386] PCI: Setting latency timer of device 0000:00:1c.0 to 64
[    7.440386] assign_interrupt_mode Found MSI capability
[    7.926210] Allocate Port Service[0000:00:1c.0:pcie00]
[    7.930302] Load service driver aer on pcie device 0000:00:03.0:pcie01
[    7.930302] isapnp: Scanning for PnP cards...
[    9.103921] isapnp: No Plug & Play device found
[    9.913339] hpet_resources: 0xfed00000 is busy
[    9.913339] Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
[   11.529239] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[   11.601288] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[   11.702619] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[   11.776850] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[   11.843718] brd: module loaded
[   11.881567] PNP: No PS/2 controller found. Probing ports directly.
[   12.138037] i8042.c: No controller found.
[   12.186909] mice: PS/2 mouse device common for all mice
[   12.251095] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
[   12.324317] rtc0: alarms up to one day, y3k
[   12.377396] cpuidle: using governor ladder
[   12.425401] cpuidle: using governor menu
[   12.473063] No iBFT detected.
[   12.509084] TCP cubic registered
[   12.546220] NET: Registered protocol family 17
[   12.600954] Using IPI No-Shortcut mode
[   12.647114] registered taskstats version 1
[   12.695376] rtc_cmos 00:04: setting system clock to 2010-06-07 09:12:48 UTC (1275901968)
[   12.788310] Freeing unused kernel memory: 256k freed
[   14.538145] Monitor-Mwait will be used to enter C-1 state
[   14.538145] Monitor-Mwait will be used to enter C-2 state
[   14.538145] Monitor-Mwait will be used to enter C-3 state
[   14.538145] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
[   15.286040] ACPI: ACPI0007:00 is registered as cooling_device0
[   15.356073] ------------[ cut here ]------------
[   15.356073] ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])
[   15.356073] ACPI: ACPI0007:01 is registered as cooling_device1
[   15.356073] WARNING: at arch/x86/kernel/hpet.c:299 hpet_legacy_next_event+0x38/0x5a()
[   15.881774] Modules linked in: processor(+) fan thermal_sys
[   15.881774] Pid: 0, comm: swapper Not tainted 2.6.26-2-686-bigmem #1
[   15.881774]  [<c0129147>] warn_on_slowpath+0x40/0x66
[   15.881774]  [<c01e7782>] __next_cpu+0x12/0x21
[   16.081772]  [<c01201f9>] find_busiest_group+0x21e/0x623
[   16.081772]  [<c01192bb>] hpet_legacy_next_event+0x38/0x5a
[   16.081772]  [<c013f68c>] clockevents_program_event+0xc4/0xd2
[   16.081772]  [<c014038e>] tick_dev_program_event+0x1e/0x82
[   16.081772]  [<c013fe14>] tick_broadcast_set_event+0xf/0x11
[   16.081772]  [<c014009e>] tick_broadcast_oneshot_control+0x8a/0xda
[   16.081772]  [<c013fae8>] tick_notify+0x1cd/0x2e3
[   16.081772]  [<c013acaa>] enqueue_hrtimer+0xc9/0xd4
[   16.081772]  [<c013bdd0>] notifier_call_chain+0x2a/0x47
[   16.081772]  [<c013be14>] raw_notifier_call_chain+0x9/0xc
[   16.081772]  [<c013f58c>] clockevents_notify+0x19/0x55
[   16.081772]  [<f8848f1b>] acpi_idle_enter_bm+0x12f/0x315 [processor]
[   16.081772]  [<c025b435>] cpuidle_idle_call+0x5b/0x86
[   16.081772]  [<c025b3da>] cpuidle_idle_call+0x0/0x86
[   16.081772]  [<c01075d3>] cpu_idle+0xb0/0xd0
[   16.081772]  =======================
[   18.026131] ---[ end trace 180ccccd85916810 ]---
[   18.026131] Marking TSC unstable due to: TSC halts in idle.
[   18.026131] ACPI: CPU2 (power states: C1[C1] C2[C2] C3[C3])
[   18.026131] ACPI: ACPI0007:02 is registered as cooling_device2
[   18.033904] ACPI: CPU3 (power states: C1[C1] C2[C2] C3[C3])
[   18.033904] ACPI: ACPI0007:03 is registered as cooling_device3
[   18.033904] ACPI: CPU4 (power states: C1[C1] C2[C2] C3[C3])
[   18.033904] ACPI: ACPI0007:04 is registered as cooling_device4
[   18.033904] ACPI: CPU5 (power states: C1[C1] C2[C2] C3[C3])
[   18.033904] ACPI: ACPI0007:05 is registered as cooling_device5
[   18.033904] ACPI: CPU6 (power states: C1[C1] C2[C2] C3[C3])
[   18.033904] ACPI: ACPI0007:06 is registered as cooling_device6
[   18.035674] ACPI: CPU7 (power states: C1[C1] C2[C2] C3[C3])
[   18.035674] ACPI: ACPI0007:07 is registered as cooling_device7
[   19.910306] e1000e: Intel(R) PRO/1000 Network Driver - 0.3.3.3-k2
[   19.984585] e1000e: Copyright (c) 1999-2008 Intel Corporation.
[   20.056632] ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 16
[   20.145363] PCI: Setting latency timer of device 0000:01:00.0 to 64
[   20.148015] usbcore: registered new interface driver usbfs
[   20.218805] usbcore: registered new interface driver hub
[   20.282794] usbcore: registered new device driver usb
[   20.348926] No dock devices found.
[   20.434205] SCSI subsystem initialized
[   20.491095] libata version 3.00 loaded.
[   20.515513] eth0: (PCI Express:2.5GB/s:Width x1) 00:1b:21:51:a5:06
[   20.634158] eth0: Intel(R) PRO/1000 Network Connection
[   20.717939] eth0: MAC: 3, PHY: 8, PBA No: e46981-003
[   20.772268] ACPI: PCI Interrupt 0000:00:1a.0[A] -> GSI 22 (level, low) -> IRQ 22
[   20.876293] PCI: Setting latency timer of device 0000:00:1a.0 to 64
[   20.876297] ehci_hcd 0000:00:1a.0: EHCI Host Controller
[   20.940363] ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
[   21.096163] ehci_hcd 0000:00:1a.0: debug port 2
[   21.182164] PCI: cache line size of 32 is not supported by device 0000:00:1a.0
[   21.182174] ehci_hcd 0000:00:1a.0: irq 22, io mem 0xdf8fa000
[   21.297066] ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[   21.385254] usb usb1: configuration #1 chosen from 1 choice
[   21.475578] hub 1-0:1.0: USB hub found
[   21.520485] hub 1-0:1.0: 2 ports detected
[   21.674646] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[   21.996465] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   22.084470] usb usb1: Product: EHCI Host Controller
[   22.142842] usb usb1: Manufacturer: Linux 2.6.26-2-686-bigmem ehci_hcd
[   22.154728] usb 1-1: new high speed USB device using ehci_hcd and address 2
[   22.287166] usb 1-1: configuration #1 chosen from 1 choice
[   22.287166] hub 1-1:1.0: USB hub found
[   22.287229] hub 1-1:1.0: 6 ports detected
[   22.390367] usb 1-1: New USB device found, idVendor=8087, idProduct=0020
[   22.390370] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[   22.630390] usb usb1: SerialNumber: 0000:00:1a.0
[   22.682406] ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 22 (level, low) -> IRQ 22
[   23.127469] PCI: Setting latency timer of device 0000:00:1d.0 to 64
[   23.127472] ehci_hcd 0000:00:1d.0: EHCI Host Controller
[   23.187971] ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
[   23.287989] ehci_hcd 0000:00:1d.0: debug port 2
[   23.340171] PCI: cache line size of 32 is not supported by device 0000:00:1d.0
[   23.340176] ehci_hcd 0000:00:1d.0: irq 22, io mem 0xdf8fc000
[   23.424255] ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[   23.513222] usb usb2: configuration #1 chosen from 1 choice
[   23.583141] hub 2-0:1.0: USB hub found
[   23.628706] hub 2-0:1.0: 2 ports detected
[   24.272184] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[   24.354065] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   24.442070] usb usb2: Product: EHCI Host Controller
[   24.500409] usb usb2: Manufacturer: Linux 2.6.26-2-686-bigmem ehci_hcd
[   24.514102] usb 2-1: new high speed USB device using ehci_hcd and address 2
[   24.646523] usb 2-1: configuration #1 chosen from 1 choice
[   24.646523] hub 2-1:1.0: USB hub found
[   24.646589] hub 2-1:1.0: 6 ports detected
[   24.750137] usb 2-1: New USB device found, idVendor=8087, idProduct=0020
[   24.750140] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[   24.950222] usb 2-1.5: new high speed USB device using ehci_hcd and address 3
[   25.042490] usb 2-1.5: configuration #1 chosen from 1 choice
[   25.042490] hub 2-1.5:1.0: USB hub found
[   25.042559] hub 2-1.5:1.0: 4 ports detected
[   25.446601] usb 2-1.5: New USB device found, idVendor=0424, idProduct=2514
[   25.446603] usb 2-1.5: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[   25.646809] usb 2-1.5.3: new high speed USB device using ehci_hcd and address 4
[   25.742051] usb 2-1.5.3: configuration #1 chosen from 1 choice
[   25.745999] usb 2-1.5.3: New USB device found, idVendor=0951, idProduct=1624
[   25.745999] usb 2-1.5.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   25.745999] usb 2-1.5.3: Product: DataTraveler G2
[   25.745999] usb 2-1.5.3: Manufacturer: Kingston
[   25.745999] usb 2-1.5.3: SerialNumber: 0014780F9955A9A085EC00BE
[   26.238512] usb usb2: SerialNumber: 0000:00:1d.0
[   26.295652] ahci 0000:00:1f.2: version 3.0
[   26.295652] ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 20 (level, low) -> IRQ 20
[   26.685567] Initializing USB Mass Storage driver...
[   26.759632] scsi0 : SCSI emulation for USB Mass Storage devices
[   26.830422] usbcore: registered new interface driver usb-storage
[   26.830422] usb-storage: device found at 4
[   26.830424] usb-storage: waiting for device to settle before scanning
[   26.910949] USB Mass Storage support registered.
[   27.711754] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
[   27.882626] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pmp pio slum part 
[   28.059553] PCI: Setting latency timer of device 0000:00:1f.2 to 64
[   28.061471] scsi1 : ahci
[   28.094211] scsi2 : ahci
[   28.125334] scsi3 : ahci
[   28.157198] scsi4 : ahci
[   28.188218] scsi5 : ahci
[   28.236327] scsi6 : ahci
[   28.267975] ata1: SATA max UDMA/133 abar m2048@0xdf8fe000 port 0xdf8fe100 irq 218
[   28.361466] ata2: SATA max UDMA/133 abar m2048@0xdf8fe000 port 0xdf8fe180 irq 218
[   28.451112] ata3: SATA max UDMA/133 abar m2048@0xdf8fe000 port 0xdf8fe200 irq 218
[   28.543119] ata4: SATA max UDMA/133 abar m2048@0xdf8fe000 port 0xdf8fe280 irq 218
[   28.632613] ata5: SATA max UDMA/133 abar m2048@0xdf8fe000 port 0xdf8fe300 irq 218
[   28.722415] ata6: SATA max UDMA/133 abar m2048@0xdf8fe000 port 0xdf8fe380 irq 218
[   29.399603] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[   29.484570] ata1.00: ATA-8: WDC WD7500AYPS-01ZKB1, 02.01B02, max UDMA/133
[   29.565992] ata1.00: 1465149168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[   29.652465] ata1.00: configured for UDMA/133
[   30.201114] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[   30.313461] ata2.00: ATA-8: WDC WD7500AYPS-01ZKB1, 02.01B02, max UDMA/133
[   30.405207] ata2.00: 1465149168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[   30.495436] ata2.00: configured for UDMA/133
[   30.874691] ata3: SATA link down (SStatus 0 SControl 300)
[   31.274925] ata4: SATA link down (SStatus 0 SControl 300)
[   31.675161] ata5: SATA link down (SStatus 0 SControl 300)
[   32.285148] ata6: SATA link down (SStatus 0 SControl 300)
[   32.364921] scsi 1:0:0:0: Direct-Access     ATA      WDC WD7500AYPS-0 02.0 PQ: 0 ANSI: 5
[   32.390095] usb-storage: device scan complete
[   32.390095] scsi 0:0:0:0: Direct-Access     Kingston DataTraveler G2  1.00 PQ: 0 ANSI: 2
[   32.901856] Driver 'sd' needs updating - please use bus_type methods
[   32.901856] scsi 2:0:0:0: Direct-Access     ATA      WDC WD7500AYPS-0 02.0 PQ: 0 ANSI: 5
[   33.077904] sd 0:0:0:0: [sda] 7831552 512-byte hardware sectors (4010 MB)
[   33.161033] sd 0:0:0:0: [sda] Write Protect is off
[   33.261524] sd 0:0:0:0: [sda] Mode Sense: 23 00 00 00
[   33.261526] sd 0:0:0:0: [sda] Assuming drive cache: write through
[   33.337817] sd 0:0:0:0: [sda] 7831552 512-byte hardware sectors (4010 MB)
[   33.422891] sd 0:0:0:0: [sda] Write Protect is off
[   33.506633] sd 0:0:0:0: [sda] Mode Sense: 23 00 00 00
[   33.506635] sd 0:0:0:0: [sda] Assuming drive cache: write through
[   33.578786]  sda: sda1
[   33.608932] sd 0:0:0:0: [sda] Attached SCSI removable disk
[   33.677442] sd 1:0:0:0: [sdb] 1465149168 512-byte hardware sectors (750156 MB)
[   33.768455] sd 1:0:0:0: [sdb] Write Protect is off
[   33.825721] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[   33.828492] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   34.037780] sd 1:0:0:0: [sdb] 1465149168 512-byte hardware sectors (750156 MB)
[   34.121780] sd 1:0:0:0: [sdb] Write Protect is off
[   34.121780] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[   34.177781] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   34.285778]  sdb: sdb1 sdb2
[   34.331956] sd 1:0:0:0: [sdb] Attached SCSI disk
[   34.391358] sd 2:0:0:0: [sdc] 1465149168 512-byte hardware sectors (750156 MB)
[   34.486374] sd 2:0:0:0: [sdc] Write Protect is off
[   34.543212] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[   34.545753] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   34.654258] sd 2:0:0:0: [sdc] 1465149168 512-byte hardware sectors (750156 MB)
[   34.837987] sd 2:0:0:0: [sdc] Write Protect is off
[   34.894175] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[   34.895577] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   35.003896]  sdc: sdc1 sdc2
[   35.050156] sd 2:0:0:0: [sdc] Attached SCSI disk
[   35.307708] device-mapper: uevent: version 1.0.3
[   35.383401] device-mapper: ioctl: 4.13.0-ioctl (2007-10-18) initialised: dm-devel@redhat.com
[   35.530895] md: raid0 personality registered for level 0
[   35.609576] md: raid1 personality registered for level 1
[   35.873259] md: md127 stopped.
[   35.991805] md: bind<dm-3>
[   36.030565] md: bind<dm-0>
[   36.079520] md: md127: raid array is not clean -- starting background reconstruction
[   36.204275] raid1: raid set md127 active with 2 out of 2 mirrors
[   36.742716] EXT3-fs: INFO: recovery required on readonly filesystem.
[   36.834254] EXT3-fs: write access will be enabled during recovery.
[   37.375441] md: resync of RAID array md127
[   37.393581] kjournald starting.  Commit interval 5 seconds
[   37.393581] EXT3-fs: recovery complete.
[   37.393778] EXT3-fs: mounted filesystem with ordered data mode.
[   37.763446] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[   37.901838] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[   37.901843] md: using 128k window, over a total of 222297980 blocks.
[   39.318926] udevd version 125 started
[   39.526894] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
[   39.768006] input: PC Speaker as /class/input/input0
[   39.850320] input: Power Button (FF) as /class/input/input1
[   39.984597] ACPI: Power Button (FF) [PWRF]
[   41.212396] EXT3 FS on dm-18, internal journal
[   41.831114] md: md126 stopped.
[   42.046012] md: bind<dm-5>
[   42.089787] md: bind<dm-2>
[   42.215263] md126: setting max_sectors to 128, segment boundary to 32767
[   42.295225] raid0: looking at dm-2
[   42.335916] raid0:   comparing dm-2(54525888) with dm-2(54525888)
[   42.407836] raid0:   END
[   42.438153] raid0:   ==> UNIQUE
[   42.475739] raid0: 1 zones
[   42.528213] raid0: looking at dm-5
[   42.581001] raid0:   comparing dm-5(54525888) with dm-2(54525888)
[   42.678343] raid0:   EQUAL
[   42.711182] raid0: FINAL 1 zones
[   42.751186] raid0: done.
[   42.781448] raid0 : md_size is 109051776 blocks.
[   42.836906] raid0 : conf->hash_spacing is 109051776 blocks.
[   42.904911] raid0 : nb_zone is 1.
[   42.961486] raid0 : Allocating 4 bytes for hash.
[   73.273813] kjournald starting.  Commit interval 5 seconds
[   73.275108] EXT3 FS on dm-15, internal journal
[   73.275108] EXT3-fs: mounted filesystem with ordered data mode.
[   73.315332] kjournald starting.  Commit interval 5 seconds
[   73.315332] EXT3 FS on dm-16, internal journal
[   73.315332] EXT3-fs: mounted filesystem with ordered data mode.
[   73.379532] kjournald starting.  Commit interval 5 seconds
[   73.380685] EXT3 FS on dm-6, internal journal
[   73.380685] EXT3-fs: mounted filesystem with ordered data mode.
[   73.449358] kjournald starting.  Commit interval 5 seconds
[   73.449358] EXT3 FS on dm-7, internal journal
[   73.449358] EXT3-fs: mounted filesystem with ordered data mode.
[   73.479249] kjournald starting.  Commit interval 5 seconds
[   73.480447] EXT3 FS on dm-8, internal journal
[   73.480447] EXT3-fs: mounted filesystem with ordered data mode.
[   73.525190] kjournald starting.  Commit interval 5 seconds
[   73.525190] EXT3 FS on dm-17, internal journal
[   73.525190] EXT3-fs: mounted filesystem with ordered data mode.
[   73.566757] kjournald starting.  Commit interval 5 seconds
[   73.568562] EXT3 FS on dm-10, internal journal
[   73.568562] EXT3-fs: mounted filesystem with ordered data mode.
[   73.574051] kjournald starting.  Commit interval 5 seconds
[   73.574051] EXT3 FS on dm-13, internal journal
[   73.574051] EXT3-fs: mounted filesystem with ordered data mode.
[   73.611615] kjournald starting.  Commit interval 5 seconds
[   73.611684] EXT3 FS on dm-14, internal journal
[   73.611684] EXT3-fs: mounted filesystem with ordered data mode.
[   73.690197] kjournald starting.  Commit interval 5 seconds
[   73.690197] EXT3 FS on dm-19, internal journal
[   73.690197] EXT3-fs: mounted filesystem with ordered data mode.
[   73.738732] kjournald starting.  Commit interval 5 seconds
[   73.739828] EXT3 FS on dm-20, internal journal
[   73.739828] EXT3-fs: mounted filesystem with ordered data mode.
[   73.786859] Adding 4194296k swap on /dev/mapper/chiark--early--a-swap.  Priority:10 extents:1 across:4194296k
[   73.847214] Adding 4194296k swap on /dev/mapper/chiark--early--b-swap.  Priority:10 extents:1 across:4194296k
[   75.677263] ip_tables: (C) 2000-2006 Netfilter Core Team
[   75.744291] NET: Registered protocol family 10
[   75.744291] lo: Disabled Privacy Extensions
[   75.765697] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   78.125485] ADDRCONF(NETDEV_UP): eth0: link is not ready

Bug reassigned from package 'linux-image-2.6.26-2-686-bigmem' to 'linux-2.6'. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Mon, 07 Jun 2010 10:51:07 GMT) Full text and rfc822 format available.

Bug No longer marked as found in versions linux-2.6/2.6.26-21lenny4. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Mon, 07 Jun 2010 10:51:07 GMT) Full text and rfc822 format available.

Bug Marked as found in versions 2.6.26-21lenny4. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Mon, 07 Jun 2010 10:51:08 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Thu, 10 Jun 2010 15:00:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Andreas Schamanek <as2007+dbts@fam.tuwien.ac.at>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Thu, 10 Jun 2010 15:00:04 GMT) Full text and rfc822 format available.

Message #16 received at 584881@bugs.debian.org (full text, mbox):

From: Andreas Schamanek <as2007+dbts@fam.tuwien.ac.at>
To: 584881@bugs.debian.org
Subject: NeilBrown added a cond_resched
Date: Thu, 10 Jun 2010 16:42:36 +0200 (CEST)
Maybe someone can verify whether the patch contributed by Neil Brown
[ http://www.ktrap.org/mailarchive/git-commits-head/2009/10/31/11499 ]
fixes this? Should be available since 2.6.32.

HTH.

-- 
-- Andreas





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Thu, 10 Jun 2010 15:42:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Thu, 10 Jun 2010 15:42:03 GMT) Full text and rfc822 format available.

Message #21 received at 584881@bugs.debian.org (full text, mbox):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>
To: 584881@bugs.debian.org
Cc: Andreas Schamanek <as2007+dbts@fam.tuwien.ac.at>
Subject: NeilBrown added a cond_resched
Date: Thu, 10 Jun 2010 16:22:02 +0100
> Maybe someone can verify whether the patch contributed by Neil Brown
> [ http://www.ktrap.org/mailarchive/git-commits-head/2009/10/31/11499 ]
> fixes this? Should be available since 2.6.32.

Thanks for the suggestion, but it's not clear to me from the commit
message and the content that that's addressing a deadlock rather than
a performance problem.

When my system is in the deadlocked state, there seems to be no
problem finding CPUs to run runnable processes on - just a shortage of
runnable processes that can make progress, because everything is stuck
in disk wait.  So additional reschedules wouldn't seem likely to help.

chiark has a total of 8 cores and only two md devices, only one of
which is a mirror (the other is stripe) so I don't think a single
management thread could cause total cpu starvation anyway.

Ian.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Sun, 20 Jun 2010 23:12:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Sun, 20 Jun 2010 23:12:03 GMT) Full text and rfc822 format available.

Message #26 received at 584881@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: Ian Jackson <ijackson@chiark.greenend.org.uk>, 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Mon, 21 Jun 2010 00:08:08 +0100
[Message part 1 (text/plain, inline)]
On Mon, 2010-06-07 at 10:53 +0100, Ian Jackson wrote:
> Package: linux-image-2.6.26-2-686-bigmem
> Version: 2.6.26-21lenny4
> 
> I keep getting soft lockups; the symptoms appear to be that user
> processes become deadlocked when they try to access the disk, but the
> kernel doesn't notice that anything is wrong.
[...]

We really need to see the kernel messages reporting soft-lockup.

Note that your disk configuration is unusual and probably not
well-tested by others.  It is unlikely that anyone in the kernel team
will be able to debug this as I don't think any of us have particular
expertise in this area.  If you have the opportunity, it would be
helpful if you could test a newer kernel version.  This would give us a
clue as to whether the bug has subsequently been fixed upstream, and if
not then it would be the basis for an upstream bug report.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Mon, 21 Jun 2010 10:15:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Mon, 21 Jun 2010 10:15:03 GMT) Full text and rfc822 format available.

Message #31 received at 584881@bugs.debian.org (full text, mbox):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Mon, 21 Jun 2010 11:11:02 +0100
Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> We really need to see the kernel messages reporting soft-lockup.

There aren't any.  Or, if there are, it isn't printing them to the
serial console.  Perhaps it is trying to send them only to syslogd
which obviously can't write to the disk either.

I know that the serial console is working after the crash as it echoes
my first carriage return (although then login or the shell wedges
before being able to print a prompt).  I could have asked for a magic
sysrq process dump but given that the process table is probably full I
expect this would take many hours at 9600bps.

> Note that your disk configuration is unusual and probably not
> well-tested by others.  It is unlikely that anyone in the kernel team
> will be able to debug this as I don't think any of us have particular
> expertise in this area.

Search the web suggests that symptoms very similar to mine are not
uncommon, including instances without soft lockup messages, and none
of the other users seem to have a similar disk layout.

I can't easily test this theory but I think the unusual disk layout is
probably simply making a race easier to trigger.

>  If you have the opportunity, it would be
> helpful if you could test a newer kernel version.  This would give us a
> clue as to whether the bug has subsequently been fixed upstream, and if
> not then it would be the basis for an upstream bug report.

Unfortunately testing this bug on the live system involves (a) risking
a crash and then if the test fails (b) an extremely vulnerable system
without backups for the following four days.

I'll see if I can borrow a spare R210 from Jump, in which case I may
be able to reproduce the problem in controlled conditions on my coffee
table at home (and with access to the VGA console).  Which kernel
should I test in that case ?

Ian.




Added tag(s) moreinfo. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Thu, 24 Jun 2010 02:18:02 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Thu, 24 Jun 2010 03:30:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Thu, 24 Jun 2010 03:30:03 GMT) Full text and rfc822 format available.

Message #38 received at 584881@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: Ian Jackson <ijackson@chiark.greenend.org.uk>, 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Thu, 24 Jun 2010 03:15:35 +0100
[Message part 1 (text/plain, inline)]
On Mon, 2010-06-21 at 11:11 +0100, Ian Jackson wrote:
> Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> > We really need to see the kernel messages reporting soft-lockup.
> 
> There aren't any.  Or, if there are, it isn't printing them to the
> serial console.  Perhaps it is trying to send them only to syslogd
> which obviously can't write to the disk either.
> 
> I know that the serial console is working after the crash as it echoes
> my first carriage return (although then login or the shell wedges
> before being able to print a prompt).  I could have asked for a magic
> sysrq process dump but given that the process table is probably full I
> expect this would take many hours at 9600bps.

Even if you can't get a process dump, you can get some useful
information with:
'd' - show locks held
'l' - show backtrace for active CPUs
'w' - show uninterruptible tasks

> > Note that your disk configuration is unusual and probably not
> > well-tested by others.  It is unlikely that anyone in the kernel team
> > will be able to debug this as I don't think any of us have particular
> > expertise in this area.
> 
> Search the web suggests that symptoms very similar to mine are not
> uncommon, including instances without soft lockup messages, and none
> of the other users seem to have a similar disk layout.
> 
> I can't easily test this theory but I think the unusual disk layout is
> probably simply making a race easier to trigger.

Thinking of some kind of lock-dependency bug?  Blocking on a mutex for a
long period should still trigger a soft-lockup message.  Since there are
no messages from the kernel it's something of a mystery what's going on.

> >  If you have the opportunity, it would be
> > helpful if you could test a newer kernel version.  This would give us a
> > clue as to whether the bug has subsequently been fixed upstream, and if
> > not then it would be the basis for an upstream bug report.
> 
> Unfortunately testing this bug on the live system involves (a) risking
> a crash and then if the test fails (b) an extremely vulnerable system
> without backups for the following four days.

That's what I suspected.

> I'll see if I can borrow a spare R210 from Jump, in which case I may
> be able to reproduce the problem in controlled conditions on my coffee
> table at home (and with access to the VGA console).  Which kernel
> should I test in that case ?

Please try 2.6.34 from experimental.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Thu, 24 Jun 2010 10:21:09 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Thu, 24 Jun 2010 10:21:09 GMT) Full text and rfc822 format available.

Message #43 received at 584881@bugs.debian.org (full text, mbox):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Thu, 24 Jun 2010 11:17:28 +0100
Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> Even if you can't get a process dump, you can get some useful
> information with:

Right, thanks.

> 'd' - show locks held
> 'l' - show backtrace for active CPUs
> 'w' - show uninterruptible tasks

I'll try these although I suspect thousands of uninterruptible tasks.

> > Search the web suggests that symptoms very similar to mine are not
> > uncommon, including instances without soft lockup messages, and none
> > of the other users seem to have a similar disk layout.
> > 
> > I can't easily test this theory but I think the unusual disk layout is
> > probably simply making a race easier to trigger.
> 
> Thinking of some kind of lock-dependency bug?  Blocking on a mutex for a
> long period should still trigger a soft-lockup message.  Since there are
> no messages from the kernel it's something of a mystery what's going on.

The RAID system (md driver) has a separate mechanism for blocking
writes, which it calls a "barrier".  I think it is quite capable of
indefinitely blocking all writes to a device without necessarily
triggering the soft lockup detector.

> > I'll see if I can borrow a spare R210 from Jump, in which case I may
> > be able to reproduce the problem in controlled conditions on my coffee
> > table at home (and with access to the VGA console).  Which kernel
> > should I test in that case ?
> 
> Please try 2.6.34 from experimental.

Will do.  I'll get back to you.

Thanks,
Ian.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Fri, 25 Jun 2010 02:03:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Fri, 25 Jun 2010 02:03:06 GMT) Full text and rfc822 format available.

Message #48 received at 584881@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: Ian Jackson <ijackson@chiark.greenend.org.uk>
Cc: 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Fri, 25 Jun 2010 01:53:30 +0100
[Message part 1 (text/plain, inline)]
On Thu, 2010-06-24 at 11:17 +0100, Ian Jackson wrote:
> Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
[...]
> > > Search the web suggests that symptoms very similar to mine are not
> > > uncommon, including instances without soft lockup messages, and none
> > > of the other users seem to have a similar disk layout.
> > > 
> > > I can't easily test this theory but I think the unusual disk layout is
> > > probably simply making a race easier to trigger.
> > 
> > Thinking of some kind of lock-dependency bug?  Blocking on a mutex for a
> > long period should still trigger a soft-lockup message.  Since there are
> > no messages from the kernel it's something of a mystery what's going on.
> 
> The RAID system (md driver) has a separate mechanism for blocking
> writes, which it calls a "barrier".  I think it is quite capable of
> indefinitely blocking all writes to a device without necessarily
> triggering the soft lockup detector.
[...]

I/O barriers are block I/O operations (not specific to md) that inhibit
reordering of read and write operations.  They certainly should not be
blocking operations.  Also, device-mapper did not support barriers until
after 2.6.26 so md will not be using them in the configuration you are
using.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Fri, 25 Jun 2010 10:54:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Fri, 25 Jun 2010 10:54:03 GMT) Full text and rfc822 format available.

Message #53 received at 584881@bugs.debian.org (full text, mbox):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Fri, 25 Jun 2010 11:50:40 +0100
Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> I/O barriers are block I/O operations (not specific to md) that inhibit
> reordering of read and write operations.  They certainly should not be
> blocking operations.  Also, device-mapper did not support barriers until
> after 2.6.26 so md will not be using them in the configuration you are
> using.

No, I think there are two meanings of the word "barrier".  AFAICT md
has its own thing which it confusingly calls a "barrier"; it can be
"raised" and "lowered".

Ian.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Fri, 25 Jun 2010 11:15:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Fri, 25 Jun 2010 11:15:03 GMT) Full text and rfc822 format available.

Message #58 received at 584881@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: Ian Jackson <ijackson@chiark.greenend.org.uk>
Cc: 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Fri, 25 Jun 2010 12:13:48 +0100
[Message part 1 (text/plain, inline)]
On Fri, 2010-06-25 at 11:50 +0100, Ian Jackson wrote:
> Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> > I/O barriers are block I/O operations (not specific to md) that inhibit
> > reordering of read and write operations.  They certainly should not be
> > blocking operations.  Also, device-mapper did not support barriers until
> > after 2.6.26 so md will not be using them in the configuration you are
> > using.
> 
> No, I think there are two meanings of the word "barrier".  AFAICT md
> has its own thing which it confusingly calls a "barrier"; it can be
> "raised" and "lowered".

Oh, great!  I wondered whether this was the case but I could only find
discussion of md vs I/O barriers.  Do you have any reference for
documentation of md barriers?

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Fri, 25 Jun 2010 11:30:55 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Fri, 25 Jun 2010 11:30:55 GMT) Full text and rfc822 format available.

Message #63 received at 584881@bugs.debian.org (full text, mbox):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Fri, 25 Jun 2010 12:27:11 +0100
Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> On Fri, 2010-06-25 at 11:50 +0100, Ian Jackson wrote:
> > No, I think there are two meanings of the word "barrier".  AFAICT md
> > has its own thing which it confusingly calls a "barrier"; it can be
> > "raised" and "lowered".
> 
> Oh, great!  I wondered whether this was the case but I could only find
> discussion of md vs I/O barriers.  Do you have any reference for
> documentation of md barriers?

No.  I just stumbled across them in the source.  Particularly this in
drivers/md/raid1.c:

/* Barriers....
 * Sometimes we need to suspend IO while we do something else,
 * either some resync/recovery, or reconfigure the array.
 * To do this we raise a 'barrier'.
 * The 'barrier' is a counter that can be raised multiple times
 * to count how many activities are happening which preclude
 * normal IO.
 * We can only raise the barrier if there is no pending IO.
 * i.e. if nr_pending == 0.
 * We choose only to raise the barrier if no-one is waiting for the
 * barrier to go down.  This means that as soon as an IO request
 * is ready, no other operations which require a barrier will start
 * until the IO request has had a chance.
 *
 * So: regular IO calls 'wait_barrier'.  When that returns there
 *    is no backgroup IO happening,  It must arrange to call
 *    allow_barrier when it has finished its IO.
 * backgroup IO calls must call raise_barrier.  Once that returns
 *    there is no normal IO happeing.  It must arrange to call
 *    lower_barrier when the particular background IO completes.
 */

Ian.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Tue, 20 Jul 2010 21:54:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Tue, 20 Jul 2010 21:54:03 GMT) Full text and rfc822 format available.

Message #68 received at 584881@bugs.debian.org (full text, mbox):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>
To: 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Tue, 20 Jul 2010 22:51:01 +0100
[Message part 1 (text/plain, inline)]
Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> Please try 2.6.34 from experimental.

I've now replicated the problem on my coffee table with a temporary
(intermittent) loan of an identical machine from Jump networks.
I can get 2.6.26-21lenny4 and 2.6.26-24 to crash on demand very
easily.  In none of the crashes do I get any kind of kernel log
messages (eg, soft lockup warnings).

2.6.34 wasn't in experimental any more by the time I had all this set
up.  Instead I tried linux-image-2.6.35-rc5-686-bigmem version
2.6.35~rc5-1~experimental.1.  I wasn't able to reproduce the problem
with 2.6.35.

> Even if you can't get a process dump, you can get some useful
> information with:
> 'd' - show locks held
> 'l' - show backtrace for active CPUs
> 'w' - show uninterruptible tasks

2.6.24 doesn't seem to have the "d" debug key.  It says:

[  723.072717] SysRq : HELP : loglevel0-8 reBoot Crashdump tErm Full
kIll saK aLlcpus showMem Nice powerOff showPc show-all-timers(Q) unRaw
Sync showTasks Unmount shoW-blocked-tasks 

But I have the other information.  It's in the attached logfile.  On
my coffee table I was able to set the serial rate to something
sensible :-).

The first 200 lines or so are just me setting things up and checking
that debug keys are working and the output is being collected.  The
message
  [  612.446061] EXT3-fs: mounted filesystem with ordered data mode.
is due to the backup process mounting (readonly) its LVM snapshot.

I'll probably have to give the machine back for a bit now, so if you'd
like to think about what set of tests you'd like me to do next that
would be great.  Next time it will be quicker to set up as I can keep
the disks in between :-).

Thanks.
Ian.

[Xterm.log.liberator.2010.07.20.22.34.11.4990 (text/plain, inline)]
Thanks for using picocom
liberator:~> really picocom -b 9600 /dev/ttyUSB0 [1@1[1@1[1@5[1@2[1@2[1@0[1@0[1@0[1@0
picocom v1.4

port is        : /dev/ttyUSB0
flowcontrol    : none
baudrate is    : 115200
parity is      : none
databits are   : 8
escape is      : C-a
noinit is      : no
noreset is     : no
nolock is      : no
send_cmd is    : ascii_xfr -s -v -l10
receive_cmd is : rz -vv

Terminal ready



Thanks for using picocom
liberator:~> really picocom -b 115200 /dev/ttyUSB0 [1@9[1@6[1@0[1@0
picocom v1.4

port is        : /dev/ttyUSB0
flowcontrol    : none
baudrate is    : 9600
parity is      : none
databits are   : 8
escape is      : C-a
noinit is      : no
noreset is     : no
nolock is      : no
send_cmd is    : ascii_xfr -s -v -l10
receive_cmd is : rz -vv

Terminal ready

-rescue:~> 
-rescue:~> 
-rescue:~> really stty </de    115200
ΓΏ
Thanks for using picocom
liberator:~> really picocom -b 9600 /dev/ttyUSB0 [1@1[1@1[1@1[1@5[1@2[1@0[1@0
picocom v1.4

port is        : /dev/ttyUSB0
flowcontrol    : none
baudrate is    : 115200
parity is      : none
databits are   : 8
escape is      : C-a
noinit is      : no
noreset is     : no
nolock is      : no
send_cmd is    : ascii_xfr -s -v -l10
receive_cmd is : rz -vv

Terminal ready

-rescue:~> 
-rescue:~> 
-rescue:~> 
-rescue:~> d h  00:37:01 up 3 min,  3 users,  load average: 1.33, 0.78, 0.32


-rescue:~> 
-rescue:~> cat /proc/syske  /kernel/sysrq 
1
-rescue:~> 
-rescue:~> #
-rescue:~> 
-rescue:~> 
-rescue:~> 
-rescue:~> 
*** break sent ***
[  415.071786] SysRq : Show Memory

-rescue:~> 
-rescue:~> 
-rescue:~> 
-rescue:~> 
-rescue:~> cat /pcat /etc/sysc
sysconfig/             sysctl.conf.dpkg-dist  sysctl.d/
sysctl.conf            sysctl.conf~           
-rescue:~> cat /etc/sysc
sysconfig/             sysctl.conf.dpkg-dist  sysctl.d/
sysctl.conf            sysctl.conf~           
-rescue:~> cat /etc/sysctl.conf
#
# /etc/sysctl.conf - Configuration file for setting system variables
# See sysctl.conf (5) for information.
#
#kernel.domainname = example.com
#net/ipv4/icmp_echo_ignore_broadcasts=1
kernel/printk= 4 4 1 4
fs/file-max= 200000
#fs/inode-max= 32768
-rescue:~> sysctl kernel/printk= 4 4 1 4
error: Malformed setting "kernel/printk="
error: "4" is an unknown key
error: "4" is an unknown key
error: "1" is an unknown key
error: "4" is an unknown key
-rescue:~> sysctl kernel/printk= 4 4 1 4printk[1@ = [1@'4 4 1 4'
kernel.printk = 4	4	1	4
error: Malformed setting "="
error: "4 4 1 4" is an unknown key
-rescue:~> sysctl kernel/printk = '4 4 1 4' 
kernel.printk = 4	4	1	4
error: Malformed setting "="
error: "4" is an unknown key
error: "4" is an unknown key
error: "1" is an unknown key
error: "4" is an unknown key
-rescue:~> sysctl kernel/printk = 4 4 1 4kernel/printk
kernel.printk = 4	4	1	4
error: "4" is an unknown key
error: "4" is an unknown key
error: "1" is an unknown key
error: "4" is an unknown key
-rescue:~> man sysctl
[?1049h[?1h=SYSCTL(8)                                                            SYSCTL(8)



NAME
       sysctl - configure kernel parameters at runtime

SYNOPSIS
       sysctl [-n] [-e] variable ...
       sysctl [-n] [-e] [-q] -w variable=value ...
       sysctl [-n] [-e] [-q] -p [filename]
       sysctl [-n] [-e] -a
       sysctl [-n] [-e] -A

DESCRIPTION
       sysctl  is used to modify kernel parameters at runtime.  The parameters
       available are those listed under /proc/sys/.  Procfs  is  required  for
       sysctl  support  in  Linux.   You can use sysctl to both read and write
       sysctl data.

PARAMETERS
       variable
              The name of a key to read from.  An  example  is  kernel.ostype.
 Manual page sysctl(8) line 1
gzip: stdout: No space left on device

[?1l>[?1049l-rescue:~> man sysctlsysctl kernel/printk 4 4 1 4[1@=[1@'4 4 1 4'
error: permission denied on key 'kernel.printk'
-rescue:~> sysctl kernel/printk='4 4 1 4'eal   
-rescue:~> [1@r[1@e[1@a[1@l[1@l[1@y[1@ 
kernel.printk = 4 4 1 4
-rescue:~> really sysctl kernel/printk='4 4 1 4'[1@7
kernel.printk = 7 4 1 4
-rescue:~> really sysctl kernel/printk='7 4 1 4''[1@0
kernel.printk = 0 4 1 4
-rescue:~> 
*** break sent ***
[  551.797057] SysRq : Show Memory

-rescue:~> 
-rescue:~> really sysctl kernel/printk='0 4 1 4'7 4 1 4'
kernel.printk = 7 4 1 4
-rescue:~>  00:42:01 up 8 min,  3 users,  load average: 0.52, 0.81, 0.47


*** break sent ***
[  560.972713] SysRq : Show Memory
[  560.972713] Mem-info:
[  560.972713] DMA per-cpu:
[  560.972713] CPU    0: hi:    0, btch:   1 usd:   0
[  560.972713] CPU    1: hi:    0, btch:   1 usd:   0
[  560.972713] CPU    2: hi:    0, btch:   1 usd:   0
[  560.972713] CPU    3: hi:    0, btch:   1 usd:   0
[  560.972713] CPU    4: hi:    0, btch:   1 usd:   0
[  560.972713] CPU    5: hi:    0, btch:   1 usd:   0
[  560.972713] CPU    6: hi:    0, btch:   1 usd:   0
[  560.972713] CPU    7: hi:    0, btch:   1 usd:   0
[  560.972713] Normal per-cpu:
[  560.972713] CPU    0: hi:  186, btch:  31 usd:  83
[  560.972713] CPU    1: hi:  186, btch:  31 usd:  51
[  560.972713] CPU    2: hi:  186, btch:  31 usd:  66
[  560.972713] CPU    3: hi:  186, btch:  31 usd:  44
[  560.972713] CPU    4: hi:  186, btch:  31 usd: 131
[  560.972713] CPU    5: hi:  186, btch:  31 usd: 104
[  560.972713] CPU    6: hi:  186, btch:  31 usd: 160
[  560.972713] CPU    7: hi:  186, btch:  31 usd:  39
[  560.972713] HighMem per-cpu:
[  560.972713] CPU    0: hi:  186, btch:  31 usd: 166
[  560.972713] CPU    1: hi:  186, btch:  31 usd:  34
[  560.972713] CPU    2: hi:  186, btch:  31 usd: 147
[  560.972713] CPU    3: hi:  186, btch:  31 usd: 175
[  560.972713] CPU    4: hi:  186, btch:  31 usd:  48
[  560.972713] CPU    5: hi:  186, btch:  31 usd: 157
[  560.972713] CPU    6: hi:  186, btch:  31 usd:  86
[  560.972713] CPU    7: hi:  186, btch:  31 usd:  10
[  560.972713] Active:10403 inactive:29112 dirty:49 writeback:0 unstable:0
[  560.972713]  free:984661 slab:4007 mapped:1923 pagetables:184 bounce:0
[  560.972713] DMA free:12820kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaimable? no
[  560.972713] lowmem_reserve[]: 0 873 5064 5064
[  560.972713] Normal free:818384kB min:3744kB low:4680kB high:5616kB active:5128kB inactive:4820kB present:894080kB pages_scanned:0 all_unreclaimable? no
[  560.972713] lowmem_reserve[]: 0 0 33528 33528
[  560.972713] HighMem free:3107440kB min:512kB low:5008kB high:9504kB active:36484kB inactive:111628kB present:4291584kB pages_scanned:0 all_unreclaimable? no
[  560.972713] lowmem_reserve[]: 0 0 0 0
[  560.972713] DMA: 5*4kB 4*8kB 4*16kB 3*32kB 3*64kB 3*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12820kB
[  560.972713] Normal: 62*4kB 15*8kB 16*16kB 11*32kB 42*64kB 21*128kB 12*256kB 4*512kB 2*1024kB 5*2048kB 194*4096kB = 818384kB
[  560.972713] HighMem: 120*4kB 28*8kB 13*16kB 9*32kB 3*64kB 4*128kB 1*256kB 1*512kB 2*1024kB 1*2048kB 757*4096kB = 3107440kB
[  560.972713] 36625 total pagecache pages
[  560.972713] Swap cache: add 0, delete 0, find 0/0
[  560.972713] Free swap  = 8388592kB
[  560.972713] Total swap = 8388592kB
[  560.972713] 1310720 pages of RAM
[  560.972713] 1081344 pages of HIGHMEM
[  560.972713] 276248 reserved pages
[  560.972713] 11078 pages shared
[  560.972713] 0 pages swap cached
[  560.972713] 49 pages dirty
[  560.972713] 0 pages writeback
[  560.972713] 1923 pages mapped
[  560.972713] 4007 pages slab
[  560.972713] 184 pages pagetables

-rescue:~> 
-rescue:~> 
-rescue:~> [  612.446061] kjournald starting.  Commit interval 5 seconds
[  612.446061] EXT3-fs: mounted filesystem with ordered data mode.

-rescue:~> 
-rescue:~> 
-rescue:~> 
-rescue:~> echo hi
hi
-rescue:~> 
-rescue:~> 
-rescue:~> 
-rescue:~> ls





*** break sent ***
[  677.678494] SysRq : HELP : loglevel0-8 reBoot Crashdump tErm Full kIll saK aLlcpus showMem Nice powerOff showPc show-all-timers(Q) unRaw Sync showTasks Unmount shoW-blocked-tasks 

*** break sent ***
[  694.121182] SysRq : Show Blocked State
[  694.121182]   task                PC stack   pid father
[  694.121182] pdflush       D 00000000     0   263      2
[  694.121182]        f75a3760 00000046 c011f085 00000000 00000003 f75a38ec c3819020 00000000 
[  694.121182]        00000001 00011b31 c012126f 00000000 00000000 00000000 0000030c ffffffff 
[  694.121182]        f6592ac0 f6592b14 f6592af4 08fd0900 f89cf352 00000000 f75a3760 c01222b0 
[  694.121182] Call Trace:
[  694.121182]  [<c011f085>] __wake_up_common+0x2e/0x58
[  694.121182]  [<c012126f>] __wake_up+0x29/0x39
[  694.121182]  [<f89cf352>] wait_barrier+0x7e/0xb7 [raid1]
[  694.121182]  [<c01222b0>] default_wake_function+0x0/0x8
[  694.121182]  [<f89d0b84>] make_request+0x70/0x5a0 [raid1]
[  694.121182]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  694.121182]  [<c015fc05>] mempool_alloc+0x1c/0xba
[  694.121182]  [<c01dd76c>] submit_bio+0xc6/0xcd
[  694.121182]  [<c019d2bf>] bio_alloc_bioset+0x9b/0xf3
[  694.121182]  [<c0199e27>] submit_bh+0xcf/0xed
[  694.121182]  [<c019b7d2>] __block_write_full_page+0x1fa/0x2da
[  694.121182]  [<f8c8c3fa>] ext3_get_block+0x0/0xd3 [ext3]
[  694.121182]  [<c019b993>] block_write_full_page+0xe1/0xea
[  694.121182]  [<f8c8c3fa>] ext3_get_block+0x0/0xd3 [ext3]
[  694.121182]  [<f8c8d889>] ext3_ordered_writepage+0xab/0x13d [ext3]
[  694.121182]  [<f8c8b030>] bget_one+0x0/0x7 [ext3]
[  694.121182]  [<c01628ed>] __writepage+0x8/0x21
[  694.121182]  [<c0162d68>] write_cache_pages+0x16a/0x27b
[  694.121182]  [<c01628e5>] __writepage+0x0/0x21
[  694.121182]  [<c0162e93>] generic_writepages+0x1a/0x21
[  694.121182]  [<c0162ec3>] do_writepages+0x29/0x30
[  694.121182]  [<c0196949>] __writeback_single_inode+0x127/0x251
[  694.121182]  [<f89626fe>] dm_table_any_congested+0x32/0x4e [dm_mod]
[  694.121182]  [<c0196d40>] sync_sb_inodes+0x17c/0x233
[  694.121182]  [<c01970b7>] writeback_inodes+0x53/0x99
[  694.121182]  [<c0163ad9>] pdflush+0x0/0x1cc
[  694.121182]  [<c0163794>] wb_kupdate+0x7b/0xdb
[  694.121182]  [<c0163c08>] pdflush+0x12f/0x1cc
[  694.121182]  [<c0163719>] wb_kupdate+0x0/0xdb
[  694.121182]  [<c013876b>] kthread+0x38/0x5d
[  694.121182]  [<c0138733>] kthread+0x0/0x5d
[  694.121182]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  694.121182]  =======================
[  694.121182] kjournald     D 00000000     0  1550      2
[  694.121182]        f78984a0 00000046 c011f085 00000000 00000003 f789862c c3866020 00000007 
[  694.121182]        00000001 000122e8 c012126f 00000000 00000000 00000000 00000000 ffffffff 
[  694.121182]        f6592ac0 f6592b14 f6592af4 08fd0900 f89cf352 00000000 f78984a0 c01222b0 
[  694.121182] Call Trace:
[  694.121182]  [<c011f085>] __wake_up_common+0x2e/0x58
[  694.121182]  [<c012126f>] __wake_up+0x29/0x39
[  694.121182]  [<f89cf352>] wait_barrier+0x7e/0xb7 [raid1]
[  694.121182]  [<c01222b0>] default_wake_function+0x0/0x8
[  694.121182]  [<f89d0b84>] make_request+0x70/0x5a0 [raid1]
[  694.121182]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  694.121182]  [<c015fc05>] mempool_alloc+0x1c/0xba
[  694.121182]  [<c01dd76c>] submit_bio+0xc6/0xcd
[  694.121182]  [<c019d2bf>] bio_alloc_bioset+0x9b/0xf3
[  694.121182]  [<c0199e27>] submit_bh+0xcf/0xed
[  694.121182]  [<f8c59d3b>] journal_commit_transaction+0x690/0xb3d [jbd]
[  694.121182]  [<c01309b4>] lock_timer_base+0x19/0x35
[  694.121182]  [<f8c5c468>] kjournald+0xa5/0x1c6 [jbd]
[  694.121182]  [<c013882c>] autoremove_wake_function+0x0/0x2d
[  694.121182]  [<f8c5c3c3>] kjournald+0x0/0x1c6 [jbd]
[  694.121182]  [<c013876b>] kthread+0x38/0x5d
[  694.121182]  [<c0138733>] kthread+0x0/0x5d
[  694.121182]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  694.121182]  =======================
[  694.121182] md127_resync  D 3ce2eced     0  2324      2
[  694.121182]        f68f1b80 00000046 c011f085 3ce2eced 00000089 f68f1d0c c383a020 00000003 
[  694.121182]        00000001 00000282 00000001 00000000 00000000 00000003 f6592ac0 f6592b14 
[  694.121182]        f6592ac0 f6592b14 f6592af4 f6592ac0 f89cf48b 00000000 f68f1b80 c01222b0 
[  694.121182] Call Trace:
[  694.121182]  [<c011f085>] __wake_up_common+0x2e/0x58
[  694.121182]  [<f89cf48b>] raise_barrier+0x100/0x133 [raid1]
[  694.121182]  [<c01222b0>] default_wake_function+0x0/0x8
[  694.121182]  [<f89d1291>] sync_request+0x1dd/0x59a [raid1]
[  694.121182]  [<f89e8b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  694.121182]  [<f89ef9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  694.121182]  [<c02c8f8d>] schedule+0x63b/0x66d
[  694.121182]  [<f89ef310>] md_thread+0x0/0xcd [md_mod]
[  694.121182]  [<f89ef3c7>] md_thread+0xb7/0xcd [md_mod]
[  694.121182]  [<f89ef310>] md_thread+0x0/0xcd [md_mod]
[  694.121182]  [<c013876b>] kthread+0x38/0x5d
[  694.121182]  [<c0138733>] kthread+0x0/0x5d
[  694.121182]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  694.121182]  =======================
[  694.121182] kjournald     D 00000000     0  2483      2
[  694.121182]        f7670e20 00000046 c011f085 00000000 00000003 f7670fac c3819020 00000000 
[  694.121182]        00000001 00014482 c012126f 00000000 00000000 00000000 00000000 ffffffff 
[  694.121182]        f6592ac0 f6592b14 f6592af4 08fd0900 f89cf352 00000000 f7670e20 c01222b0 
[  694.121182] Call Trace:
[  694.121182]  [<c011f085>] __wake_up_common+0x2e/0x58
[  694.121182]  [<c012126f>] __wake_up+0x29/0x39
[  694.121182]  [<f89cf352>] wait_barrier+0x7e/0xb7 [raid1]
[  694.121182]  [<c01222b0>] default_wake_function+0x0/0x8
[  694.121182]  [<f89d0b84>] make_request+0x70/0x5a0 [raid1]
[  694.121182]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  694.121182]  [<c015fc05>] mempool_alloc+0x1c/0xba
[  694.121182]  [<c01dd76c>] submit_bio+0xc6/0xcd
[  694.121182]  [<c019d2bf>] bio_alloc_bioset+0x9b/0xf3
[  694.121182]  [<c0199e27>] submit_bh+0xcf/0xed
[  694.121182]  [<f8c59d3b>] journal_commit_transaction+0x690/0xb3d [jbd]
[  694.121182]  [<c01309b4>] lock_timer_base+0x19/0x35
[  694.121182]  [<f8c5c468>] kjournald+0xa5/0x1c6 [jbd]
[  694.121182]  [<c013882c>] autoremove_wake_function+0x0/0x2d
[  694.121182]  [<f8c5c3c3>] kjournald+0x0/0x1c6 [jbd]
[  694.121182]  [<c013876b>] kthread+0x38/0x5d
[  694.121182]  [<c0138733>] kthread+0x0/0x5d
[  694.121182]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  694.121182]  =======================
[  694.121182] kjournald     D d3d0808b     0  2484      2
[  694.121182]        f61775a0 00000046 c38372f8 d3d0808b 00000089 f617772c c383a020 00000003 
[  694.121182]        c383a454 c38372f8 000048f9 00000000 d632cf95 00000089 00000246 c013893b 
[  694.121182]        f613ec14 f44a7cfc f613ec74 f613ec00 f8c5976b f44a7cc0 c383a020 f593d800 
[  694.121182] Call Trace:
[  694.121182]  [<c013893b>] prepare_to_wait+0x12/0x4d
[  694.121182]  [<f8c5976b>] journal_commit_transaction+0xc0/0xb3d [jbd]
[  694.121182]  [<c02c8f8d>] schedule+0x63b/0x66d
[  694.121182]  [<c013882c>] autoremove_wake_function+0x0/0x2d
[  694.121182]  [<c0130a14>] try_to_del_timer_sync+0x44/0x4a
[  694.121182]  [<f8c5c468>] kjournald+0xa5/0x1c6 [jbd]
[  694.121182]  [<c013882c>] autoremove_wake_function+0x0/0x2d
[  694.121182]  [<f8c5c3c3>] kjournald+0x0/0x1c6 [jbd]
[  694.121182]  [<c013876b>] kthread+0x38/0x5d
[  694.121182]  [<c0138733>] kthread+0x0/0x5d
[  694.121182]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  694.121182]  =======================
[  694.121182] kjournald     D 00000000     0  2485      2
[  694.121182]        f6a651c0 00000046 c011f085 00000000 00000003 f6a6534c c3845020 00000004 
[  694.121182]        00000001 00013f05 c012126f 00000000 00000000 00000000 0000030c ffffffff 
[  694.121182]        f6592ac0 f6592b14 f6592af4 08fd0900 f89cf352 00000000 f6a651c0 c01222b0 
[  694.121182] Call Trace:
[  694.121182]  [<c011f085>] __wake_up_common+0x2e/0x58
[  694.121182]  [<c012126f>] __wake_up+0x29/0x39
[  694.121182]  [<f89cf352>] wait_barrier+0x7e/0xb7 [raid1]
[  694.121182]  [<c01222b0>] default_wake_function+0x0/0x8
[  694.121182]  [<f89d0b84>] make_request+0x70/0x5a0 [raid1]
[  694.121182]  [<c0161cc3>] get_page_from_freelist+0xc1/0x3e9
[  694.121182]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  694.121182]  [<c015fc05>] mempool_alloc+0x1c/0xba
[  694.121182]  [<c01dd76c>] submit_bio+0xc6/0xcd
[  694.121182]  [<c019d2bf>] bio_alloc_bioset+0x9b/0xf3
[  694.121182]  [<c0199e27>] submit_bh+0xcf/0xed
[  694.121182]  [<c019ccab>] sync_dirty_buffer+0x4d/0x91
[  694.121182]  [<f8c59ed6>] journal_commit_transaction+0x82b/0xb3d [jbd]
[  694.121182]  [<c01309b4>] lock_timer_base+0x19/0x35
[  694.121182]  [<c0130a14>] try_to_del_timer_sync+0x44/0x4a
[  694.121182]  [<f8c5c468>] kjournald+0xa5/0x1c6 [jbd]
[  694.121182]  [<c013882c>] autoremove_wake_function+0x0/0x2d
[  694.121182]  [<f8c5c3c3>] kjournald+0x0/0x1c6 [jbd]
[  694.121182]  [<c013876b>] kthread+0x38/0x5d
[  694.121182]  [<c0138733>] kthread+0x0/0x5d
[  694.121182]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  694.121182]  =======================
[  694.121182] summer        D 00000000     0  3433   3430
[  694.121182]        f7806560 00000082 f6827a90 00000000 00000003 f78066ec c3866020 00000007 
[  694.121182]        00000001 00011aaa c012126f 00000000 00000000 00000000 00000000 ffffffff 
[  694.121182]        f6592ac0 f6592b14 f6592af4 08fd0900 f89cf352 00000000 f7806560 c01222b0 
[  694.121182] Call Trace:
[  694.121182]  [<c012126f>] __wake_up+0x29/0x39
[  694.121182]  [<f89cf352>] wait_barrier+0x7e/0xb7 [raid1]
[  694.121182]  [<c01222b0>] default_wake_function+0x0/0x8
[  694.121182]  [<f89d0b84>] make_request+0x70/0x5a0 [raid1]
[  694.121182]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  694.121182]  [<c01dd76c>] submit_bio+0xc6/0xcd
[  694.121182]  [<c015dbd9>] add_to_page_cache+0x64/0x7d
[  694.121182]  [<c01a126d>] mpage_end_io_read+0x0/0x53
[  694.121182]  [<c01a0546>] mpage_bio_submit+0x19/0x1d
[  694.121182]  [<c01a136e>] mpage_readpages+0xae/0xb8
[  694.121182]  [<f8c8ba5f>] ext3_readpages+0x0/0x15 [ext3]
[  694.121182]  [<c0163e40>] __do_page_cache_readahead+0xea/0x163
[  694.121182]  [<f8c8c3fa>] ext3_get_block+0x0/0xd3 [ext3]
[  694.121182]  [<c016418d>] page_cache_sync_readahead+0x15/0x19
[  694.121182]  [<c015f429>] generic_file_aio_read+0x200/0x4e1
[  694.121182]  [<c016e9a7>] vma_adjust+0x1c6/0x399
[  694.121182]  [<c017e4d2>] do_sync_read+0xbf/0xfe
[  694.121182]  [<c013882c>] autoremove_wake_function+0x0/0x2d
[  694.121182]  [<c01c4da9>] security_file_permission+0xc/0xd
[  694.121182]  [<c017e413>] do_sync_read+0x0/0xfe
[  694.121182]  [<c017ec62>] vfs_read+0x81/0x11e
[  694.121182]  [<c017f0b3>] sys_read+0x3c/0x63
[  694.121182]  [<c0108857>] sysenter_past_esp+0x78/0xb1
[  694.121182]  =======================
[  694.121182] ls            D f48dc588     0  3465   3453
[  694.121182]        f7670a00 00000082 f48dc4e0 f48dc588 f3b735c0 f7670b8c c3850020 00000005 
[  694.121182]        f6390b40 00011d6d c019a2ea 00000400 00000000 00000000 00000000 ffffffff 
[  694.121182]        f4c5be70 f60ff2c0 c3802624 f7c4047c f8c58617 00000000 f48fc70c 00000000 
[  694.121182] Call Trace:
[  694.121182]  [<c019a2ea>] __getblk+0x27/0x24e
[  694.121182]  [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[  694.121182]  [<c0138859>] wake_bit_function+0x0/0x3c
[  694.121182]  [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[  694.121182]  [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[  694.121182]  [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[  694.121182]  [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[  694.121182]  [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[  694.121182]  [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[  694.121182]  [<c018f21e>] touch_atime+0xc7/0xd1
[  694.121182]  [<c015d5fa>] generic_file_mmap+0x2a/0x3e
[  694.121182]  [<c016f604>] mmap_region+0x1c7/0x392
[  694.121182]  [<c016fb07>] do_mmap_pgoff+0x25d/0x2b0
[  694.121182]  [<c0167004>] sys_mmap_pgoff+0x9b/0xc2
[  694.121182]  [<c01088d2>] syscall_call+0x7/0xb
[  694.121182]  [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[  694.121182]  =======================
[  694.121182] bash          D f3305be0     0  3466   3132
[  694.121182]        f597ca00 00000082 f1d7ac50 f3305be0 f41836a0 f597cb8c c385b020 00000006 
[  694.121182]        f6390ae0 00012e32 c019a2ea 00000400 00000000 00000000 00000000 ffffffff 
[  694.121182]        f5473d8c f60ff2c0 c380080c f7deeafc f8c58617 00000000 f48fcaa4 00000000 
[  694.121182] Call Trace:
[  694.121182]  [<c019a2ea>] __getblk+0x27/0x24e
[  694.121182]  [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[  694.121182]  [<c0138859>] wake_bit_function+0x0/0x3c
[  694.121182]  [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[  694.121182]  [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[  694.121182]  [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[  694.121182]  [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[  694.121182]  [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[  694.121182]  [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[  694.121182]  [<c018f21e>] touch_atime+0xc7/0xd1
[  694.121182]  [<c015f6c3>] generic_file_aio_read+0x49a/0x4e1
[  694.121182]  [<c017e4d2>] do_sync_read+0xbf/0xfe
[  694.121182]  [<c013882c>] autoremove_wake_function+0x0/0x2d
[  694.121182]  [<f8c98905>] ext3_xattr_security_get+0x31/0x3b [ext3]
[  694.121182]  [<c01c4da9>] security_file_permission+0xc/0xd
[  694.121182]  [<c017e413>] do_sync_read+0x0/0xfe
[  694.121182]  [<c017ec62>] vfs_read+0x81/0x11e
[  694.121182]  [<c0181c56>] kernel_read+0x32/0x43
[  694.121182]  [<c0181d27>] prepare_binprm+0xc0/0xc4
[  694.121182]  [<c0182b07>] do_execve+0xd2/0x1c6
[  694.121182]  [<c010713b>] sys_execve+0x2a/0x4a
[  694.121182]  [<c0108857>] sysenter_past_esp+0x78/0xb1
[  694.121182]  [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[  694.121182]  =======================
[  694.121182] cat           D f3305ba8     0  3467   3311
[  694.121182]        f5824f20 00000086 f7031668 f3305ba8 f1d7ac88 f58250ac c385b020 00000006 
[  694.121182]        f6390b40 000144cd c019a2ea 00000400 00000000 00000000 00000000 ffffffff 
[  694.121182]        f4cdbe70 f60ff2c0 c3802624 f7c4047c f8c58617 00000000 f48fcbf8 00000000 
[  694.121182] Call Trace:
[  694.121182]  [<c019a2ea>] __getblk+0x27/0x24e
[  694.121182]  [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[  694.121182]  [<c0138859>] wake_bit_function+0x0/0x3c
[  694.121182]  [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[  694.121182]  [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[  694.121182]  [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[  694.121182]  [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[  694.121182]  [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[  694.121182]  [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[  694.121182]  [<c018f21e>] touch_atime+0xc7/0xd1
[  694.121182]  [<c015d5fa>] generic_file_mmap+0x2a/0x3e
[  694.121182]  [<c016f604>] mmap_region+0x1c7/0x392
[  694.121182]  [<c016fb07>] do_mmap_pgoff+0x25d/0x2b0
[  694.121182]  [<c0167004>] sys_mmap_pgoff+0x9b/0xc2
[  694.121182]  [<c01088d2>] syscall_call+0x7/0xb
[  694.121182]  [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[  694.121182]  =======================
[  694.121182] cron          D f71b3908     0  3468   2969
[  694.121182]        f7b23200 00000082 f7057160 f71b3908 f71074a8 f7b2338c c385b020 00000006 
[  694.121182]        f6390960 000144ea c019a2ea 00000400 00000000 00000000 00000000 ffffffff 
[  694.121182]        f54cbd80 f60ff2c0 c3800c74 f7dec6ec f8c58617 00000000 f48fca04 00000000 
[  694.121182] Call Trace:
[  694.121182]  [<c019a2ea>] __getblk+0x27/0x24e
[  694.121182]  [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[  694.121182]  [<c0138859>] wake_bit_function+0x0/0x3c
[  694.121182]  [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[  694.121182]  [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[  694.121182]  [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[  694.121182]  [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[  694.121182]  [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[  694.121182]  [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[  694.121182]  [<c018f21e>] touch_atime+0xc7/0xd1
[  694.121182]  [<c01864e2>] __link_path_walk+0x7d8/0xb24
[  694.121182]  [<c018d022>] __d_lookup+0x98/0xd7
[  694.121182]  [<c0186865>] path_walk+0x37/0x70
[  694.121182]  [<c0186b14>] do_path_lookup+0x122/0x184
[  694.121182]  [<c01874f6>] __path_lookup_intent_open+0x42/0x72
[  694.121182]  [<c0187575>] path_lookup_open+0xf/0x13
[  694.121182]  [<c0187600>] do_filp_open+0x87/0x684
[  694.121182]  [<c011b3e1>] do_page_fault+0x4b2/0x8f9
[  694.121182]  [<c011b448>] do_page_fault+0x519/0x8f9
[  694.121182]  [<c017ce8c>] do_sys_open+0x40/0xb0
[  694.121182]  [<c017cf40>] sys_open+0x1e/0x23
[  694.121182]  [<c01088d2>] syscall_call+0x7/0xb
[  694.121182]  [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[  694.121182]  =======================
[  694.121182] cron          D f1e4ce80     0  3469   2969
[  694.121182]        f7661620 00000082 f3b3b390 f1e4ce80 f1e4cef0 f76617ac c3866020 00000007 
[  694.121182]        f6390960 000144ea c019a2ea 00000400 00000000 00000000 00000000 ffffffff 
[  694.121182]        f442ddc8 f60ff2c0 c3802b10 f7c40684 f8c58617 00000000 f48fca18 00000000 
[  694.121182] Call Trace:
[  694.121182]  [<c019a2ea>] __getblk+0x27/0x24e
[  694.121182]  [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[  694.121182]  [<c0138859>] wake_bit_function+0x0/0x3c
[  694.121182]  [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[  694.121182]  [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[  694.121182]  [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[  694.121182]  [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[  694.121182]  [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[  694.121182]  [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[  694.121182]  [<c018f21e>] touch_atime+0xc7/0xd1
[  694.121182]  [<c015f6c3>] generic_file_aio_read+0x49a/0x4e1
[  694.121182]  [<c017e4d2>] do_sync_read+0xbf/0xfe
[  694.121182]  [<c013882c>] autoremove_wake_function+0x0/0x2d
[  694.121182]  [<c01c4da9>] security_file_permission+0xc/0xd
[  694.121182]  [<c017e413>] do_sync_read+0x0/0xfe
[  694.121182]  [<c017ec62>] vfs_read+0x81/0x11e
[  694.121182]  [<c017f0b3>] sys_read+0x3c/0x63
[  694.121182]  [<c01088d2>] syscall_call+0x7/0xb
[  694.121182]  [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[  694.121182]  =======================
[  694.121182] cron          D 00000002     0  3470   2969
[  694.121182]        f76605a0 00000086 f799eaf8 00000002 00000000 f766072c c3845020 00000004 
[  694.121182]        00000000 000144ce 0003404e 00000000 00000000 00000000 00000000 ffffffff 
[  694.121182]        f613ec38 f613ec00 f4433e00 f44a7cc0 f8c58a05 f48fc360 00000002 00000000 
[  694.121182] Call Trace:
[  694.121182]  [<f8c58a05>] start_this_handle+0x23b/0x2aa [jbd]
[  694.121182]  [<c013882c>] autoremove_wake_function+0x0/0x2d
[  694.121182]  [<f8c58af1>] journal_start+0x7d/0xa9 [jbd]
[  694.121182]  [<f8c8ded9>] ext3_dirty_inode+0x21/0x63 [ext3]
[  694.121182]  [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[  694.121182]  [<c018f21e>] touch_atime+0xc7/0xd1
[  694.121182]  [<c015f6c3>] generic_file_aio_read+0x49a/0x4e1
[  694.121182]  [<c017e4d2>] do_sync_read+0xbf/0xfe
[  694.121182]  [<c013882c>] autoremove_wake_function+0x0/0x2d
[  694.121182]  [<c018c410>] fcntl_setlk+0x202/0x22a
[  694.121182]  [<c01c4da9>] security_file_permission+0xc/0xd
[  694.121182]  [<c017e413>] do_sync_read+0x0/0xfe
[  694.121182]  [<c017ec62>] vfs_read+0x81/0x11e
[  694.121182]  [<c017f0b3>] sys_read+0x3c/0x63
[  694.121182]  [<c0108857>] sysenter_past_esp+0x78/0xb1
[  694.121182]  [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[  694.121182]  =======================
[  694.121182] sh            D f7c2cef0     0  3472   3471
[  694.121182]        f7b9ce20 00000086 f7c11e10 f7c2cef0 f7c2c2b0 f7b9cfac c3866020 00000007 
[  694.121182]        f6390960 000144ea c019a2ea 00000400 00000000 00000000 00000000 ffffffff 
[  694.121182]        f4547d80 f60ff2c0 c3800c74 f7dec6ec f8c58617 00000000 f48fc5f4 00000000 
[  694.121182] Call Trace:
[  694.121182]  [<c019a2ea>] __getblk+0x27/0x24e
[  694.121182]  [<f8c58617>] do_get_write_access+0x1a4/0x331 [jbd]
[  694.121182]  [<c0138859>] wake_bit_function+0x0/0x3c
[  694.121182]  [<f8c587bc>] journal_get_write_access+0x18/0x26 [jbd]
[  694.121182]  [<f8c964f7>] __ext3_journal_get_write_access+0x13/0x32 [ext3]
[  694.121182]  [<f8c8b716>] ext3_reserve_inode_write+0x2d/0x5d [ext3]
[  694.121182]  [<f8c8b757>] ext3_mark_inode_dirty+0x11/0x27 [ext3]
[  694.121182]  [<f8c8df08>] ext3_dirty_inode+0x50/0x63 [ext3]
[  694.121182]  [<c0196f5b>] __mark_inode_dirty+0x21/0x12a
[  694.121182]  [<c018f21e>] touch_atime+0xc7/0xd1
[  694.121182]  [<c01864e2>] __link_path_walk+0x7d8/0xb24
[  694.121182]  [<c018d022>] __d_lookup+0x98/0xd7
[  694.121182]  [<c0186865>] path_walk+0x37/0x70
[  694.121182]  [<c0186b14>] do_path_lookup+0x122/0x184
[  694.121182]  [<c01874f6>] __path_lookup_intent_open+0x42/0x72
[  694.121182]  [<c0187575>] path_lookup_open+0xf/0x13
[  694.121182]  [<c0187600>] do_filp_open+0x87/0x684
[  694.121182]  [<c011b3e1>] do_page_fault+0x4b2/0x8f9
[  694.121182]  [<c011b448>] do_page_fault+0x519/0x8f9
[  694.121182]  [<c017ce8c>] do_sys_open+0x40/0xb0
[  694.121182]  [<c017cf40>] sys_open+0x1e/0x23
[  694.121182]  [<c01088d2>] syscall_call+0x7/0xb
[  694.121182]  [<c02c0000>] quirk_vt82c686_acpi+0x55/0x7b
[  694.121182]  =======================
[  694.121182] Sched Debug Version: v0.07, 2.6.26-2-686-bigmem #1
[  694.121182] now at 639513.043198 msecs
[  694.121182]   .sysctl_sched_latency                    : 80.000000
[  694.121182]   .sysctl_sched_min_granularity            : 16.000000
[  694.121182]   .sysctl_sched_wakeup_granularity         : 40.000000
[  694.121182]   .sysctl_sched_child_runs_first           : 0.000001
[  694.121182]   .sysctl_sched_features                   : 895
[  694.121182] 
[  694.121182] cpu#0, 1862.045 MHz
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_switches                   : 72329
[  694.121182]   .nr_load_updates               : 18539
[  694.121182]   .nr_uninterruptible            : 4294967243
[  694.121182]   .jiffies                       : 84895
[  694.121182]   .next_balance                  : 0.084896
[  694.121182]   .curr->pid                     : 0
[  694.121182]   .clock                         : 639560.113202
[  694.121182]   .cpu_load[0]                   : 0
[  694.121182]   .cpu_load[1]                   : 0
[  694.121182]   .cpu_load[2]                   : 0
[  694.121182]   .cpu_load[3]                   : 0
[  694.121182]   .cpu_load[4]                   : 26
[  694.121182] 
[  694.121182] cfs_rq[0]:
[  694.121182]   .exec_clock                    : 0.000000
[  694.121182]   .MIN_vruntime                  : 0.000001
[  694.121182]   .min_vruntime                  : 16348.622085
[  694.121182]   .max_vruntime                  : 0.000001
[  694.121182]   .spread                        : 0.000000
[  694.121182]   .spread0                       : 0.000000
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_spread_over                : 0
[  694.121182] 
[  694.121182] runnable tasks:
[  694.121182]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[  694.121182] ----------------------------------------------------------------------------------------------------------
[  694.121182] 
[  694.121182] cpu#1, 1862.045 MHz
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_switches                   : 29556
[  694.121182]   .nr_load_updates               : 5341
[  694.121182]   .nr_uninterruptible            : 4294967270
[  694.121182]   .jiffies                       : 84932
[  694.121182]   .next_balance                  : 0.084934
[  694.121182]   .curr->pid                     : 0
[  694.121182]   .clock                         : 639080.236160
[  694.121182]   .cpu_load[0]                   : 0
[  694.121182]   .cpu_load[1]                   : 0
[  694.121182]   .cpu_load[2]                   : 0
[  694.121182]   .cpu_load[3]                   : 0
[  694.121182]   .cpu_load[4]                   : 0
[  694.121182] 
[  694.121182] cfs_rq[1]:
[  694.121182]   .exec_clock                    : 0.000000
[  694.121182]   .MIN_vruntime                  : 0.000001
[  694.121182]   .min_vruntime                  : 7320.183309
[  694.121182]   .max_vruntime                  : 0.000001
[  694.121182]   .spread                        : 0.000000
[  694.121182]   .spread0                       : -9028.438776
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_spread_over                : 0
[  694.121182] 
[  694.121182] runnable tasks:
[  694.121182]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[  694.121182] ----------------------------------------------------------------------------------------------------------
[  694.121182] 
[  694.121182] cpu#2, 1862.045 MHz
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_switches                   : 66329
[  694.121182]   .nr_load_updates               : 5613
[  694.121182]   .nr_uninterruptible            : 4294967199
[  694.121182]   .jiffies                       : 84970
[  694.121182]   .next_balance                  : 0.084972
[  694.121182]   .curr->pid                     : 0
[  694.121182]   .clock                         : 639084.061939
[  694.121182]   .cpu_load[0]                   : 0
[  694.121182]   .cpu_load[1]                   : 0
[  694.121182]   .cpu_load[2]                   : 0
[  694.121182]   .cpu_load[3]                   : 0
[  694.121182]   .cpu_load[4]                   : 60
[  694.121182] 
[  694.121182] cfs_rq[2]:
[  694.121182]   .exec_clock                    : 0.000000
[  694.121182]   .MIN_vruntime                  : 0.000001
[  694.121182]   .min_vruntime                  : 12327.667563
[  694.121182]   .max_vruntime                  : 0.000001
[  694.121182]   .spread                        : 0.000000
[  694.121182]   .spread0                       : -4020.954522
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_spread_over                : 0
[  694.121182] 
[  694.121182] runnable tasks:
[  694.121182]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[  694.121182] ----------------------------------------------------------------------------------------------------------
[  694.121182] 
[  694.121182] cpu#3, 1862.045 MHz
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_switches                   : 1898699
[  694.121182]   .nr_load_updates               : 92084
[  694.121182]   .nr_uninterruptible            : 4294966813
[  694.121182]   .jiffies                       : 85008
[  694.121182]   .next_balance                  : 0.085009
[  694.121182]   .curr->pid                     : 0
[  694.121182]   .clock                         : 640051.975065
[  694.121182]   .cpu_load[0]                   : 0
[  694.121182]   .cpu_load[1]                   : 0
[  694.121182]   .cpu_load[2]                   : 0
[  694.121182]   .cpu_load[3]                   : 0
[  694.121182]   .cpu_load[4]                   : 0
[  694.121182] 
[  694.121182] cfs_rq[3]:
[  694.121182]   .exec_clock                    : 0.000000
[  694.121182]   .MIN_vruntime                  : 0.000001
[  694.121182]   .min_vruntime                  : 55749.380669
[  694.121182]   .max_vruntime                  : 0.000001
[  694.121182]   .spread                        : 0.000000
[  694.121182]   .spread0                       : 39400.758584
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_spread_over                : 0
[  694.121182] 
[  694.121182] runnable tasks:
[  694.121182]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[  694.121182] ----------------------------------------------------------------------------------------------------------
[  694.121182] 
[  694.121182] cpu#4, 1862.045 MHz
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_switches                   : 65011
[  694.121182]   .nr_load_updates               : 7806
[  694.121182]   .nr_uninterruptible            : 214
[  694.121182]   .jiffies                       : 85046
[  694.121182]   .next_balance                  : 0.085047
[  694.121182]   .curr->pid                     : 0
[  694.121182]   .clock                         : 637517.396452
[  694.121182]   .cpu_load[0]                   : 0
[  694.121182]   .cpu_load[1]                   : 511
[  694.121182]   .cpu_load[2]                   : 750
[  694.121182]   .cpu_load[3]                   : 853
[  694.121182]   .cpu_load[4]                   : 868
[  694.121182] 
[  694.121182] cfs_rq[4]:
[  694.121182]   .exec_clock                    : 0.000000
[  694.121182]   .MIN_vruntime                  : 0.000001
[  694.121182]   .min_vruntime                  : 45024.900733
[  694.121182]   .max_vruntime                  : 0.000001
[  694.121182]   .spread                        : 0.000000
[  694.121182]   .spread0                       : 28676.278648
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_spread_over                : 0
[  694.121182] 
[  694.121182] runnable tasks:
[  694.121182]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[  694.121182] ----------------------------------------------------------------------------------------------------------
[  694.121182] 
[  694.121182] cpu#5, 1862.045 MHz
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_switches                   : 34811
[  694.121182]   .nr_load_updates               : 5207
[  694.121182]   .nr_uninterruptible            : 128
[  694.121182]   .jiffies                       : 85084
[  694.121182]   .next_balance                  : 0.085085
[  694.121182]   .curr->pid                     : 0
[  694.121182]   .clock                         : 640352.338449
[  694.121182]   .cpu_load[0]                   : 0
[  694.121182]   .cpu_load[1]                   : 0
[  694.121182]   .cpu_load[2]                   : 0
[  694.121182]   .cpu_load[3]                   : 0
[  694.121182]   .cpu_load[4]                   : 30
[  694.121182] 
[  694.121182] cfs_rq[5]:
[  694.121182]   .exec_clock                    : 0.000000
[  694.121182]   .MIN_vruntime                  : 0.000001
[  694.121182]   .min_vruntime                  : 26072.394124
[  694.121182]   .max_vruntime                  : 0.000001
[  694.121182]   .spread                        : 0.000000
[  694.121182]   .spread0                       : 9723.772039
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_spread_over                : 0
[  694.121182] 
[  694.121182] runnable tasks:
[  694.121182]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[  694.121182] ----------------------------------------------------------------------------------------------------------
[  694.121182] 
[  694.121182] cpu#6, 1862.045 MHz
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_switches                   : 32304
[  694.121182]   .nr_load_updates               : 11575
[  694.121182]   .nr_uninterruptible            : 172
[  694.121182]   .jiffies                       : 85121
[  694.121182]   .next_balance                  : 0.085124
[  694.121182]   .curr->pid                     : 0
[  694.121182]   .clock                         : 640500.418721
[  694.121182]   .cpu_load[0]                   : 0
[  694.121182]   .cpu_load[1]                   : 0
[  694.121182]   .cpu_load[2]                   : 0
[  694.121182]   .cpu_load[3]                   : 0
[  694.121182]   .cpu_load[4]                   : 0
[  694.121182] 
[  694.121182] cfs_rq[6]:
[  694.121182]   .exec_clock                    : 0.000000
[  694.121182]   .MIN_vruntime                  : 0.000001
[  694.121182]   .min_vruntime                  : 37684.668142
[  694.121182]   .max_vruntime                  : 0.000001
[  694.121182]   .spread                        : 0.000000
[  694.121182]   .spread0                       : 21336.046057
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_spread_over                : 0
[  694.121182] 
[  694.121182] runnable tasks:
[  694.121182]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[  694.121182] ----------------------------------------------------------------------------------------------------------
[  694.121182] 
[  694.121182] cpu#7, 1862.045 MHz
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_switches                   : 2325749
[  694.121182]   .nr_load_updates               : 10926
[  694.121182]   .nr_uninterruptible            : 159
[  694.121182]   .jiffies                       : 85159
[  694.121182]   .next_balance                  : 0.085162
[  694.121182]   .curr->pid                     : 0
[  694.121182]   .clock                         : 640653.165765
[  694.121182]   .cpu_load[0]                   : 0
[  694.121182]   .cpu_load[1]                   : 0
[  694.121182]   .cpu_load[2]                   : 0
[  694.121182]   .cpu_load[3]                   : 0
[  694.121182]   .cpu_load[4]                   : 0
[  694.121182] 
[  694.121182] cfs_rq[7]:
[  694.121182]   .exec_clock                    : 0.000000
[  694.121182]   .MIN_vruntime                  : 0.000001
[  694.121182]   .min_vruntime                  : 31973.568990
[  694.121182]   .max_vruntime                  : 0.000001
[  694.121182]   .spread                        : 0.000000
[  694.121182]   .spread0                       : 15624.946905
[  694.121182]   .nr_running                    : 0
[  694.121182]   .load                          : 0
[  694.121182]   .nr_spread_over                : 0
[  694.121182] 
[  694.121182] runnable tasks:
[  694.121182]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[  694.121182] ----------------------------------------------------------------------------------------------------------
[  694.121182] 






*** break sent ***
[  708.530043] SysRq : Show backtrace of all active CPUs
[  708.530043] CPU4:
[  708.530043] 
[  708.530043] Pid: 0, comm: swapper Tainted: G        W (2.6.26-2-686-bigmem #1)
[  708.530043] EIP: 0060:[<f884908c>] EFLAGS: 00000202 CPU: 4
[  708.530043] EIP is at acpi_idle_enter_bm+0x2a0/0x315 [processor]
[  708.530043] EAX: c03d5430 EBX: 00001033 ECX: 00005050 EDX: 0346d000
[  708.530043] ESI: 00000000 EDI: 002cad1d EBP: 002c9cea ESP: f74b3f84
[  708.530043]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[  708.530043] CR0: 8005003b CR2: b77861be CR3: 00382000 CR4: 000006f0
[  708.530043] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[  708.530043] DR6: ffff0ff0 DR7: 00000400
[  708.530043]  [<c025b4f9>] ? cpuidle_idle_call+0x5b/0x86
[  708.530043]  [<c025b49e>] ? cpuidle_idle_call+0x0/0x86
[  708.530043]  [<c01075d3>] ? cpu_idle+0xb0/0xd0
[  708.530043]  =======================






*** break sent ***
[  723.072717] SysRq : HELP : loglevel0-8 reBoot Crashdump tErm Full kIll saK aLlcpus showMem Nice powerOff showPc show-all-timers(Q) unRaw Sync showTasks Unmount shoW-blocked-tasks 

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Thu, 22 Jul 2010 13:15:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Thu, 22 Jul 2010 13:15:03 GMT) Full text and rfc822 format available.

Message #73 received at 584881@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: Ian Jackson <ijackson@chiark.greenend.org.uk>, 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Thu, 22 Jul 2010 13:33:41 +0100
[Message part 1 (text/plain, inline)]
On Tue, 2010-07-20 at 22:51 +0100, Ian Jackson wrote: 
> Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> > Please try 2.6.34 from experimental.
> 
> I've now replicated the problem on my coffee table with a temporary
> (intermittent) loan of an identical machine from Jump networks.
> I can get 2.6.26-21lenny4 and 2.6.26-24 to crash on demand very
> easily.  In none of the crashes do I get any kind of kernel log
> messages (eg, soft lockup warnings).
[...]

But I do see a deadlock on the barrier between resync and normal I/O, as
you suspected.  Every process is blocked in wait_barrier(),
raise_barrier() or a filesystem lock.

The only explanations I can think of are:
1. The regular I/O requests were miscounted so that the resync process
will wait forever for them to complete in raise_barrier(), while the
other processes wait forever for the barrier to be lowered.  (It seems
like the directions of this barrier are inverted.  Maybe it is really a
bollard.)
2. The wait condition of one of the waiters was met, but it wasn't
woken.
3. A thread which raised the barrier failed to lower the barrier.
4. A thread which raised the barrier caused paging.

Unfortunately I can't see any evidence for any of these.

The following patch should give us some chance of detecting cases 1 and
3.  The instructions at
<http://kernel-handbook.alioth.debian.org/ch-common-tasks.html#s-common-official> explain how to rebuild an official kernel package with a patch applied.

Ben.

--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -657,6 +657,7 @@
 
 	/* block any new IO from starting */
 	conf->barrier++;
+	WARN_ON(conf->barrier != 1);
 
 	/* No wait for all pending IO to complete */
 	wait_event_lock_irq(conf->wait_barrier,
@@ -671,6 +672,7 @@
 {
 	unsigned long flags;
 	spin_lock_irqsave(&conf->resync_lock, flags);
+	WARN_ON(conf->barrier == 0);
 	conf->barrier--;
 	spin_unlock_irqrestore(&conf->resync_lock, flags);
 	wake_up(&conf->wait_barrier);
@@ -694,6 +696,7 @@
 {
 	unsigned long flags;
 	spin_lock_irqsave(&conf->resync_lock, flags);
+	WARN_ON(conf->nr_pending == 0);
 	conf->nr_pending--;
 	spin_unlock_irqrestore(&conf->resync_lock, flags);
 	wake_up(&conf->wait_barrier);
--- END ----

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Fri, 23 Jul 2010 20:51:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Fri, 23 Jul 2010 20:51:04 GMT) Full text and rfc822 format available.

Message #78 received at 584881@bugs.debian.org (full text, mbox):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Fri, 23 Jul 2010 21:47:22 +0100
[Message part 1 (text/plain, inline)]
Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> The following patch should give us some chance of detecting cases 1 and
> 3.  The instructions at
> <http://kernel-handbook.alioth.debian.org/ch-common-tasks.html#s-common-official> explain how to rebuild an official kernel package with a patch applied.

Thanks.  I did this, and the result was that this warning:

> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -657,6 +657,7 @@
>  
>  	/* block any new IO from starting */
>  	conf->barrier++;
> +	WARN_ON(conf->barrier != 1);

triggered an enormous number of times during boot and afterwards.  It
was a bit hard to tell but it seemed to be correlated with IO done by
the boot process.  It was spinning past at 115200bps at top whack,
mostly.

Eventually the system did boot.  I was able to log in and perform my
usual "get it to crash" test, with it spewing these warnings
continuously all the while.  I did have a bit of difficulty at first,
but I did succeed.  Between my last successful bulk disk access and my
being sure that the system had crashed, it stopped spewing the
warnings and became entirely silent.  Now it sits there locked up, and
I can still do IO on the console, but nothing is spewing out.

I'll attach the output for the booting, and the last bit before the
lockup.  I'm going to compile the kernel again without that particular
warning (and with kgdb support) and see if I can dig out anything
interesting.

Ian.

[lockup-log-head.txt (text/plain, inline)]
    GNU GRUB  version 0.97  (628K lower / 3135076K upper memory)

+-------------------------------------------------------------------------+||||||||||||||||||||||||+-------------------------------------------------------------------------+
    Use the ^ and v keys to select which entry is highlighted.
    Press enter to boot the selected OS, 'e' to edit the
    commands before booting, or 'c' for a command-line.  Debian GNU/Linux, kernel 2.6.35-rc5-686-bigmem                           Debian GNU/Linux, kernel 2.6.35-rc5-686-bigmem (recovery mode)           Debian GNU/Linux, kernel 2.6.26-2-686-bigmem                             Debian GNU/Linux, kernel 2.6.26-2-686-bigmem (recovery mode)             Debian GNU/Linux, kernel 2.6.26-2-686                                    Debian GNU/Linux, kernel 2.6.26-2-686 (recovery mode)                                                                                                                                                                                                                                                                                                                                                                                                                                                                           The highlighted entry will be booted automatically in 10 seconds.                                                                        Debian GNU/Linux, kernel 2.6.35-rc5-686-bigmem                           Debian GNU/Linux, kernel 2.6.35-rc5-686-bigmem (recovery mode)           Debian GNU/Linux, kernel 2.6.35-rc5-686-bigmem (recovery mode)           Debian GNU/Linux, kernel 2.6.26-2-686-bigmem                             Debian GNU/Linux, kernel 2.6.26-2-686-bigmem                             Debian GNU/Linux, kernel 2.6.26-2-686-bigmem (recovery mode)             Debian GNU/Linux, kernel 2.6.26-2-686-bigmem (recovery mode)             Debian GNU/Linux, kernel 2.6.26-2-686                                    Debian GNU/Linux, kernel 2.6.26-2-686                                    Debian GNU/Linux, kernel 2.6.26-2-686-bigmem (recovery mode)             Debian GNU/Linux, kernel 2.6.26-2-686-bigmem (recovery mode)             Debian GNU/Linux, kernel 2.6.26-2-686-bigmem                              Booting 'Debian GNU/Linux, kernel 2.6.26-2-686-bigmem'

root  (hd0,0)
 Filesystem type is ext2fs, partition type 0x83
kernel  /vmlinuz-2.6.26-2-686-bigmem ro root=/dev/vg-main/lv-root console=tty0 
console=ttyS0,115200n8 TERM=dumb reboot=bios,smp,warm 
   [Linux-bzImage, setup=0x3000, size=0x1772d0]
initrd  /initrd.img-2.6.26-2-686-bigmem
   [Linux-initrd @ 0x3788e000, 0x761b98 bytes]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.26-2-686-bigmem (Debian 2.6.26-24) (dannf@debian.org) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Thu Jul 22 21:32:17 BST 2010
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009d000 (usable)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000bf699000 (usable)
[    0.000000]  BIOS-e820: 00000000bf699000 - 00000000bf6af000 (reserved)
[    0.000000]  BIOS-e820: 00000000bf6af000 - 00000000bf6ce000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000bf6ce000 - 00000000c0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
[    0.000000] 4224MB HIGHMEM available.
[    0.000000] 896MB LOWMEM available.
[    0.000000] found SMP MP-table at [c00fe710] 000fe710
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA             0 ->     4096
[    0.000000]   Normal       4096 ->   229376
[    0.000000]   HighMem    229376 ->  1310720
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[1] active PFN ranges
[    0.000000]     0:        0 ->  1310720
[    0.000000] DMI 2.6 present.
[    0.000000] Using APIC driver default
[    0.000000] ACPI: RSDP 000F0B40, 0024 (r2 DELL  )
[    0.000000] ACPI: XSDT 000F0C40, 0094 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: FACP BF6C3BB4, 00F4 (r3 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: DSDT BF6AF000, 398C (r1 DELL   PE_SC3          1 INTL 20050624)
[    0.000000] ACPI: FACS BF6C6000, 0040
[    0.000000] ACPI: APIC BF6C3478, 0152 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: SPCR BF6C35CC, 0050 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: HPET BF6C3620, 0038 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: DM__ BF6C365C, 00A8 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: MCFG BF6C3850, 003C (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: WD__ BF6C3890, 0134 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: SLIC BF6C39C8, 0024 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: ERST BF6B2B0C, 0270 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: HEST BF6B2D7C, 03A8 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: BERT BF6B298C, 0030 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: EINJ BF6B29BC, 0150 (r1 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: TCPA BF6C3B4C, 0064 (r2 DELL   PE_SC3          1 DELL        1)
[    0.000000] ACPI: SSDT BF6C7000, 2164 (r1  INTEL PPM RCM  80000001 INTL 20061109)
[    0.000000] ACPI: PM-Timer IO Port: 0x808
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x28] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x29] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x2b] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2c] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2d] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x2e] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x2f] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x30] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x31] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x32] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x33] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x34] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x35] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x36] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x37] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0x38] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x39] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x3a] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x3b] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x3c] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x3d] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x3e] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0x3f] disabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
[    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] Allocating PCI resources starting at c2000000 (gap: c0000000:20000000)
[    0.000000] PM: Registered nosave memory: 000000000009d000 - 0000000000100000
[    0.000000] SMP: Allowing 32 CPUs, 24 hotplug CPUs
[    0.000000] PERCPU: Allocating 42184 bytes of per cpu data
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 1300480
[    0.000000] Kernel command line: ro root=/dev/vg-main/lv-root console=tty0 console=ttyS0,115200n8 TERM=dumb reboot=bios,smp,warm 
[    0.000000] Enabling fast FPU save and restore... done.
[    0.000000] Enabling unmasked SIMD FPU exception support... done.
[    0.000000] Initializing CPU#0
[    0.000000] PID hash table entries: 4096 (order: 12, 16384 bytes)
[    0.000000] Extended CMOS year: 2000
[    0.000000] Detected 1862.039 MHz processor.
[    0.004000] Console: colour VGA+ 80x25
[    0.004000] console [tty0] enabled
[    0.004000] console [ttyS0] enabled
[    0.004000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.004000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.004000] Memory: 4129272k/5242880k available (1836k kernel code, 54204k reserved, 790k data, 256k init, 3267172k highmem)
[    0.004000] virtual kernel memory layout:
[    0.004000]     fixmap  : 0xffe13000 - 0xfffff000   (1968 kB)
[    0.004000]     pkmap   : 0xffa00000 - 0xffc00000   (2048 kB)
[    0.004000]     vmalloc : 0xf8800000 - 0xff9fe000   ( 113 MB)
[    0.004000]     lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
[    0.004000]       .init : 0xc0399000 - 0xc03d9000   ( 256 kB)
[    0.004000]       .data : 0xc02cb1cd - 0xc0390ac0   ( 790 kB)
[    0.004000]       .text : 0xc0100000 - 0xc02cb1cd   (1836 kB)
[    0.004000] Checking if this processor honours the WP bit even in supervisor mode...Ok.
[    0.083864] Calibrating delay using timer specific routine.. 3727.31 BogoMIPS (lpj=7454621)
[    0.090895] Security Framework initialized
[    0.095052] SELinux:  Disabled at boot.
[    0.098953] Capability LSM initialized
[    0.102782] Mount-cache hash table entries: 512
[    0.107490] Initializing cgroup subsys ns
[    0.111567] Initializing cgroup subsys cpuacct
[    0.116007] Initializing cgroup subsys devices
[    0.120007] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.124007] CPU: L2 cache: 256K
[    0.127209] CPU: L3 cache: 8192K
[    0.130497] CPU: Physical Processor ID: 0
[    0.134573] CPU: Processor Core ID: 0
[    0.138303] Intel machine check architecture supported.
[    0.143593] Intel machine check reporting enabled on CPU#0.
[    0.148009] using mwait in idle threads.
[    0.151997] Checking 'hlt' instruction... OK.
[    0.172010] ACPI: Core revision 20080321
[    0.180011] ENABLING IO-APIC IRQs
[    0.183571] ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.227660] CPU0: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.236014] Booting processor 1/2 ip 6000
[    0.248015] Initializing CPU#1
[    0.248015] Calibrating delay using timer specific routine.. 3723.80 BogoMIPS (lpj=7447619)
[    0.248015] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.248015] CPU: L2 cache: 256K
[    0.248015] CPU: L3 cache: 8192K
[    0.248015] CPU: Physical Processor ID: 0
[    0.248015] CPU: Processor Core ID: 1
[    0.248015] Intel machine check architecture supported.
[    0.248015] Intel machine check reporting enabled on CPU#1.
[    0.328020] CPU1: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.337468] checking TSC synchronization [CPU#0 -> CPU#1]: passed.
[    0.344021] Booting processor 2/4 ip 6000
[    0.360022] Initializing CPU#2
[    0.360022] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447627)
[    0.360022] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.360022] CPU: L2 cache: 256K
[    0.360022] CPU: L3 cache: 8192K
[    0.360022] CPU: Physical Processor ID: 0
[    0.360022] CPU: Processor Core ID: 2
[    0.360022] Intel machine check architecture supported.
[    0.360022] Intel machine check reporting enabled on CPU#2.
[    0.436558] CPU2: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.448028] checking TSC synchronization [CPU#0 -> CPU#2]: passed.
[    0.452028] Booting processor 3/6 ip 6000
[    0.468029] Initializing CPU#3
[    0.468029] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447625)
[    0.468029] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.468029] CPU: L2 cache: 256K
[    0.468029] CPU: L3 cache: 8192K
[    0.468029] CPU: Physical Processor ID: 0
[    0.468029] CPU: Processor Core ID: 3
[    0.468029] Intel machine check architecture supported.
[    0.468029] Intel machine check reporting enabled on CPU#3.
[    0.544619] CPU3: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.556032] checking TSC synchronization [CPU#0 -> CPU#3]: passed.
[    0.560032] Booting processor 4/1 ip 6000
[    0.576036] Initializing CPU#4
[    0.576036] Calibrating delay using timer specific routine.. 3723.80 BogoMIPS (lpj=7447605)
[    0.576036] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.576036] CPU: L2 cache: 256K
[    0.576036] CPU: L3 cache: 8192K
[    0.576036] CPU: Physical Processor ID: 0
[    0.576036] CPU: Processor Core ID: 0
[    0.576036] Intel machine check architecture supported.
[    0.576036] Intel machine check reporting enabled on CPU#4.
[    0.652650] CPU4: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.664036] checking TSC synchronization [CPU#0 -> CPU#4]: passed.
[    0.668036] Booting processor 5/3 ip 6000
[    0.684042] Initializing CPU#5
[    0.684042] Calibrating delay using timer specific routine.. 3723.80 BogoMIPS (lpj=7447618)
[    0.684042] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.684042] CPU: L2 cache: 256K
[    0.684042] CPU: L3 cache: 8192K
[    0.684042] CPU: Physical Processor ID: 0
[    0.684042] CPU: Processor Core ID: 1
[    0.684042] Intel machine check architecture supported.
[    0.684042] Intel machine check reporting enabled on CPU#5.
[    0.760708] CPU5: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.772042] checking TSC synchronization [CPU#0 -> CPU#5]: passed.
[    0.776043] Booting processor 6/5 ip 6000
[    0.788049] Initializing CPU#6
[    0.788049] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447621)
[    0.788049] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.788049] CPU: L2 cache: 256K
[    0.788049] CPU: L3 cache: 8192K
[    0.788049] CPU: Physical Processor ID: 0
[    0.788049] CPU: Processor Core ID: 2
[    0.788049] Intel machine check architecture supported.
[    0.788049] Intel machine check reporting enabled on CPU#6.
[    0.868204] CPU6: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.880020] checking TSC synchronization [CPU#0 -> CPU#6]: passed.
[    0.884043] Booting processor 7/7 ip 6000
[    0.896056] Initializing CPU#7
[    0.896056] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447620)
[    0.896056] CPU: L1 I cache: 32K, L1 D cache: 32K
[    0.896056] CPU: L2 cache: 256K
[    0.896056] CPU: L3 cache: 8192K
[    0.896056] CPU: Physical Processor ID: 0
[    0.896056] CPU: Processor Core ID: 3
[    0.896056] Intel machine check architecture supported.
[    0.896056] Intel machine check reporting enabled on CPU#7.
[    0.976727] CPU7: Intel(R) Xeon(R) CPU           L3426  @ 1.87GHz stepping 05
[    0.988042] checking TSC synchronization [CPU#0 -> CPU#7]: passed.
[    0.992042] Brought up 8 CPUs
[    0.996024] Total of 8 processors activated (29793.97 BogoMIPS).
[    1.000997] net_namespace: 660 bytes
[    1.004640] Booting paravirtualized kernel on bare hardware
[    1.008997] NET: Registered protocol family 16
[    1.012997] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    1.019721] ACPI: bus type pci registered
[    1.023721] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
[    1.027721] PCI: MCFG area at e0000000 reserved in E820
[    1.034779] PCI: Using MMCONFIG for extended config space
[    1.038779] PCI: Using configuration type 1 for base access
[    1.042779] Setting up standard PCI resources
[    1.050819] ACPI: BIOS _OSI(Linux) query ignored
[    1.055495] ACPI: DMI System Vendor: Dell Inc.
[    1.059992] ACPI: DMI Product Name: PowerEdge R210
[    1.064838] ACPI: DMI Product Version: 
[    1.068729] ACPI: DMI Board Name: 0M877N
[    1.072710] ACPI: DMI BIOS Vendor: Dell Inc.
[    1.076974] ACPI: DMI BIOS Date: 01/28/2010
[    1.080974] ACPI: Please send DMI info above to linux-acpi@vger.kernel.org
[    1.086735] ACPI: If "acpi_osi=Linux" works better, please notify linux-acpi@vger.kernel.org
[    1.090735] ACPI: Interpreter enabled
[    1.094737] ACPI: (supports S0 S4 S5)
[    1.099730] ACPI: Using IOAPIC for interrupt routing
[    1.108079] ACPI: PCI Root Bridge [PCI0] (0000:00)
[    1.113170] PCI: Transparent bridge - 0000:00:1e.0
[    1.120794] ACPI: PCI Interrupt Link [LK00] (IRQs 3 4 5 6 7 10 11 14 *15)
[    1.124794] ACPI: PCI Interrupt Link [LK01] (IRQs 3 4 5 6 7 *10 11 14 15)
[    1.128794] ACPI: PCI Interrupt Link [LK02] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
[    1.137190] ACPI: PCI Interrupt Link [LK03] (IRQs 3 4 5 *6 7 10 11 14 15)
[    1.144891] ACPI: PCI Interrupt Link [LK04] (IRQs 3 4 5 6 7 10 *11 14 15)
[    1.152587] ACPI: PCI Interrupt Link [LK05] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
[    1.160794] ACPI: PCI Interrupt Link [LK06] (IRQs 3 4 5 6 7 10 11 *14 15)
[    1.168911] ACPI: PCI Interrupt Link [LK07] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
[    1.177834] Linux Plug and Play Support v0.97 (c) Adam Belay
[    1.181834] pnp: PnP ACPI init
[    1.185479] ACPI: bus type pnp registered
[    1.196246] pnp: PnP ACPI: found 13 devices
[    1.200246] ACPI: ACPI bus type pnp unregistered
[    1.204246] PnPBIOS: Disabled by ACPI PNP
[    1.208247] PCI: Using ACPI for IRQ routing
[    1.220247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    1.227050] hpet0: 8 64-bit timers, 14318180 Hz
[    1.232683] ACPI: RTC can wake from S4
[    1.258734] system 00:07: ioport range 0x800-0x87f has been reserved
[    1.265143] system 00:07: ioport range 0x880-0x8ff has been reserved
[    1.273026] system 00:07: ioport range 0x900-0x91f has been reserved
[    1.279436] system 00:07: ioport range 0x920-0x923 has been reserved
[    1.285868] system 00:07: ioport range 0x924-0x924 has been reserved
[    1.292291] system 00:07: ioport range 0xca0-0xca7 has been reserved
[    1.298703] system 00:07: ioport range 0xca9-0xcab has been reserved
[    1.305134] system 00:07: ioport range 0xcad-0xcaf has been reserved
[    1.311547] system 00:08: ioport range 0xca8-0xca8 has been reserved
[    1.317978] system 00:08: ioport range 0xcac-0xcac has been reserved
[    1.324406] system 00:0a: iomem range 0xe0000000-0xefffffff could not be reserved
[    1.331963] system 00:0c: iomem range 0xfed50000-0xfed53fff could not be reserved
[    1.370778] PCI: Bridge: 0000:00:03.0
[    1.374773]   IO window: f000-ffff
[    1.378781]   MEM window: 0xdf100000-0xdf1fffff
[    1.383367]   PREFETCH window: disabled.
[    1.387356] PCI: Bridge: 0000:00:1c.0
[    1.391081]   IO window: disabled.
[    1.394782]   MEM window: 0xda000000-0xddffffff
[    1.399376]   PREFETCH window: disabled.
[    1.403366] PCI: Bridge: 0000:00:1e.0
[    1.407088]   IO window: disabled.
[    1.410781]   MEM window: 0xde000000-0xdeffffff
[    1.415370]   PREFETCH window: 0x00000000d9800000-0x00000000d9ffffff
[    1.421167] ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 16 (level, low) -> IRQ 16
[    1.428350] NET: Registered protocol family 2
[    1.450143] IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
[    1.454656] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[    1.462810] TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
[    1.469822] TCP: Hash tables configured (established 131072 bind 65536)
[    1.476810] TCP reno registered
[    1.490368] NET: Registered protocol family 1
[    1.494850] checking if image is initramfs... it is
[    2.155659] Freeing initrd memory: 7558k freed
[    2.163663] audit: initializing netlink socket (disabled)
[    2.275640] type=2000 audit(1279924502.004:1): initialized
[    2.281857] highmem bounce pool size: 64 pages
[    2.572644] Total HugeTLB memory allocated, 0
[    2.581869] VFS: Disk quotas dquot_6.5.1
[    2.581891] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    2.590252] msgmni has been set to 1700
[    2.598154] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    2.614156] io scheduler noop registered
[    2.618155] io scheduler anticipatory registered
[    2.622248] io scheduler deadline registered
[    2.626589] io scheduler cfq registered (default)
[    2.648060] assign_interrupt_mode Found MSI capability
[    2.655515] assign_interrupt_mode Found MSI capability
[    2.665231] isapnp: Scanning for PnP cards...
[    3.022514] isapnp: No Plug & Play device found
[    3.032771] Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
[    3.038315] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    3.050941] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    3.057118] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    3.063028] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    3.069223] brd: module loaded
[    3.189669] PNP: No PS/2 controller found. Probing ports directly.
[    3.194528] i8042.c: No controller found.
[    3.200944] mice: PS/2 mouse device common for all mice
[    3.206276] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
[    3.211615] rtc0: alarms up to one day, y3k
[    3.218083] cpuidle: using governor ladder
[    3.222085] cpuidle: using governor menu
[    3.226064] No iBFT detected.
[    3.229286] TCP cubic registered
[    3.232572] NET: Registered protocol family 17
[    3.236888] Using IPI No-Shortcut mode
[    3.241211] registered taskstats version 1
[    3.245450] rtc_cmos 00:04: setting system clock to 2010-07-23 22:35:04 UTC (1279924504)
[    3.253504] Freeing unused kernel memory: 256k freed
Loading, please wait...
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... [    3.938926] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
[    3.943258] ACPI: ACPI0007:00 is registered as cooling_device0
[    4.863816] ------------[ cut here ]------------
[    4.867480] ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])
[    4.867480] ACPI: ACPI0007:01 is registered as cooling_device1
[    4.883473] WARNING: at arch/x86/kernel/hpet.c:299 hpet_legacy_next_event+0x38/0x5a()
[    4.883473] Modules linked in: processor(+) fan thermal_sys
[    4.883473] Pid: 0, comm: swapper Not tainted 2.6.26-2-686-bigmem #1
[    4.883473]  [<c0129183>] warn_on_slowpath+0x40/0x66
[    4.883473]  [<c01e7836>] __next_cpu+0x12/0x21
[    4.883473]  [<c0120239>] find_busiest_group+0x21e/0x623
[    4.883473]  [<c013cfa0>] getnstimeofday+0x37/0xbc
[    4.883473]  [<c01192bb>] hpet_legacy_next_event+0x38/0x5a
[    4.883473]  [<c013f6c8>] clockevents_program_event+0xc4/0xd2
[    4.883473]  [<c01403ca>] tick_dev_program_event+0x1e/0x82
[    4.883473]  [<c013fe50>] tick_broadcast_set_event+0xf/0x11
[    4.883473]  [<c01400da>] tick_broadcast_oneshot_control+0x8a/0xda
[    4.883473]  [<c013fb24>] tick_notify+0x1cd/0x2e3
[    4.883473]  [<c013ace6>] enqueue_hrtimer+0xc9/0xd4
[    4.883473]  [<c013be0c>] notifier_call_chain+0x2a/0x47
[    4.883473]  [<c013be50>] raw_notifier_call_chain+0x9/0xc
[    4.883473]  [<c013f5c8>] clockevents_notify+0x19/0x55
[    4.883473]  [<f8848f1b>] acpi_idle_enter_bm+0x12f/0x315 [processor]
[    4.883473]  [<c025b4f9>] cpuidle_idle_call+0x5b/0x86
[    4.883473]  [<c025b49e>] cpuidle_idle_call+0x0/0x86
[    4.883473]  [<c01075d3>] cpu_idle+0xb0/0xd0
[    4.883473]  =======================
[    4.883473] ---[ end trace 66948f2233c04087 ]---
[    5.007472] Marking TSC unstable due to: TSC halts in idle.
[    5.147488] ACPI: CPU2 (power states: C1[C1] C2[C2] C3[C3])
[    5.147488] ACPI: ACPI0007:02 is registered as cooling_device2
[    5.159525] ACPI: CPU3 (power states: C1[C1] C2[C2] C3[C3])
[    5.175510] ACPI: ACPI0007:03 is registered as cooling_device3
[    5.343505] ACPI: CPU4 (power states: C1[C1] C2[C2] C3[C3])
[    5.789408] ACPI: ACPI0007:04 is registered as cooling_device4
[    5.795299] ACPI: CPU5 (power states: C1[C1] C2[C2] C3[C3])
[    5.803608] ACPI: ACPI0007:05 is registered as cooling_device5
[    5.809538] ACPI: CPU6 (power states: C1[C1] C2[C2] C3[C3])
[    5.826296] ACPI: ACPI0007:06 is registered as cooling_device6
[    5.832181] ACPI: CPU7 (power states: C1[C1] C2[C2] C3[C3])
[    5.873327] ACPI: ACPI0007:07 is registered as cooling_device7
[    6.013722] usbcore: registered new interface driver usbfs
[    6.019313] usbcore: registered new interface driver hub
[    6.031034] usbcore: registered new device driver usb
[    6.032670] e1000e: Intel(R) PRO/1000 Network Driver - 0.3.3.3-k2
[    6.037893] e1000e: Copyright (c) 1999-2008 Intel Corporation.
[    6.047075] ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 16
[    6.063206] No dock devices found.
[    6.072740] SCSI subsystem initialized
[    6.173171] eth0: (PCI Express:2.5GB/s:Width x1) 00:1b:21:6b:5a:d0
[    6.179686] eth0: Intel(R) PRO/1000 Network Connection
[    6.184910] eth0: MAC: 3, PHY: 8, PBA No: e46981-003
[    6.191507] ACPI: PCI Interrupt 0000:00:1a.0[A] -> <6>udev: renamed network interface eth0 to eth3
[    6.203205] GSI 22 (level, low) -> IRQ 22
[    6.207288] ehci_hcd 0000:00:1a.0: EHCI Host Controller
[    6.212634] ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
[    6.224179] ehci_hcd 0000:00:1a.0: debug port 2
[    6.232062] ehci_hcd 0000:00:1a.0: irq 22, io mem 0xdf0fa000
[    6.250606] ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[    6.254705] usb usb1: configuration #1 chosen from 1 choice
[    6.264296] hub 1-0:1.0: USB hub found
[    6.272278] hub 1-0:1.0: 2 ports detected
[    6.378630] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    6.383032] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    6.390355] usb usb1: Product: EHCI Host Controller
[    6.395319] usb usb1: Manufacturer: Linux 2.6.26-2-686-bigmem ehci_hcd
[    6.402337] usb usb1: SerialNumber: 0000:00:1a.0
[    6.409887] ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 22 (level, low) -> IRQ 22
[    6.414823] ehci_hcd 0000:00:1d.0: EHCI Host Controller
[    6.422695] ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
[    6.438065] ehci_hcd 0000:00:1d.0: debug port 2
[    6.442705] ehci_hcd 0000:00:1d.0: irq 22, io mem 0xdf0fc000
[    6.462483] ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[    6.470090] usb usb2: configuration #1 chosen from 1 choice
[    6.474068] hub 2-0:1.0: USB hub found
[    6.482051] hub 2-0:1.0: 2 ports detected
[    6.589036] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[    6.591983] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    6.599985] usb usb2: Product: EHCI Host Controller
[    6.605015] usb usb2: Manufacturer: Linux 2.6.26-2-686-bigmem ehci_hcd
[    6.612353] usb usb2: SerialNumber: 0000:00:1d.0
[    6.617034] Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.7.5 (April 29, 2008)
[    6.617034] ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 20 (level, low) -> IRQ 20
[    6.620611] usb 1-1: new high speed USB device using ehci_hcd and address 2
[    6.642800] ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
[    6.652158] firmware: requesting bnx2-09-4.0.5.fw
[    6.665878] eth0: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem da000000, IRQ 16, node addr b8:ac:6f:87:77:89
[    6.695737] ACPI: PCI Interrupt 0000:02:00.1[B] -> GSI 17 (level, low) -> IRQ 17
[    6.704703] firmware: requesting bnx2-09-4.0.5.fw
[    6.715935] eth1: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem dc000000, IRQ 17, node addr b8:ac:6f:87:77:8a
[    6.739893] udev: renamed network interface eth1 to eth2
[    6.790342] udev: renamed network interface eth0_rename to eth1
[    6.791754] usb 1-1: configuration #1 chosen from 1 choice
[    6.791914] hub 1-1:1.0: USB hub found
[    6.792005] hub 1-1:1.0: 6 ports detected
[    6.899618] usb 1-1: New USB device found, idVendor=8087, idProduct=0020
[    6.904523] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    7.193415] usb 2-1: new high speed USB device using ehci_hcd and address 2
[    7.331178] usb 2-1: configuration #1 chosen from 1 choice
[    7.355242] hub 2-1:1.0: USB hub found
[    7.360394] hub 2-1:1.0: 8 ports detected
[    7.488156] usb 2-1: New USB device found, idVendor=8087, idProduct=0020
[    7.493867] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    7.699771] usb 2-1.1: new low speed USB device using ehci_hcd and address 3
[    7.741535] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
[    7.749852] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pmp pio slum part 
[    7.758594] scsi0 : ahci
[    7.758594] scsi1 : ahci
[    7.778883] scsi2 : ahci
[    7.782658] scsi3 : ahci
[    7.790221] scsi4 : ahci
[    7.790221] scsi5 : ahci
[    7.790221] ata1: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe100 irq 218
[    7.790221] ata2: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe180 irq 218
[    7.790221] ata3: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe200 irq 218
[    7.790221] ata4: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe280 irq 218
[    7.790221] ata5: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe300 irq 218
[    7.827018] usb 2-1.1: configuration #1 chosen from 1 choice
[    7.827185] usb 2-1.1: New USB device found, idVendor=0a81, idProduct=0101
[    7.827185] usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    7.827185] usb 2-1.1: Product: USB Keyboard
[    7.827185] usb 2-1.1: Manufacturer: CHESEN
[    7.945282] ata6: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe380 irq 218
[    8.122697] usb 2-1.5: new high speed USB device using ehci_hcd and address 4
[    8.220329] usb 2-1.5: configuration #1 chosen from 1 choice
[    8.225119] hub 2-1.5:1.0: USB hub found
[    8.231311] hub 2-1.5:1.0: 4 ports detected
[    8.346824] usb 2-1.5: New USB device found, idVendor=0424, idProduct=2514
[    8.351725] usb 2-1.5: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    8.362258] usbcore: registered new interface driver hiddev
[    8.374754] input: CHESEN USB Keyboard as /class/input/input0
[    8.386158] input,hidraw0: USB HID v1.10 Keyboard [CHESEN USB Keyboard] on usb-0000:00:1d.0-1.1
[    8.409581] input: CHESEN USB Keyboard as /class/input/input1
[    8.416083] input,hidraw1: USB HID v1.10 Device [CHESEN USB Keyboard] on usb-0000:00:1d.0-1.1
[    8.424940] usbcore: registered new interface driver usbhid
[    8.432929] usbhid: v2.6:USB HID core driver
[    8.463900] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    8.482198] ata1.00: ATA-8: WDC WD1602ABKS-18N8A0, 02.03B04, max UDMA/133
[    8.489167] ata1.00: 312500000 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    8.497295] ata1.00: configured for UDMA/133
[    8.996655] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    9.003969] ata2.00: ATA-8: WDC WD1602ABKS-18N8A0, 02.03B04, max UDMA/133
[    9.010969] ata2.00: 312500000 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    9.020017] ata2.00: configured for UDMA/133
[    9.359143] ata3: SATA link down (SStatus 0 SControl 300)
[    9.862534] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[    9.878187] ata4.00: ATAPI: TEAC DVD-ROM DV-28SW, R.2A, max UDMA/100, ATAPI AN
[    9.902001] ata4.00: configured for UDMA/100
[   10.242612] ata5: SATA link down (SStatus 0 SControl 300)
[   10.582428] ata6: SATA link down (SStatus 0 SControl 300)
[   10.709539] scsi 0:0:0:0: Direct-Access     ATA      WDC WD1602ABKS-1 02.0 PQ: 0 ANSI: 5
[   10.709539] scsi 1:0:0:0: Direct-Access     ATA      WDC WD1602ABKS-1 02.0 PQ: 0 ANSI: 5
[   10.849809] scsi 3:0:0:0: CD-ROM            TEAC     DVD-ROM DV-28SW  R.2A PQ: 0 ANSI: 5
[   10.892024] Driver 'sr' needs updating - please use bus_type methods
[   10.913465] Driver 'sd' needs updating - please use bus_type methods
[   10.924140] sd 0:0:0:0: [sda] 312500000 512-byte hardware sectors (160000 MB)
[   10.930641] sd 0:0:0:0: [sda] Write Protect is off
[   10.936887] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   10.938056] sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
[   10.938061] Uniform CD-ROM driver Revision: 3.20
[   10.972205] sd 0:0:0:0: [sda] 312500000 512-byte hardware sectors (160000 MB)
[   10.979552] sd 0:0:0:0: [sda] Write Protect is off
[   10.984492] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   10.993864]  sda:<5>sd 0:0:0:0: Attached scsi generic sg0 type 0
[   11.009149]  sda1 sda2
[   11.011450] scsi 1:0:0:0: Attached scsi generic sg1 type 0
[   11.013753] sd 0:0:0:0: [sda] Attached SCSI disk
[   11.013786] sd 1:0:0:0: [sdb] 312500000 512-byte hardware sectors (160000 MB)
[   11.017184] sd 1:0:0:0: [sdb] Write Protect is off
[   11.017208] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   11.017225] sd 1:0:0:0: [sdb] 312500000 512-byte hardware sectors (160000 MB)
[   11.013815] sd 1:0:0:0: [sdb] Write Protect is off
[   11.013815] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   11.013815]  sdb: sdb1 sdb2
[   11.076682] sr 3:0:0:0: Attached scsi generic sg2 type 5
[   11.076682] sd 1:0:0:0: [sdb] Attached SCSI disk
done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... Begin: Loading[   11.337836] device-mapper: uevent: version 1.0.3
[   11.355285] device-mapper: ioctl: 4.13.0-ioctl (2007-10-18) initialised: dm-devel@redhat.com
Success: loaded module dm-mod.
Success: loaded module dm-snapshot.
[   11.388754] md: raid0 personality registered for level 0
Success: loaded module raid0.
[   11.401037] md: raid1 personality registered for level 1
Success: loaded module raid1.
done.
+ lvm vgchange -a y chiark-early-a
  2 logical volume(s) in volume group "chiark-early-a" now active
+ lvm vgchange -a y chiark-early-b
  2 logical volume(s) in volume group "chiark-early-b" now active
mdadm: cannot open /dev/md/chiark-mirror: No [   11.529335] md: md127 stopped.
such file or directory
+ /sbin/mdadm --assemble --no-degraded --auto=md /dev/md/chiark-mirror
[   11.573007] md: bind<dm-2>
[   11.576085] md: bind<dm-0>
[   11.578882] md: md127: raid array is not clean -- starting background reconstruction
[   11.624950] raid1: raid set md127 active with 2 out of 2 mirrors
mdadm: /dev/md/chiark-mirror has been started with 2 drives.
  Reading all physical volumes.  This may take a while...
  Found volume group "vg-main" using metadata type lvm2
  Found volume group "chiark-early-b" using metadata type lvm2
  Found volume group "chiark-early-a" using metadata type lvm2
  9 logical volume(s) in volume group "vg-main" now active
EARLY ROOT LVM OK
Begin: Loadingdone.
Begin: Waiting for udev to process events ... done.
done.
Begin: Running /scripts/local-premount ... done.
[   12.243768] kjournald starting.  Commit interval 5 seconds
Begin: Running /[   12.243768] EXT3-fs: mounted filesystem with ordered data mode.
scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.

INIT: version 2.86 booting

Setting hostname to 'rescue'...done.
Files under mount point '/lib/init/rw' will be hidden. (warning).
Starting the hotplug events dispatcher: udevd[   13.196885] udevd version 125 started
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...[   13.619201] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
[   13.648270] input: PC Speaker as /class/input/input2
[   13.769390] input: Power Button (FF) as /class/input/input3
[   13.817584] ACPI: Power Button (FF) [PWRF]
done.
RAID status not OK. Exiting. failed!
Setting the system clock.
System Clock set to: Fri Jul 23 22:35:12 UTC 2010.
Not activating swap on logical volume. (warning).
Will now check root file system:fsck 1.41.3 (12-Oct-2008)
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/mapper/vg--main-lv--root 
/dev/mapper/vg--main-lv--root: clean, 16743/98304 files, 298478/393216 blocks
.
[   14.976504] md: resync of RAID array md127
[   14.981175] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[   14.986643] EXT3 FS on dm-4, internal journal
[   14.994599] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[   15.004184] md: using 128k window, over a total of 75400320 blocks.
[   15.011625] md: resuming resync of md127 from checkpoint.
[   15.020348] ------------[ cut here ]------------
[   15.023576] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   15.031350] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   15.059078] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   15.059078]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   15.059078]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   15.059078]  [<c01db25d>] elv_insert+0x102/0x196
[   15.059078]  [<c01dda71>] __make_request+0x2fe/0x348
[   15.059078]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   15.059078]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   15.059078]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   15.059078]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   15.059078]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   15.059078]  [<c012126f>] __wake_up+0x29/0x39
[   15.059078]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   15.059078]  [<c02c8f8d>] schedule+0x63b/0x66d
[   15.059078]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   15.059078]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   15.059078]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   15.059078]  [<c013876b>] kthread+0x38/0x5d
[   15.059078]  [<c0138733>] kthread+0x0/0x5d
[   15.059078]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   15.059078]  =======================
[   15.059078] ---[ end trace 66948f2233c04087 ]---
[   15.208006] ------------[ cut here ]------------
[   15.208006] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
Setting the syst[   15.208006] Modules linked in:em clock.
 button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   15.251058] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   15.251058]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   15.251058]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   15.251058]  [<c01db25d>] elv_insert+0x102/0x196
[   15.251058]  [<c01dda71>] __make_request+0x2fe/0x348
[   15.251058]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   15.251058]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   15.251058]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   15.251058]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   15.251058]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   15.251058]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   15.251058]  [<c02c8f8d>] schedule+0x63b/0x66d
[   15.251058]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   15.251058]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   15.251058]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   15.251058]  [<c013876b>] kthread+0x38/0x5d
[   15.251058]  [<c0138733>] kthread+0x0/0x5d
[   15.251058]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   15.251058]  =======================
[   15.251058] ---[ end trace 66948f2233c04087 ]---
[   15.377706] ------------[ cut here ]------------
System Clock set[   15.382407] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
 to: Fri Jul 23 [   15.394056] Modules linked in:22:35:13 UTC 201 button0.
 pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   15.426049] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   15.426049]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   15.426049]  [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[   15.426049]  [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[   15.426049]  [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[   15.426049]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[   15.426049]  [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[   15.426049]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[   15.426049]  [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[   15.426049]  [<c013cfa0>] getnstimeofday+0x37/0xbc
[   15.426049]  [<c013b418>] ktime_get_ts+0x22/0x49
[   15.426049]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   15.426049]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   15.426049]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[   15.426049]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   15.426049]  [<c02c8f8d>] schedule+0x63b/0x66d
[   15.426049]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   15.426049]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   15.426049]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   15.426049]  [<c013876b>] kthread+0x38/0x5d
[   15.426049]  [<c0138733>] kthread+0x0/0x5d
[   15.426049]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   15.426049]  =======================
[   15.426049] ---[ end trace 66948f2233c04087 ]---
[   15.565176] ------------[ cut here ]------------
[   15.569881] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   15.577155] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   15.605927] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   15.605927]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   15.605927]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   15.605927]  [<c01db25d>] elv_insert+0x102/0x196
[   15.605927]  [<c01dda71>] __make_request+0x2fe/0x348
[   15.605927]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   15.605927]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   15.605927]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   15.605927]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   15.605927]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   15.605927]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   15.605927]  [<c02c8f8d>] schedule+0x63b/0x66d
[   15.605927]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   15.605927]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   15.605927]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   15.605927]  [<c013876b>] kthread+0x38/0x5d
[   15.605927]  [<c0138733>] kthread+0x0/0x5d
[   15.605927]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   15.605927]  =======================
[   15.605927] ---[ end trace 66948f2233c04087 ]---
[   15.716679] ------------[ cut here ]------------
[   15.721394] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   15.728630] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   15.753489] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   15.753489]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   15.753489]  [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[   15.753489]  [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[   15.753489]  [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[   15.753489]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[   15.753489]  [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[   15.753489]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[   15.753489]  [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[   15.753489]  [<c013cfa0>] getnstimeofday+0x37/0xbc
[   15.753489]  [<c013b418>] ktime_get_ts+0x22/0x49
[   15.753489]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   15.753489]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   15.753489]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[   15.753489]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   15.753489]  [<c02c8f8d>] schedule+0x63b/0x66d
[   15.753489]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   15.753489]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   15.753489]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   15.753489]  [<c013876b>] kthread+0x38/0x5d
[   15.753489]  [<c0138733>] kthread+0x0/0x5d
[   15.753489]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   15.753489]  =======================
[   15.753489] ---[ end trace 66948f2233c04087 ]---
[   15.972618] ------------[ cut here ]------------
[   15.977362] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   15.984597] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   16.009364] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   16.009364]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   16.009364]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   16.009364]  [<c01db25d>] elv_insert+0x102/0x196
[   16.009364]  [<c01dda71>] __make_request+0x2fe/0x348
[   16.009364]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   16.009364]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   16.009364]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   16.009364]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   16.009364]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   16.009364]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   16.009364]  [<c02c8f8d>] schedule+0x63b/0x66d
[   16.009364]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   16.009364]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   16.009364]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   16.009364]  [<c013876b>] kthread+0x38/0x5d
[   16.009364]  [<c0138733>] kthread+0x0/0x5d
[   16.009364]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   16.009364]  =======================
[   16.009364] ---[ end trace 66948f2233c04087 ]---
[   16.133608] ------------[ cut here ]------------
[   16.136222] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   16.142045] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   16.171563] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   16.171563]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   16.171563]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   16.171563]  [<c01db25d>] elv_insert+0x102/0x196
[   16.171563]  [<c01dda71>] __make_request+0x2fe/0x348
[   16.171563]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   16.171563]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   16.171563]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   16.171563]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   16.171563]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   16.171563]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   16.171563]  [<c02c8f8d>] schedule+0x63b/0x66d
[   16.171563]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   16.171563]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   16.171563]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   16.171564]  [<c013876b>] kthread+0x38/0x5d
[   16.171564]  [<c0138733>] kthread+0x0/0x5d
[   16.171564]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   16.171564]  =======================
[   16.171564] ---[ end trace 66948f2233c04087 ]---
[   16.303096] ------------[ cut here ]------------
[   16.303096] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   16.303096] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   16.349056] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   16.349056]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   16.349056]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   16.349056]  [<c01db25d>] elv_insert+0x102/0x196
[   16.349056]  [<c01dda71>] __make_request+0x2fe/0x348
[   16.349056]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   16.349056]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   16.349056]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   16.349056]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   16.349056]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   16.349056]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   16.349056]  [<c02c8f8d>] schedule+0x63b/0x66d
[   16.349056]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   16.349056]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   16.349056]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   16.349056]  [<c013876b>] kthread+0x38/0x5d
[   16.349056]  [<c0138733>] kthread+0x0/0x5d
[   16.349056]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   16.349056]  =======================
[   16.349056] ---[ end trace 66948f2233c04087 ]---
[   16.657662] ------------[ cut here ]------------
[   16.657662] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   16.665653] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   16.697660] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   16.697660]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   16.697660]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   16.697660]  [<c01db25d>] elv_insert+0x102/0x196
[   16.697660]  [<c01dda71>] __make_request+0x2fe/0x348
[   16.697660]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   16.697660]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   16.697660]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   16.697660]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   16.697660]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   16.697660]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   16.697660]  [<c02c8f8d>] schedule+0x63b/0x66d
[   16.697660]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   16.697660]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   16.697660]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   16.697660]  [<c013876b>] kthread+0x38/0x5d
[   16.697660]  [<c0138733>] kthread+0x0/0x5d
[   16.697660]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   16.697660]  =======================
[   16.697660] ---[ end trace 66948f2233c04087 ]---
[   16.835780] ------------[ cut here ]------------
[   16.835780] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
Cleaning up ifup[   16.835780] Modules linked in:down... button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   16.871876] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   16.871876]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   16.871876]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   16.871876]  [<c01db25d>] elv_insert+0x102/0x196
[   16.871876]  [<c01dda71>] __make_request+0x2fe/0x348
[   16.871876]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   16.871876]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   16.871876]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   16.871876]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   16.871876]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   16.871876]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   16.871876]  [<c02c8f8d>] schedule+0x63b/0x66d
[   16.871876]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   16.871876]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   16.871876]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   16.871876]  [<c013876b>] kthread+0x38/0x5d
[   16.871876]  [<c0138733>] kthread+0x0/0x5d
[   16.871876]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   16.871876]  =======================
[   16.871876] ---[ end trace 66948f2233c04087 ]---
[   17.056673] ------------[ cut here ]------------
[   17.056673] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   17.056673] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   17.097066] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   17.097066]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   17.097066]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   17.097066]  [<c01db25d>] elv_insert+0x102/0x196
[   17.097066]  [<c01dda71>] __make_request+0x2fe/0x348
[   17.097066]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   17.097066]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   17.097066]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   17.097066]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   17.097066]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   17.097066]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   17.097066]  [<c02c8f8d>] schedule+0x63b/0x66d
[   17.097066]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   17.097066]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   17.097066]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   17.097066]  [<c013876b>] kthread+0x38/0x5d
[   17.097066]  [<c0138733>] kthread+0x0/0x5d
[   17.097066]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   17.097066]  =======================
[   17.097066] ---[ end trace 66948f2233c04087 ]---
[   17.246032] ------------[ cut here ]------------
.
[   17.246032] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   17.246032] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   17.300851] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   17.300851]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   17.300851]  [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[   17.300851]  [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[   17.300851]  [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[   17.300851]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[   17.300851]  [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[   17.300851]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[   17.300851]  [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[   17.300851]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[   17.300851]  [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[   17.300851]  [<c011fc26>] __resched_task+0x5c/0x5f
[   17.300851]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   17.300851]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   17.300851]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[   17.300851]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   17.300851]  [<c02c8f8d>] schedule+0x63b/0x66d
[   17.300851]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   17.300851]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   17.300851]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   17.300851]  [<c013876b>] kthread+0x38/0x5d
[   17.300851]  [<c0138733>] kthread+0x0/0x5d
[   17.300851]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   17.300851]  =======================
[   17.300851] ---[ end trace 66948f2233c04087 ]---
[   18.260106] ------------[ cut here ]------------
[   18.264845] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   18.272085] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   18.300856] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   18.300856]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   18.300856]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   18.300856]  [<c01db25d>] elv_insert+0x102/0x196
[   18.300856]  [<c01dda71>] __make_request+0x2fe/0x348
[   18.300856]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   18.300856]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   18.300856]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   18.300856]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   18.300856]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   18.300856]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   18.300856]  [<c02c8f8d>] schedule+0x63b/0x66d
[   18.300856]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   18.300856]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   18.300856]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   18.300856]  [<c013876b>] kthread+0x38/0x5d
[   18.300856]  [<c0138733>] kthread+0x0/0x5d
[   18.300856]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   18.300856]  =======================
[   18.300856] ---[ end trace 66948f2233c04087 ]---
[   18.435184] ------------[ cut here ]------------
[   18.435184] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   18.435184] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   18.474951] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   18.474951]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   18.474951]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   18.474951]  [<c01db25d>] elv_insert+0x102/0x196
[   18.474951]  [<c01dda71>] __make_request+0x2fe/0x348
[   18.474951]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   18.474951]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   18.474951]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   18.474951]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   18.474951]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   18.474951]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   18.474951]  [<c02c8f8d>] schedule+0x63b/0x66d
[   18.474951]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   18.474951]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   18.474951]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   18.474951]  [<c013876b>] kthread+0x38/0x5d
[   18.474951]  [<c0138733>] kthread+0x0/0x5d
[   18.474951]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   18.474951]  =======================
[   18.474951] ---[ end trace 66948f2233c04087 ]---
[   18.624832] ------------[ cut here ]------------
[   18.624832] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   18.624832] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   18.662446] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   18.662446]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   18.662446]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   18.662447]  [<c01db25d>] elv_insert+0x102/0x196
[   18.662447]  [<c01dda71>] __make_request+0x2fe/0x348
[   18.662447]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   18.662447]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   18.662447]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   18.662447]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   18.662447]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   18.662447]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   18.662447]  [<c02c8f8d>] schedule+0x63b/0x66d
[   18.662447]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   18.662447]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   18.662447]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   18.662447]  [<c013876b>] kthread+0x38/0x5d
[   18.662447]  [<c0138733>] kthread+0x0/0x5d
[   18.662447]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   18.662447]  =======================
[   18.662447] ---[ end trace 66948f2233c04087 ]---
[   18.792884] ------------[ cut here ]------------
[   18.792884] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   18.804882] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   18.832882] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   18.832882]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   18.832882]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   18.832882]  [<c01db25d>] elv_insert+0x102/0x196
[   18.832882]  [<c01dda71>] __make_request+0x2fe/0x348
[   18.832882]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   18.832882]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   18.832882]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   18.832882]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   18.832882]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   18.832882]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   18.832882]  [<c02c8f8d>] schedule+0x63b/0x66d
[   18.832882]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   18.832882]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   18.832882]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   18.832882]  [<c013876b>] kthread+0x38/0x5d
[   18.832882]  [<c0138733>] kthread+0x0/0x5d
[   18.832882]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   18.832882]  =======================
[   18.832882] ---[ end trace 66948f2233c04087 ]---
[   18.971732] ------------[ cut here ]------------
[   18.971732] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   18.971732] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   19.013689] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   19.013689]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   19.013689]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   19.013689]  [<c01db25d>] elv_insert+0x102/0x196
[   19.013689]  [<c01dda71>] __make_request+0x2fe/0x348
[   19.013689]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   19.013689]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   19.013689]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   19.013689]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   19.013689]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   19.013689]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   19.013689]  [<c02c8f8d>] schedule+0x63b/0x66d
[   19.013689]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   19.013689]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   19.013689]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   19.013689]  [<c013876b>] kthread+0x38/0x5d
[   19.013689]  [<c0138733>] kthread+0x0/0x5d
[   19.013689]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   19.013689]  =======================
[   19.013689] ---[ end trace 66948f2233c04087 ]---
Regulating system clock...[   19.180059] ------------[ cut here ]------------
[   19.180059] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   19.180059] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   19.227395] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   19.227395]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   19.227395]  [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[   19.227395]  [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[   19.227395]  [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[   19.227395]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[   19.227395]  [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[   19.227395]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[   19.227395]  [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[   19.227395]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[   19.227395]  [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[   19.227395]  [<c011fc26>] __resched_task+0x5c/0x5f
[   19.227395]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   19.227395]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   19.227395]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[   19.227395]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   19.227395]  [<c02c8f8d>] schedule+0x63b/0x66d
[   19.227395]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   19.227395]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   19.227395]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   19.227395]  [<c013876b>] kthread+0x38/0x5d
[   19.227395]  [<c0138733>] kthread+0x0/0x5d
[   19.227395]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   19.227395]  =======================
[   19.227395] ---[ end trace 66948f2233c04087 ]---
[   19.434488] ------------[ cut here ]------------
done.
[   19.434488] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   19.434488] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   19.434488] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   19.434488]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   19.434488]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   19.434488]  [<c01db25d>] elv_insert+0x102/0x196
[   19.434488]  [<c01dda71>] __make_request+0x2fe/0x348
[   19.434488]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   19.434488]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   19.434488]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   19.434488]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   19.434488]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   19.434488]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   19.434488]  [<c02c8f8d>] schedule+0x63b/0x66d
[   19.434488]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   19.434488]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   19.434488]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   19.434488]  [<c013876b>] kthread+0x38/0x5d
[   19.434488]  [<c0138733>] kthread+0x0/0x5d
[   19.434488]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   19.434488]  =======================
[   19.434488] ---[ end trace 66948f2233c04087 ]---
[   19.587444] ------------[ cut here ]------------
[   19.591555] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   19.595555] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   19.595555] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   19.595555]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   19.595555]  [<c01db14e>] elv_next_request+0x181/0x18e
[   19.595555]  [<f8913bf5>] scsi_request_fn+0x2e6/0x32a [scsi_mod]
[   19.595555]  [<c01dd25e>] __generic_unplug_device+0x1a/0x1c
[   19.595555]  [<c01db25d>] elv_insert+0x102/0x196
[   19.595555]  [<c01dda71>] __make_request+0x2fe/0x348
[   19.595555]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   19.595555]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   19.595555]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   19.726182]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   19.726182]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   19.726182]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   19.726182]  [<c02c8f8d>] schedule+0x63b/0x66d
[   19.726182]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   19.726182]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   19.726182]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   19.726182]  [<c013876b>] kthread+0x38/0x5d
[   19.726182]  [<c0138733>] kthread+0x0/0x5d
[   19.726182]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   19.726182]  =======================
[   19.726182] ---[ end trace 66948f2233c04087 ]---
[   19.827446] ------------[ cut here ]------------
[   19.827446] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   19.827446] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   19.827446] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   19.827446]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   19.827446]  [<c01db14e>] elv_next_request+0x181/0x18e
[   19.827446]  [<f8913bf5>] scsi_request_fn+0x2e6/0x32a [scsi_mod]
[   19.827446]  [<c01dd25e>] __generic_unplug_device+0x1a/0x1c
[   19.827446]  [<c01db25d>] elv_insert+0x102/0x196
[   19.827446]  [<c01dda71>] __make_request+0x2fe/0x348
[   19.827446]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   19.827446]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[   19.827446]  [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[   19.827446]  [<c011fc26>] __resched_task+0x5c/0x5f
[   19.827446]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   19.827446]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   19.827446]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[   19.827446]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   19.827446]  [<c02c8f8d>] schedule+0x63b/0x66d
[   19.827446]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   19.827446]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   19.827446]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   19.827446]  [<c013876b>] kthread+0x38/0x5d
[   19.827446]  [<c0138733>] kthread+0x0/0x5d
[   19.827446]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   19.827446]  =======================
[   19.827446] ---[ end trace 66948f2233c04087 ]---
[   19.995328] ------------[ cut here ]------------
[   19.999330] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   19.999330] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   19.999330] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   19.999330]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   19.999330]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   19.999330]  [<c01db25d>] elv_insert+0x102/0x196
[   19.999330]  [<c01dda71>] __make_request+0x2fe/0x348
[   19.999330]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   19.999330]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   19.999330]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   19.999330]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   19.999330]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   19.999330]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   19.999330]  [<c02c8f8d>] schedule+0x63b/0x66d
[   19.999330]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   19.999330]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   20.111311]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   20.111311]  [<c013876b>] kthread+0x38/0x5d
[   20.111311]  [<c0138733>] kthread+0x0/0x5d
[   20.111311]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   20.111311]  =======================
[   20.111311] ---[ end trace 66948f2233c04087 ]---
[   20.143237] ------------[ cut here ]------------
[   20.143237] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   20.143237] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   20.143237] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   20.143237]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   20.143237]  [<c01db14e>] elv_next_request+0x181/0x18e
[   20.143237]  [<f8913bf5>] scsi_request_fn+0x2e6/0x32a [scsi_mod]
[   20.143237]  [<c01dd25e>] __generic_unplug_device+0x1a/0x1c
[   20.143237]  [<c01db25d>] elv_insert+0x102/0x196
[   20.216826]  [<c01dda71>] __make_request+0x2fe/0x348
[   20.216826]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   20.216826]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[   20.216826]  [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[   20.216826]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   20.216826]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   20.216826]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[   20.216826]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   20.216826]  [<c02c8f8d>] schedule+0x63b/0x66d
[   20.216826]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   20.216826]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   20.216826]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   20.216826]  [<c013876b>] kthread+0x38/0x5d
[   20.216826]  [<c0138733>] kthread+0x0/0x5d
[   20.216826]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   20.216826]  =======================
[   20.216826] ---[ end trace 66948f2233c04087 ]---
[   20.334224] ------------[ cut here ]------------
[   20.420388] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   20.428360] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   20.500599] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   20.500599]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   20.500599]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   20.500599]  [<c01db25d>] elv_insert+0x102/0x196
[   20.500599]  [<c01dda71>] __make_request+0x2fe/0x348
[   20.500599]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   20.500599]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   20.500599]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   20.500599]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   20.500599]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   20.500599]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   20.500599]  [<c02c8f8d>] schedule+0x63b/0x66d
[   20.500599]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   20.500599]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   20.500599]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   20.500599]  [<c013876b>] kthread+0x38/0x5d
[   20.500599]  [<c0138733>] kthread+0x0/0x5d
[   20.500599]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   20.500599]  =======================
[   20.500599] ---[ end trace 66948f2233c04087 ]---
[   20.776950] ------------[ cut here ]------------
[   20.776950] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   20.776950] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   20.776950] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   20.776950]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   20.776950]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[   20.776950]  [<c01db25d>] elv_insert+0x102/0x196
[   20.776950]  [<c01dda71>] __make_request+0x2fe/0x348
[   20.776950]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[   20.776950]  [<c01dc690>] generic_make_request+0x34d/0x37b
[   20.776950]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   20.776950]  [<c019e0a3>] bio_add_page+0x25/0x2e
[   20.776950]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   20.776950]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   20.776950]  [<c02c8f8d>] schedule+0x63b/0x66d
[   20.776950]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   20.776950]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   20.776950]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   20.776950]  [<c013876b>] kthread+0x38/0x5d
[   20.776950]  [<c0138733>] kthread+0x0/0x5d
[   20.776950]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   20.776950]  =======================
[   20.776950] ---[ end trace 66948f2233c04087 ]---
[   20.925108] ------------[ cut here ]------------
[   20.929874] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[   20.934628] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[   20.962782] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bigmem #1
[   20.962782]  [<c0129183>] warn_on_slowpath+0x40/0x66
[   20.962782]  [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[   20.962782]  [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[   20.962782]  [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[   20.962782]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[   20.962782]  [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[   20.962782]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[   20.962782]  [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[   20.962782]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[   20.962782]  [<c01140ec>] smp_reschedule_interrupt+0xc/0x1e
[   20.962782]  [<c01092d8>] reschedule_interrupt+0x28/0x30
[   20.962782]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[   20.962782]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[   20.962782]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[   20.962782]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[   20.962782]  [<c02c8f8d>] schedule+0x63b/0x66d
[   20.962782]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   20.962782]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[   20.962782]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[   20.962782]  [<c013876b>] kthread+0x38/0x5d
[   20.962782]  [<c0138733>] kthread+0x0/0x5d
[   20.962782]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[   20.962782]  =======================
[   20.962782] ---[ end trace 66948f2233c04087 ]---
[   21.109461] ------------[ cut here ]------------
[   21.114153] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[lockup-log-tail.txt (text/plain, inline)]
[  302.224706]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.224709]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  302.224713]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.224714]  [<c019e0a3>] bio_add_page+0x25/0x2e
[  302.224717]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.224722]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.224732]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.224734]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.224739]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.224744]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.224748]  [<c013876b>] kthread+0x38/0x5d
[  302.224750]  [<c0138733>] kthread+0x0/0x5d
[  302.224751]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.224753]  =======================
[  302.224754] ---[ end trace 66948f2233c04087 ]---
[  302.224808] ------------[ cut here ]------------
[  302.224809] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.224811] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdev
[  302.224823] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-b1
[  302.224825]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.224831]  [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[  302.224835]  [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[  302.224841]  [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[  302.224854]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[  302.224866]  [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[  302.224880]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[  302.224892]  [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[  302.224906]  [<c01db14e>] elv_next_request+0x181/0x18e
[  302.224909]  [<f8913bf5>] scsi_request_fn+0x2e6/0x32a [scsi_mod]
[  302.224923]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.224926]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.224930]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  302.224935]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.224945]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.224947]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.224952]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.224957]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.224961]  [<c013876b>] kthread+0x38/0x5d
[  302.224963]  [<c0138733>] kthread+0x0/0x5d
[  302.224965]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.224967]  =======================
[  302.224968] ---[ end trace 66948f2233c04087 ]---
[  302.225005] ------------[ cut here ]------------
[  302.225007] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [rai)
[  302.225008] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdev
[  302.225020] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.225022]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.225027]  [<c01db14e>] elv_next_request+0x181/0x18e
[  302.225031]  [<f8913bf5>] scsi_request_fn+0x2e6/0x32a [scsi_mod]
[  302.225045]  [<c01dd25e>] __generic_unplug_device+0x1a/0x1c
[  302.225048]  [<c01db25d>] elv_insert+0x102/0x196
[  302.225051]  [<c01dda71>] __make_request+0x2fe/0x348
[  302.225053]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.225058]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  302.225063]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.225067]  [<c019e0a3>] bio_add_page+0x25/0x2e
[  302.225070]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.225077]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.225092]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.225095]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.225102]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.225109]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.225115]  [<c013876b>] kthread+0x38/0x5d
[  302.225117]  [<c0138733>] kthread+0x0/0x5d
[  302.225120]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.225123]  =======================
[  302.225127] ---[ end trace 66948f2233c04087 ]---
[  302.225148] ------------[ cut here ]------------
[  302.225149] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.225150] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evde]
[  302.225159] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.225160]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.225170]  [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[  302.225174]  [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[  302.225179]  [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[  302.225193]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[  302.225205]  [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[  302.225218]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[  302.225227]  [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[  302.225237]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[  302.225239]  [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[  302.225241]  [<c011fc26>] __resched_task+0x5c/0x5f
[  302.225244]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.225247]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.225251]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  302.225256]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.225266]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.225268]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.225273]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.225278]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.225282]  [<c013876b>] kthread+0x38/0x5d
[  302.225284]  [<c0138733>] kthread+0x0/0x5d
[  302.225285]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.225288]  =======================
[  302.225288] ---[ end trace 66948f2233c04087 ]---
[  302.225330] ------------[ cut here ]------------
[  302.225331] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.225332] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evde1
[  302.225345] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.225347]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.225352]  [<c01db14e>] elv_next_request+0x181/0x18e
[  302.225356]  [<f8913bf5>] scsi_request_fn+0x2e6/0x32a [scsi_mod]
[  302.225369]  [<c01dd25e>] __generic_unplug_device+0x1a/0x1c
[  302.225372]  [<c01db25d>] elv_insert+0x102/0x196
[  302.225375]  [<c01dda71>] __make_request+0x2fe/0x348
[  302.225378]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.225382]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  302.225388]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.225391]  [<c019e0a3>] bio_add_page+0x25/0x2e
[  302.225394]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.225402]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.225416]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.225419]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.225426]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.225433]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.225440]  [<c013876b>] kthread+0x38/0x5d
[  302.225442]  [<c0138733>] kthread+0x0/0x5d
[  302.225444]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.225450]  =======================
[  302.225451] ---[ end trace 66948f2233c04087 ]---
[  302.225472] ------------[ cut here ]------------
[  302.225473] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [rai)
[  302.225473] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdev
[  302.225482] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.225483]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.225493]  [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[  302.225498]  [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[  302.225503]  [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[  302.225516]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[  302.225528]  [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[  302.225542]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[  302.225554]  [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[  302.225569]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[  302.225572]  [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[  302.225575]  [<c011fc26>] __resched_task+0x5c/0x5f
[  302.225578]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.225583]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.225588]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  302.225596]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.225610]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.225613]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.225620]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.225628]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.225633]  [<c013876b>] kthread+0x38/0x5d
[  302.225634]  [<c0138733>] kthread+0x0/0x5d
[  302.225636]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.225638]  =======================
[  302.225639] ---[ end trace 66948f2233c04087 ]---
[  302.225691] ------------[ cut here ]------------
[  302.225692] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.225693] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdev
[  302.225706] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.225708]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.225713]  [<c01db14e>] elv_next_request+0x181/0x18e
[  302.225717]  [<f8913bf5>] scsi_request_fn+0x2e6/0x32a [scsi_mod]
[  302.225730]  [<c01dd25e>] __generic_unplug_device+0x1a/0x1c
[  302.225733]  [<c01db25d>] elv_insert+0x102/0x196
[  302.225736]  [<c01dda71>] __make_request+0x2fe/0x348
[  302.225739]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.225744]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  302.225749]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.225752]  [<c019e0a3>] bio_add_page+0x25/0x2e
[  302.225755]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.225763]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.225777]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.225780]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.225787]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.225794]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.225802]  [<c013876b>] kthread+0x38/0x5d
[  302.225804]  [<c0138733>] kthread+0x0/0x5d
[  302.225806]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.225808]  =======================
[  302.225809] ---[ end trace 66948f2233c04087 ]---
[  302.225846] ------------[ cut here ]------------
[  302.225847] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.225848] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evder
[  302.225861] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.225862]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.225869]  [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[  302.225873]  [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[  302.225878]  [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[  302.225892]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[  302.225904]  [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[  302.225917]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[  302.225929]  [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[  302.225943]  [<c01db14e>] elv_next_request+0x181/0x18e
[  302.225947]  [<f8913bf5>] scsi_request_fn+0x2e6/0x32a [scsi_mod]
[  302.225961]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.225966]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.225974]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  302.225979]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.225989]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.225991]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.225996]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.226001]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226005]  [<c013876b>] kthread+0x38/0x5d
[  302.226007]  [<c0138733>] kthread+0x0/0x5d
[  302.226009]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.226011]  =======================
[  302.226012] ---[ end trace 66948f2233c04087 ]---
[  302.226024] ------------[ cut here ]------------
[  302.226025] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.226026] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evde 
[  302.226035] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.226036]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.226040]  [<c01db14e>] elv_next_request+0x181/0x18e
[  302.226042]  [<f8913bf5>] scsi_request_fn+0x2e6/0x32a [scsi_mod]
[  302.226052]  [<c01dd25e>] __generic_unplug_device+0x1a/0x1c
[  302.226054]  [<c01db25d>] elv_insert+0x102/0x196
[  302.226056]  [<c01dda71>] __make_request+0x2fe/0x348
[  302.226058]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.226061]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  302.226065]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.226067]  [<c019e0a3>] bio_add_page+0x25/0x2e
[  302.226069]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.226074]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.226084]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.226086]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226091]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.226096]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226100]  [<c013876b>] kthread+0x38/0x5d
[  302.226102]  [<c0138733>] kthread+0x0/0x5d
[  302.226104]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.226106]  =======================
[  302.226107] ---[ end trace 66948f2233c04087 ]---
[  302.226145] ------------[ cut here ]------------
[  302.226146] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.226147] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdev
[  302.226160] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.226162]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.226167]  [<c01e6994>] cfq_completed_request+0x1c9/0x1ec
[  302.226171]  [<f8912400>] scsi_run_queue+0x200/0x219 [scsi_mod]
[  302.226184]  [<c01da971>] elv_queue_empty+0x1d/0x1e
[  302.226187]  [<c01dd47e>] __blk_run_queue+0xf/0xbe
[  302.226189]  [<c01dd545>] blk_run_queue+0x18/0x27
[  302.226192]  [<f89129b4>] scsi_next_command+0x25/0x2f [scsi_mod]
[  302.226206]  [<f8912b84>] scsi_end_request+0x62/0x6b [scsi_mod]
[  302.226219]  [<f8913680>] scsi_io_completion+0x1a6/0x363 [scsi_mod]
[  302.226234]  [<c01dd609>] blk_done_softirq+0x56/0x61
[  302.226237]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.226241]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.226244]  [<c01092ab>] common_interrupt+0x23/0x28
[  302.226248]  [<c02ca24b>] _spin_unlock_irqrestore+0x7/0x10
[  302.226251]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  302.226259]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.226274]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.226276]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226281]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.226286]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226290]  [<c013876b>] kthread+0x38/0x5d
[  302.226292]  [<c0138733>] kthread+0x0/0x5d
[  302.226293]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.226295]  =======================
[  302.226296] ---[ end trace 66948f2233c04087 ]---
[  302.226317] ------------[ cut here ]------------
[  302.226318] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.226319] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdev
[  302.226328] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.226329]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.226333]  [<c01db14e>] elv_next_request+0x181/0x18e
[  302.226335]  [<f8913bf5>] scsi_request_fn+0x2e6/0x32a [scsi_mod]
[  302.226345]  [<c01dd25e>] __generic_unplug_device+0x1a/0x1c
[  302.226347]  [<c01db25d>] elv_insert+0x102/0x196
[  302.226349]  [<c01dda71>] __make_request+0x2fe/0x348
[  302.226351]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.226354]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  302.226358]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.226360]  [<c019e0a3>] bio_add_page+0x25/0x2e
[  302.226362]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.226367]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.226377]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.226380]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226384]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.226389]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226394]  [<c013876b>] kthread+0x38/0x5d
[  302.226395]  [<c0138733>] kthread+0x0/0x5d
[  302.226397]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.226399]  =======================
[  302.226400] ---[ end trace 66948f2233c04087 ]---
[  302.226410] ------------[ cut here ]------------
[  302.226411] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.226412] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdea
[  302.226426] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.226428]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.226434]  [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[  302.226438]  [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[  302.226443]  [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[  302.226457]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[  302.226469]  [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[  302.226482]  [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[  302.226494]  [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[  302.226509]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[  302.226513]  [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[  302.226515]  [<c011fc26>] __resched_task+0x5c/0x5f
[  302.226519]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.226523]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.226529]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  302.226536]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.226551]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.226554]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226561]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.226567]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226571]  [<c013876b>] kthread+0x38/0x5d
[  302.226572]  [<c0138733>] kthread+0x0/0x5d
[  302.226574]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.226576]  =======================
[  302.226577] ---[ end trace 66948f2233c04087 ]---
[  302.226592] ------------[ cut here ]------------
[  302.226593] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.226594] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evded
[  302.226603] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.226604]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.226607]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[  302.226611]  [<c01db25d>] elv_insert+0x102/0x196
[  302.226613]  [<c01dda71>] __make_request+0x2fe/0x348
[  302.226615]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.226618]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  302.226622]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.226624]  [<c019e0a3>] bio_add_page+0x25/0x2e
[  302.226626]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.226631]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.226641]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.226643]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226648]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.226653]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226657]  [<c013876b>] kthread+0x38/0x5d
[  302.226659]  [<c0138733>] kthread+0x0/0x5d
[  302.226660]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.226663]  =======================
[  302.226663] ---[ end trace 66948f2233c04087 ]---
[  302.226672] ------------[ cut here ]------------
[  302.226673] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [rai)
[  302.226674] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdev
[  302.226682] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.226689]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.226695]  [<c01e67a4>] cfq_allow_merge+0x33/0x5a
[  302.226697]  [<c01e041a>] ll_back_merge_fn+0x18a/0x1e8
[  302.226701]  [<c01dad5b>] elv_merged_request+0x28/0x30
[  302.226704]  [<c01dda07>] __make_request+0x294/0x348
[  302.226706]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.226712]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[  302.226715]  [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[  302.226718]  [<c011fc26>] __resched_task+0x5c/0x5f
[  302.226721]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.226725]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.226730]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  302.226738]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.226752]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.226755]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226762]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.226769]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226775]  [<c013876b>] kthread+0x38/0x5d
[  302.226778]  [<c0138733>] kthread+0x0/0x5d
[  302.226780]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.226783]  =======================
[  302.226784] ---[ end trace 66948f2233c04087 ]---
[  302.226820] ------------[ cut here ]------------
[  302.226822] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.226823] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdem
[  302.226835] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.226837]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.226842]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[  302.226847]  [<c01db25d>] elv_insert+0x102/0x196
[  302.226850]  [<c01dda71>] __make_request+0x2fe/0x348
[  302.226853]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.226857]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  302.226863]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.226866]  [<c019e0a3>] bio_add_page+0x25/0x2e
[  302.226869]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.226876]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.226890]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.226893]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226900]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.226907]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.226913]  [<c013876b>] kthread+0x38/0x5d
[  302.226916]  [<c0138733>] kthread+0x0/0x5d
[  302.226918]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.226921]  =======================
[  302.226922] ---[ end trace 66948f2233c04087 ]---
[  302.226931] ------------[ cut here ]------------
[  302.226933] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.226934] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdev
[  302.226946] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.226948]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.226954]  [<c01e67a4>] cfq_allow_merge+0x33/0x5a
[  302.226957]  [<c01e041a>] ll_back_merge_fn+0x18a/0x1e8
[  302.226960]  [<c01dad5b>] elv_merged_request+0x28/0x30
[  302.226963]  [<c01dda07>] __make_request+0x294/0x348
[  302.226965]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.226971]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[  302.226974]  [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[  302.226977]  [<c011fc26>] __resched_task+0x5c/0x5f
[  302.226980]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.226984]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.226990]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  302.226997]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.227011]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.227014]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.227021]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.227028]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.227034]  [<c013876b>] kthread+0x38/0x5d
[  302.227036]  [<c0138733>] kthread+0x0/0x5d
[  302.227039]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.227042]  =======================
[  302.227043] ---[ end trace 66948f2233c04087 ]---
[  302.227120] ------------[ cut here ]------------
[  302.227121] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.227122] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evder
[  302.227135] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.227137]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.227142]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[  302.227146]  [<c01db25d>] elv_insert+0x102/0x196
[  302.227150]  [<c01dda71>] __make_request+0x2fe/0x348
[  302.227154]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  302.227158]  [<c01092ab>] common_interrupt+0x23/0x28
[  302.227161]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.227164]  [<c019e0a3>] bio_add_page+0x25/0x2e
[  302.227168]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.227174]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.227189]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.227192]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.227199]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.227206]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.227212]  [<c013876b>] kthread+0x38/0x5d
[  302.227214]  [<c0138733>] kthread+0x0/0x5d
[  302.227217]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.227220]  =======================
[  302.227221] ---[ end trace 66948f2233c04087 ]---
[  302.227230] ------------[ cut here ]------------
[  302.227232] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.227233] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdev
[  302.227245] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.227247]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.227253]  [<c01e67a4>] cfq_allow_merge+0x33/0x5a
[  302.227255]  [<c01e041a>] ll_back_merge_fn+0x18a/0x1e8
[  302.227259]  [<c01dad5b>] elv_merged_request+0x28/0x30
[  302.227262]  [<c01dda07>] __make_request+0x294/0x348
[  302.227264]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.227270]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[  302.227273]  [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[  302.227276]  [<c011fc26>] __resched_task+0x5c/0x5f
[  302.227279]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.227283]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.227288]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  302.227296]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.227310]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.227313]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.227320]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.227327]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.227333]  [<c013876b>] kthread+0x38/0x5d
[  302.227335]  [<c0138733>] kthread+0x0/0x5d
[  302.227338]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.227341]  =======================
[  302.227342] ---[ end trace 66948f2233c04087 ]---
[  302.227394] ------------[ cut here ]------------
[  302.227395] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  302.227396] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evde 
[  302.227409] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  302.227411]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  302.227416]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[  302.227420]  [<c01db25d>] elv_insert+0x102/0x196
[  302.227424]  [<c01dda71>] __make_request+0x2fe/0x348
[  302.227426]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  302.227430]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  302.227436]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  302.227439]  [<c019e0a3>] bio_add_page+0x25/0x2e
[  302.227442]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  302.227449]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  302.227463]  [<c02c8f8d>] schedule+0x63b/0x66d
[  302.227466]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.227473]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  302.227480]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  302.227486]  [<c013876b>] kthread+0x38/0x5d
[  302.227489]  [<c0138733>] kthread+0x0/0x5d
[  302.227491]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  302.227494]  =======================
[  302.227495] ---[ end trace 66948f2233c04087 ]---
[  303.862186] ------------[ cut here ]------------
[  303.862188] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  303.862190] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evded
[  303.862411] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  303.862411]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  303.862411]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[  303.862411]  [<c01db25d>] elv_insert+0x102/0x196
[  303.862411]  [<c01dda71>] __make_request+0x2fe/0x348
[  303.862411]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  303.862411]  [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[  303.862411]  [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[  303.862411]  [<c011fc26>] __resched_task+0x5c/0x5f
[  303.862411]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  303.862411]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  303.862411]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  303.862411]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  303.862411]  [<c02c8f8d>] schedule+0x63b/0x66d
[  303.862411]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  303.862411]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  303.862411]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  303.862411]  [<c013876b>] kthread+0x38/0x5d
[  303.862411]  [<c0138733>] kthread+0x0/0x5d
[  303.862411]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  303.862411]  =======================
[  303.862411] ---[ end trace 66948f2233c04087 ]---
[  303.893767] ------------[ cut here ]------------
[  303.893769] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  303.893771] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdeo
[  303.893785] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  303.893788]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  303.893794]  [<c01e5f68>] cfq_insert_request+0x34/0x39e
[  303.893800]  [<c01db25d>] elv_insert+0x102/0x196
[  303.893803]  [<c01dda71>] __make_request+0x2fe/0x348
[  303.893806]  [<c013cfa0>] getnstimeofday+0x37/0xbc
[  303.893810]  [<c013b418>] ktime_get_ts+0x22/0x49
[  303.893813]  [<c013b44c>] ktime_get+0xd/0x21
[  303.893816]  [<c011fd14>] hrtick_start_fair+0xeb/0x12c
[  303.893820]  [<c0126950>] check_preempt_wakeup+0xa6/0xc6
[  303.893824]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  303.893829]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  303.893835]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  303.893843]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  303.893859]  [<c02c8f8d>] schedule+0x63b/0x66d
[  303.893862]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  303.893869]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  303.893877]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  303.893883]  [<c013876b>] kthread+0x38/0x5d
[  303.893885]  [<c0138733>] kthread+0x0/0x5d
[  303.893888]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  303.893891]  =======================
[  303.893893] ---[ end trace 66948f2233c04087 ]---
[  303.893902] ------------[ cut here ]------------
[  303.893904] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  303.893905] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evdev
[  303.893918] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  303.893921]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  303.893927]  [<c01e67a4>] cfq_allow_merge+0x33/0x5a
[  303.893930]  [<c01e041a>] ll_back_merge_fn+0x18a/0x1e8
[  303.893934]  [<c01dad5b>] elv_merged_request+0x28/0x30
[  303.893937]  [<c01dda07>] __make_request+0x294/0x348
[  303.893940]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  303.893945]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  303.893951]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  303.893954]  [<c019e0a3>] bio_add_page+0x25/0x2e
[  303.893959]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  303.893967]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  303.893982]  [<c02c8f8d>] schedule+0x63b/0x66d
[  303.893986]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  303.893993]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  303.894000]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  303.894006]  [<c013876b>] kthread+0x38/0x5d
[  303.894009]  [<c0138733>] kthread+0x0/0x5d
[  303.894011]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  303.894015]  =======================
[  303.894016] ---[ end trace 66948f2233c04087 ]---
[  303.894026] ------------[ cut here ]------------
[  303.894028] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid
[  303.894029] Modules linked in: slip slhc ipv6 ext2 button pcspkr dcdbas evder
[  303.894043] Pid: 2287, comm: md127_resync Tainted: G        W 2.6.26-2-686-bi
[  303.894045]  [<c0129183>] warn_on_slowpath+0x40/0x66
[  303.894052]  [<c01e67a4>] cfq_allow_merge+0x33/0x5a
[  303.894055]  [<c01e041a>] ll_back_merge_fn+0x18a/0x1e8
[  303.894059]  [<c01dad5b>] elv_merged_request+0x28/0x30
[  303.894062]  [<c01dda07>] __make_request+0x294/0x348
[  303.894065]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[  303.894069]  [<c01dc690>] generic_make_request+0x34d/0x37b
[  303.894075]  [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[  303.894080]  [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[  303.894085]  [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[  303.894093]  [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[  303.894107]  [<c02c8f8d>] schedule+0x63b/0x66d
[  303.894110]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  303.894117]  [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[  303.894124]  [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[  303.894130]  [<c013876b>] kthread+0x38/0x5d
[  303.894132]  [<c0138733>] kthread+0x0/0x5d
[  303.894135]  [<c01094f7>] kernel_thread_helper+0x7/0x10
[  303.894138]  =======================
[  303.894139] ---[ end trace 66948f2233c04087 ]---
   
    This is a private system.  Unauthorised access prohibited.
    If you do not know that you are authorised then you are not.
rescue login:     
    This is a private system.  Unauthorised access prohibited.
    If you do not know that you are authorised then you are not.
rescue login: ian


Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Fri, 23 Jul 2010 23:03:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Fri, 23 Jul 2010 23:03:03 GMT) Full text and rfc822 format available.

Message #83 received at 584881@bugs.debian.org (full text, mbox):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>
To: Ben Hutchings <ben@decadent.org.uk>, 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Fri, 23 Jul 2010 23:58:24 +0100
I wrote:
> ...  I'm going to compile the kernel again without that particular
> warning (and with kgdb support) and see if I can dig out anything
> interesting.

My attempt to compile the kernel with kgdb support failed.  Something
in the Debian kernel packaging thingy complained like this:
  ABI has changed!  Refusing to continue.
  Added symbols:
  kgdb_breakpoint                                  module: vmlinux,
  version: 0x01e1a8de, export: EXPORT_SYMBOL_GPL
...
  Changed symbols:
  __audit_inode_child                              module: vmlinux, version: 0xead6af60 -> 0xe3835d2d, export: EXPORT_SYMBOL_GPL
  __fatal_signal_pending                           module: vmlinux, version: 0xb5428735 -> 0xee8d0603, export: EXPORT_SYMBOL
  [lots more]

I've run out of time to do more on this now.

Ian.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Sat, 24 Jul 2010 00:15:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Sat, 24 Jul 2010 00:15:03 GMT) Full text and rfc822 format available.

Message #88 received at 584881@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: Ian Jackson <ijackson@chiark.greenend.org.uk>
Cc: 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Sat, 24 Jul 2010 01:10:24 +0100
[Message part 1 (text/plain, inline)]
On Fri, 2010-07-23 at 23:58 +0100, Ian Jackson wrote:
> I wrote:
> > ...  I'm going to compile the kernel again without that particular
> > warning (and with kgdb support) and see if I can dig out anything
> > interesting.
> 
> My attempt to compile the kernel with kgdb support failed.  Something
> in the Debian kernel packaging thingy complained like this:
>   ABI has changed!  Refusing to continue.
>   Added symbols:
>   kgdb_breakpoint                                  module: vmlinux,
>   version: 0x01e1a8de, export: EXPORT_SYMBOL_GPL
> ...
>   Changed symbols:
>   __audit_inode_child                              module: vmlinux, version: 0xead6af60 -> 0xe3835d2d, export: EXPORT_SYMBOL_GPL
>   __fatal_signal_pending                           module: vmlinux, version: 0xb5428735 -> 0xee8d0603, export: EXPORT_SYMBOL
>   [lots more]
> 
> I've run out of time to do more on this now.

If you remove the 'debian/abi' directory this check will be skipped.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Fri, 17 Sep 2010 16:21:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Tim Small <tim@seoss.co.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Fri, 17 Sep 2010 16:21:03 GMT) Full text and rfc822 format available.

Message #93 received at 584881@bugs.debian.org (full text, mbox):

From: Tim Small <tim@seoss.co.uk>
To: 584881@bugs.debian.org
Subject: Deadlock in md barrier code? / RAID1 / LVM CoW snapshot + ext3 / Debian 5.0 - lenny 2.6.26 kernel
Date: Fri, 17 Sep 2010 17:18:37 +0100
Hi,

I recently posted the below message to linux-raid, but perhaps it should 
have gone here first...  Perhaps Neil Brown will have some bright ideas.

Common factors on all three pieces of hardware seeing the problem seem 
to have been:

Lenny 2.6.26 kernel
Serial console
md with lvm snapshots
Dell hardware
4 or more cores


HTH,

Tim.




Hi,

I have a box with a relatively simple setup:

sda + sdb are 1TB SATA drives attached to an Intel ICH10.
Three partitions on each drive, three md raid1s built on top of these:

md0 /
md1 swap
md2 LVM PV


During resync about a week ago, processes seemed to deadlock on I/O, the 
machine was still alive but with a load of 100+.  A USB drive happened 
to be mounted, so I managed to save /var/log/kern.log  At the time of 
the problem, the monthly RAID check was in progress.  On reboot, a 
rebuild commenced, and the same deadlock seemed to occur between roughly 
2 minutes and 15 minutes after boot.

At this point, the server was running on a Dell PE R300 (12G RAM, 
quad-core), with an LSI SAS controller and 2x 500G SATA drives.  I 
shifted all the data onto a spare box (Dell PE R210, ICH10R, 1x1TB 
drive, 8G RAM, quad-core+HT), with only a single drive, so I created the 
md RAID1s with just a single drive in each.  The original box was put 
offline with the idea of me debugging it "soon".

This morning, I added in a second 1TB drive, and during the resync 
(approx 1 hour in), the deadlock up occurred again.  The resync had 
stopped, and any attempt to write to md2 would deadlock the process in 
question.  I think it was doing an rsnaphot backup to a USB drive at the 
time the initial problem occurred - this creates an LVM snapshot device 
on top of md2 for the duration of the backup for each filesystem backed 
up (there are two at the moment), and I suppose this results in lots of 
read-copy-update operations - the mounting of the snapshots shows up in 
the logs as the fs-mounts, and subsequent orphan_cleanups.  As the 
snapshot survives the reboot, I assume this is what triggers the 
subsequent lockup after the machine has rebooted.

I got a couple of 'echo w > /proc/sysrq-trigger' sets of output this 
time...  Edited copies of kern.log are attached - looks like it's 
barrier related.  I'd guess the combination of the LVM CoW snapshot, and 
the RAID resync are tickling this bug.


Any thoughts?  Maybe this is related to Debian bug #584881 - 
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=584881

... since the kernel is essentially the same.

I can do some debugging on this out-of-office-hours, or can probably 
resurrect the original hardware to debug that too.

Logs are here:

http://buttersideup.com/files/md-raid1-lockup-lvm-snapshot/

I think vger binned the first version of this email (with the logs 
attached) - so apologies if you've ended up with two copies of this email...

Tim.


-- 
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53  http://seoss.co.uk/ +44-(0)1273-808309







Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Mon, 17 Jan 2011 13:06:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Mark" <mark-afnpe9pf@mstier.de>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Mon, 17 Jan 2011 13:06:07 GMT) Full text and rfc822 format available.

Message #98 received at 584881@bugs.debian.org (full text, mbox):

From: "Mark" <mark-afnpe9pf@mstier.de>
To: 584881@bugs.debian.org
Subject: Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Mon, 17 Jan 2011 13:56:51 +0100
Related:

https://bugzilla.kernel.org/show_bug.cgi?id=12905






Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Fri, 08 Apr 2011 09:12:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Thomas Aeby <aeby@graeff.com>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Fri, 08 Apr 2011 09:12:09 GMT) Full text and rfc822 format available.

Message #103 received at 584881@bugs.debian.org (full text, mbox):

From: Thomas Aeby <aeby@graeff.com>
To: 584881@bugs.debian.org
Subject: Also in backported squeeze kernel 2.6.32-bpo.5-xen-amd64
Date: Fri, 08 Apr 2011 11:02:01 +0200
Potentially the same effect with

  Kernel 2.6.32-bpo.5-xen-amd64
  DRBD on top of LVM on top of a RAID1-Volume
  2 SATA-Disks
  Fujitsu Hardware
  one machine with 4, the other with 8 cores
  running XEN

This makes me fear that upgrading to Squeeze would not solve
the problem.

Removing one of the mirrors, thus radically avoiding check/resync
worked around (or at least soothed) the problem.

Interestingly enough, the hung has never occured on the respective
DRBD peers, only on the machines where the heavy loaded VM is
actually running on.

Best regards,
Tom




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Sun, 07 Aug 2011 06:48:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Marcus Furlong <furlongm@vpac.org>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Sun, 07 Aug 2011 06:48:06 GMT) Full text and rfc822 format available.

Message #108 received at 584881@bugs.debian.org (full text, mbox):

From: Marcus Furlong <furlongm@vpac.org>
To: 584881@bugs.debian.org
Date: Sun, 7 Aug 2011 16:26:13 +1000 (EST)
We seem to be hitting a similar problem on 2 different machines. When the monthly checkarray script is run, the resync blocks on the raid partition that holds the LVMs. 

Both machines have three disks, running raid10 across partitions. As /proc/mdstat shows, it has hit 0K/sec. Smartctl reports no errors for any of the drives. The machines are running squeeze with all packages up to date. /dev/md2 holds an LVM partition that is used for Xen disks. Load average is above 100. The machines run ganeti, with xen-pvm, xen-hvm as the hypervisors and there is some drbd mirroring between these machines, for some of the logical volumes. lvdisplay, vgdisplay, pvdisplay all hang when run. Most of the VMs show very high load as well (through ganglia and snmp reporting) but most are not accessible via ssh or xm console.

Is there any other information I can provide to help to debug this?

# cat /proc/mdstat
Personalities : [raid10]
md2 : active raid10 sda3[0] sdc3[2] sdb3[1]
      1448908608 blocks super 1.2 64K chunks 2 near-copies [3/3] [UUU]
      [====>................]  check = 23.1% (335167872/1448908608) finish=156694820.0min speed=0K/sec

md0 : active raid10 sda1[0] sdc1[2] sdb1[1]
      14644736 blocks super 1.2 512K chunks 2 near-copies [3/3] [UUU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>

 # cat /proc/`pidof mdadm`/status
Name:   mdadm
State:  S (sleeping)
Tgid:   3298
Pid:    3298
PPid:   1
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 64
Groups:
VmPeak:    12832 kB
VmSize:    12768 kB
VmLck:         0 kB
VmHWM:       768 kB
VmRSS:       604 kB
VmData:      364 kB
VmStk:        88 kB
VmExe:       316 kB
VmLib:      1692 kB
VmPTE:        48 kB
Threads:        1
SigQ:   9/7244
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000002
SigCgt: 0000000000000000
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: ffffffffffffffff
CapBnd: ffffffffffffffff
Cpus_allowed:   1
Cpus_allowed_list:      0
Mems_allowed:   00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        5739
nonvoluntary_ctxt_switches:     12

[5391047.833632] md: data-check of RAID array md0
[5391047.833636] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[5391047.833639] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[5391047.833644] md: using 128k window, over a total of 14644736 blocks.
[5391048.277677] md: delaying data-check of md2 until md0 has finished (they share one or more physical units)
[5391235.279026] md: md0: data-check done.
[5391235.496633] md: data-check of RAID array md2
[5391235.496638] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[5391235.496641] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[5391235.496647] md: using 128k window, over a total of 1448908608 blocks.
[5410976.055527] INFO: task kdmflush:1035 blocked for more than 120 seconds.
[5410976.055566] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[5410976.055619] kdmflush      D ffff880002818e08     0  1035      2 0x00000000
[5410976.055625]  ffff88002e1a5530 0000000000000246 0000000000000002 0000000000000010
[5410976.055633]  0000000000000000 ffff880002d81e80 000000000000f9e0 ffff88003e44dfd8
[5410976.055640]  0000000000015780 0000000000015780 ffff88003ec59530 ffff88003ec59828
[5410976.055647] Call Trace:
[5410976.055657]  [<ffffffff8100ece2>] ? check_events+0x12/0x20
[5410976.055665]  [<ffffffff811804bb>] ? generic_unplug_device+0x0/0x34
[5410976.055680]  [<ffffffffa020e6f0>] ? wait_barrier+0x9a/0xd7 [raid10]
[5410976.055685]  [<ffffffff8104b430>] ? default_wake_function+0x0/0x9
[5410976.055691]  [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[5410976.055698]  [<ffffffffa0210fa2>] ? make_request+0x16f/0x5cd [raid10]
[5410976.055703]  [<ffffffff8100eccf>] ? xen_restore_fl_direct_end+0x0/0x1
[5410976.055709]  [<ffffffff810e81c5>] ? kmem_cache_alloc+0x8c/0xf0
[5410976.055717]  [<ffffffffa01f5b9a>] ? md_make_request+0xb6/0xf1 [md_mod]
[5410976.055723]  [<ffffffff8100eccf>] ? xen_restore_fl_direct_end+0x0/0x1
[5410976.055728]  [<ffffffff8117f6b7>] ? generic_make_request+0x299/0x2f9
[5410976.055737]  [<ffffffffa021a308>] ? clone_bio+0x44/0xce [dm_mod]
[5410976.055745]  [<ffffffffa021b5e9>] ? __split_and_process_bio+0x2ac/0x56b [dm_mod]
[5410976.055753]  [<ffffffffa021ba38>] ? dm_wq_work+0x137/0x167 [dm_mod]
[5410976.055760]  [<ffffffff810628d3>] ? worker_thread+0x188/0x21d
[5410976.055768]  [<ffffffffa021b901>] ? dm_wq_work+0x0/0x167 [dm_mod]
[5410976.055773]  [<ffffffff81065f06>] ? autoremove_wake_function+0x0/0x2e
[5410976.055778]  [<ffffffff8106274b>] ? worker_thread+0x0/0x21d
[5410976.055783]  [<ffffffff81065c39>] ? kthread+0x79/0x81
[5410976.055788]  [<ffffffff81012baa>] ? child_rip+0xa/0x20
[5410976.055793]  [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[5410976.055798]  [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[5410976.055803]  [<ffffffff81012ba0>] ? child_rip+0x0/0x20

# free
             total       used       free     shared    buffers     cached
Mem:       1045340    1026512      18828          0      61688     401856
-/+ buffers/cache:     562968     482372
Swap:      3161640      29072    3132568

# mount
/dev/md0 on / type ext4 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
xenfs on /proc/xen type xenfs (rw)
fusectl on /sys/fs/fuse/connections type fusectl (rw)

# fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000600d9

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1        1216     9764864   fd  Linux raid autodetect
/dev/sda2            1216        1347     1053889+  82  Linux swap / Solaris
/dev/sda3            1348      121601   965940255   fd  Linux raid autodetect

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0007482c

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1        1216     9764864   fd  Linux raid autodetect
/dev/sdc2            1216        1347     1053889+  82  Linux swap / Solaris
/dev/sdc3            1348      121601   965940255   fd  Linux raid autodetect

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006adc2

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1216     9764864   fd  Linux raid autodetect
/dev/sdb2            1216        1347     1053889+  82  Linux swap / Solaris
/dev/sdb3            1348      121601   965940255   fd  Linux raid autodetect

# uname -a
Linux barwon 2.6.32-5-xen-amd64 #1 SMP Thu May 19 01:16:47 UTC 2011 x86_64 GNU/Linux

-- 
Marcus Furlong - VPAC Systems Administrator
http://www.vpac.org
+61 3 9925 4574




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Sun, 01 Apr 2012 20:15:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Sun, 01 Apr 2012 20:15:03 GMT) Full text and rfc822 format available.

Message #113 received at 584881@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: Ian Jackson <ijackson@chiark.greenend.org.uk>, Marcus Furlong <furlongm@vpac.org>, Thomas Aeby <aeby@graeff.com>, Tim Small <tim@seoss.co.uk>
Cc: 584881@bugs.debian.org, NeilBrown <neilb@suse.de>
Subject: Re: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Sun, 01 Apr 2012 21:13:25 +0100
[Message part 1 (text/plain, inline)]
There is a change in Linux 3.3, also intended to go into Linux 3.2.14,
which looks like a fix for bug #584881.

I'm attaching a backported version of this bug fix for Debian 6.0
'squeeze', which you may wish to test.  You can build a kernel package
with this patch by following the instructions at
<http://kernel-handbook.alioth.debian.org/ch-common-tasks.html#s-common-official>.

Note that no further bug fixes will be made in Debian 5.0 'lenny'.

Ben.

-- 
Ben Hutchings
I'm always amazed by the number of people who take up solipsism because
they heard someone else explain it. - E*Borg on alt.fan.pratchett
[0001-md-raid1-raid10-avoid-deadlock-during-resync-recover.patch (text/x-patch, attachment)]
[signature.asc (application/pgp-signature, inline)]

Added tag(s) patch. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Sun, 01 Apr 2012 20:15:08 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Mon, 02 Apr 2012 10:33:15 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Mon, 02 Apr 2012 10:33:18 GMT) Full text and rfc822 format available.

Message #120 received at 584881@bugs.debian.org (full text, mbox):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: Marcus Furlong <furlongm@vpac.org>, Thomas Aeby <aeby@graeff.com>, Tim Small <tim@seoss.co.uk>, 584881@bugs.debian.org, NeilBrown <neilb@suse.de>
Subject: Re: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Mon, 2 Apr 2012 11:29:58 +0100
Ben Hutchings writes ("Re: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> There is a change in Linux 3.3, also intended to go into Linux 3.2.14,
> which looks like a fix for bug #584881.

Thanks.  I haven't experienced the bug in production in squeeze.
I will try to set up a test environment to repro and hopefully confirm
the fix.

Ian.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#584881; Package linux-2.6. (Mon, 02 Apr 2012 11:39:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Tim Small <tim@seoss.co.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Mon, 02 Apr 2012 11:39:09 GMT) Full text and rfc822 format available.

Message #125 received at 584881@bugs.debian.org (full text, mbox):

From: Tim Small <tim@seoss.co.uk>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: Ian Jackson <ijackson@chiark.greenend.org.uk>, Marcus Furlong <furlongm@vpac.org>, Thomas Aeby <aeby@graeff.com>, 584881@bugs.debian.org, NeilBrown <neilb@suse.de>
Subject: Re: Lockups under heavy disk IO; md (RAID) resync/check implicated
Date: Mon, 02 Apr 2012 12:09:26 +0100
FWIW,  I was only able to reproduce the problem which I was seeing on
lenny+openvz (running the same workload on lenny+chroot, or
squeeze+openvz didn't trigger it).

The fix you attached does sound like a plausible fix for the issue I was
seeing (having spent a day or two peering at the code and sprinkling
printks all over it, but not ultimately getting anywhere), but I don't
think I still have my test environment archived anywhere.  If I remember
correctly, things always ended up deadlocking with 1 request sitting in
md's queue.

Tim.

-- 
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.  
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53  http://seoss.co.uk/ +44-(0)1273-808309





Added tag(s) pending. Request was from Ben Hutchings <benh@alioth.debian.org> to control@bugs.debian.org. (Tue, 03 Apr 2012 02:57:07 GMT) Full text and rfc822 format available.

Reply sent to Ben Hutchings <ben@decadent.org.uk>:
You have taken responsibility. (Fri, 06 Apr 2012 03:51:04 GMT) Full text and rfc822 format available.

Notification sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
Bug acknowledged by developer. (Fri, 06 Apr 2012 03:51:04 GMT) Full text and rfc822 format available.

Message #132 received at 584881-close@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: 584881-close@bugs.debian.org
Subject: Bug#584881: fixed in linux-2.6 3.2.14-1
Date: Fri, 06 Apr 2012 03:49:28 +0000
Source: linux-2.6
Source-Version: 3.2.14-1

We believe that the bug you reported is fixed in the latest version of
linux-2.6, which is due to be installed in the Debian FTP archive:

linux-2.6_3.2.14-1.diff.gz
  to main/l/linux-2.6/linux-2.6_3.2.14-1.diff.gz
linux-2.6_3.2.14-1.dsc
  to main/l/linux-2.6/linux-2.6_3.2.14-1.dsc
linux-2.6_3.2.14.orig.tar.gz
  to main/l/linux-2.6/linux-2.6_3.2.14.orig.tar.gz
linux-doc-3.2_3.2.14-1_all.deb
  to main/l/linux-2.6/linux-doc-3.2_3.2.14-1_all.deb
linux-manual-3.2_3.2.14-1_all.deb
  to main/l/linux-2.6/linux-manual-3.2_3.2.14-1_all.deb
linux-patch-debian-3.2_3.2.14-1_all.deb
  to main/l/linux-2.6/linux-patch-debian-3.2_3.2.14-1_all.deb
linux-source-3.2_3.2.14-1_all.deb
  to main/l/linux-2.6/linux-source-3.2_3.2.14-1_all.deb
linux-support-3.2.0-2_3.2.14-1_all.deb
  to main/l/linux-2.6/linux-support-3.2.0-2_3.2.14-1_all.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 584881@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Ben Hutchings <ben@decadent.org.uk> (supplier of updated linux-2.6 package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Thu, 05 Apr 2012 05:02:45 +0100
Source: linux-2.6
Binary: linux-source-3.2 linux-doc-3.2 linux-manual-3.2 linux-patch-debian-3.2 linux-support-3.2.0-2 linux-libc-dev linux-headers-3.2.0-2-all linux-headers-3.2.0-2-all-alpha linux-headers-3.2.0-2-common linux-image-3.2.0-2-alpha-generic linux-headers-3.2.0-2-alpha-generic linux-image-3.2.0-2-alpha-smp linux-headers-3.2.0-2-alpha-smp linux-image-3.2.0-2-alpha-legacy linux-headers-3.2.0-2-alpha-legacy linux-headers-3.2.0-2-all-amd64 kernel-image-3.2.0-2-amd64-di nic-modules-3.2.0-2-amd64-di nic-extra-modules-3.2.0-2-amd64-di nic-wireless-modules-3.2.0-2-amd64-di nic-shared-modules-3.2.0-2-amd64-di serial-modules-3.2.0-2-amd64-di usb-serial-modules-3.2.0-2-amd64-di ppp-modules-3.2.0-2-amd64-di pata-modules-3.2.0-2-amd64-di cdrom-core-modules-3.2.0-2-amd64-di firewire-core-modules-3.2.0-2-amd64-di scsi-core-modules-3.2.0-2-amd64-di scsi-modules-3.2.0-2-amd64-di scsi-common-modules-3.2.0-2-amd64-di scsi-extra-modules-3.2.0-2-amd64-di plip-modules-3.2.0-2-amd64-di
 floppy-modules-3.2.0-2-amd64-di loop-modules-3.2.0-2-amd64-di btrfs-modules-3.2.0-2-amd64-di ext2-modules-3.2.0-2-amd64-di ext3-modules-3.2.0-2-amd64-di ext4-modules-3.2.0-2-amd64-di isofs-modules-3.2.0-2-amd64-di jfs-modules-3.2.0-2-amd64-di ntfs-modules-3.2.0-2-amd64-di reiserfs-modules-3.2.0-2-amd64-di xfs-modules-3.2.0-2-amd64-di fat-modules-3.2.0-2-amd64-di ufs-modules-3.2.0-2-amd64-di qnx4-modules-3.2.0-2-amd64-di md-modules-3.2.0-2-amd64-di multipath-modules-3.2.0-2-amd64-di usb-modules-3.2.0-2-amd64-di usb-storage-modules-3.2.0-2-amd64-di pcmcia-storage-modules-3.2.0-2-amd64-di fb-modules-3.2.0-2-amd64-di input-modules-3.2.0-2-amd64-di event-modules-3.2.0-2-amd64-di mouse-modules-3.2.0-2-amd64-di irda-modules-3.2.0-2-amd64-di parport-modules-3.2.0-2-amd64-di nic-pcmcia-modules-3.2.0-2-amd64-di pcmcia-modules-3.2.0-2-amd64-di nic-usb-modules-3.2.0-2-amd64-di sata-modules-3.2.0-2-amd64-di core-modules-3.2.0-2-amd64-di acpi-modules-3.2.0-2-amd64-di
 i2c-modules-3.2.0-2-amd64-di crc-modules-3.2.0-2-amd64-di crypto-modules-3.2.0-2-amd64-di crypto-dm-modules-3.2.0-2-amd64-di efi-modules-3.2.0-2-amd64-di ata-modules-3.2.0-2-amd64-di mmc-core-modules-3.2.0-2-amd64-di mmc-modules-3.2.0-2-amd64-di nbd-modules-3.2.0-2-amd64-di squashfs-modules-3.2.0-2-amd64-di speakup-modules-3.2.0-2-amd64-di virtio-modules-3.2.0-2-amd64-di uinput-modules-3.2.0-2-amd64-di sound-modules-3.2.0-2-amd64-di zlib-modules-3.2.0-2-amd64-di linux-image-3.2.0-2-amd64 linux-headers-3.2.0-2-amd64 linux-image-3.2.0-2-amd64-dbg xen-linux-system-3.2.0-2-amd64 linux-headers-3.2.0-2-common-rt linux-image-3.2.0-2-rt-amd64 linux-headers-3.2.0-2-rt-amd64 linux-image-3.2.0-2-rt-amd64-dbg linux-headers-3.2.0-2-all-armel kernel-image-3.2.0-2-iop32x-di nic-modules-3.2.0-2-iop32x-di nic-shared-modules-3.2.0-2-iop32x-di usb-serial-modules-3.2.0-2-iop32x-di ppp-modules-3.2.0-2-iop32x-di pata-modules-3.2.0-2-iop32x-di cdrom-core-modules-3.2.0-2-iop32x-di
 scsi-core-modules-3.2.0-2-iop32x-di loop-modules-3.2.0-2-iop32x-di btrfs-modules-3.2.0-2-iop32x-di ext2-modules-3.2.0-2-iop32x-di ext3-modules-3.2.0-2-iop32x-di ext4-modules-3.2.0-2-iop32x-di isofs-modules-3.2.0-2-iop32x-di jffs2-modules-3.2.0-2-iop32x-di jfs-modules-3.2.0-2-iop32x-di reiserfs-modules-3.2.0-2-iop32x-di fat-modules-3.2.0-2-iop32x-di md-modules-3.2.0-2-iop32x-di multipath-modules-3.2.0-2-iop32x-di usb-modules-3.2.0-2-iop32x-di usb-storage-modules-3.2.0-2-iop32x-di event-modules-3.2.0-2-iop32x-di nic-usb-modules-3.2.0-2-iop32x-di sata-modules-3.2.0-2-iop32x-di core-modules-3.2.0-2-iop32x-di crc-modules-3.2.0-2-iop32x-di crypto-modules-3.2.0-2-iop32x-di crypto-dm-modules-3.2.0-2-iop32x-di ata-modules-3.2.0-2-iop32x-di nbd-modules-3.2.0-2-iop32x-di squashfs-modules-3.2.0-2-iop32x-di zlib-modules-3.2.0-2-iop32x-di kernel-image-3.2.0-2-kirkwood-di nic-modules-3.2.0-2-kirkwood-di nic-shared-modules-3.2.0-2-kirkwood-di usb-serial-modules-3.2.0-2-kirkwood-di
 ppp-modules-3.2.0-2-kirkwood-di cdrom-core-modules-3.2.0-2-kirkwood-di scsi-core-modules-3.2.0-2-kirkwood-di loop-modules-3.2.0-2-kirkwood-di btrfs-modules-3.2.0-2-kirkwood-di ext2-modules-3.2.0-2-kirkwood-di ext3-modules-3.2.0-2-kirkwood-di ext4-modules-3.2.0-2-kirkwood-di isofs-modules-3.2.0-2-kirkwood-di jfs-modules-3.2.0-2-kirkwood-di reiserfs-modules-3.2.0-2-kirkwood-di fat-modules-3.2.0-2-kirkwood-di minix-modules-3.2.0-2-kirkwood-di md-modules-3.2.0-2-kirkwood-di multipath-modules-3.2.0-2-kirkwood-di usb-modules-3.2.0-2-kirkwood-di usb-storage-modules-3.2.0-2-kirkwood-di fb-modules-3.2.0-2-kirkwood-di input-modules-3.2.0-2-kirkwood-di event-modules-3.2.0-2-kirkwood-di mouse-modules-3.2.0-2-kirkwood-di nic-usb-modules-3.2.0-2-kirkwood-di sata-modules-3.2.0-2-kirkwood-di core-modules-3.2.0-2-kirkwood-di crc-modules-3.2.0-2-kirkwood-di crypto-modules-3.2.0-2-kirkwood-di crypto-dm-modules-3.2.0-2-kirkwood-di mmc-modules-3.2.0-2-kirkwood-di
 nbd-modules-3.2.0-2-kirkwood-di squashfs-modules-3.2.0-2-kirkwood-di uinput-modules-3.2.0-2-kirkwood-di kernel-image-3.2.0-2-orion5x-di nic-modules-3.2.0-2-orion5x-di nic-shared-modules-3.2.0-2-orion5x-di usb-serial-modules-3.2.0-2-orion5x-di ppp-modules-3.2.0-2-orion5x-di cdrom-core-modules-3.2.0-2-orion5x-di scsi-core-modules-3.2.0-2-orion5x-di loop-modules-3.2.0-2-orion5x-di btrfs-modules-3.2.0-2-orion5x-di ext2-modules-3.2.0-2-orion5x-di ext3-modules-3.2.0-2-orion5x-di ext4-modules-3.2.0-2-orion5x-di isofs-modules-3.2.0-2-orion5x-di jffs2-modules-3.2.0-2-orion5x-di jfs-modules-3.2.0-2-orion5x-di reiserfs-modules-3.2.0-2-orion5x-di fat-modules-3.2.0-2-orion5x-di minix-modules-3.2.0-2-orion5x-di md-modules-3.2.0-2-orion5x-di multipath-modules-3.2.0-2-orion5x-di usb-modules-3.2.0-2-orion5x-di usb-storage-modules-3.2.0-2-orion5x-di event-modules-3.2.0-2-orion5x-di nic-usb-modules-3.2.0-2-orion5x-di sata-modules-3.2.0-2-orion5x-di core-modules-3.2.0-2-orion5x-di
 crc-modules-3.2.0-2-orion5x-di crypto-modules-3.2.0-2-orion5x-di crypto-dm-modules-3.2.0-2-orion5x-di nbd-modules-3.2.0-2-orion5x-di squashfs-modules-3.2.0-2-orion5x-di zlib-modules-3.2.0-2-orion5x-di kernel-image-3.2.0-2-versatile-di nic-modules-3.2.0-2-versatile-di nic-shared-modules-3.2.0-2-versatile-di usb-serial-modules-3.2.0-2-versatile-di ppp-modules-3.2.0-2-versatile-di cdrom-core-modules-3.2.0-2-versatile-di scsi-core-modules-3.2.0-2-versatile-di scsi-common-modules-3.2.0-2-versatile-di loop-modules-3.2.0-2-versatile-di btrfs-modules-3.2.0-2-versatile-di ext2-modules-3.2.0-2-versatile-di ext3-modules-3.2.0-2-versatile-di ext4-modules-3.2.0-2-versatile-di isofs-modules-3.2.0-2-versatile-di reiserfs-modules-3.2.0-2-versatile-di fat-modules-3.2.0-2-versatile-di md-modules-3.2.0-2-versatile-di multipath-modules-3.2.0-2-versatile-di usb-modules-3.2.0-2-versatile-di usb-storage-modules-3.2.0-2-versatile-di nic-usb-modules-3.2.0-2-versatile-di
 sata-modules-3.2.0-2-versatile-di core-modules-3.2.0-2-versatile-di crc-modules-3.2.0-2-versatile-di crypto-modules-3.2.0-2-versatile-di crypto-dm-modules-3.2.0-2-versatile-di nbd-modules-3.2.0-2-versatile-di squashfs-modules-3.2.0-2-versatile-di virtio-modules-3.2.0-2-versatile-di zlib-modules-3.2.0-2-versatile-di linux-image-3.2.0-2-iop32x linux-headers-3.2.0-2-iop32x linux-image-3.2.0-2-ixp4xx linux-headers-3.2.0-2-ixp4xx linux-image-3.2.0-2-kirkwood linux-headers-3.2.0-2-kirkwood linux-image-3.2.0-2-mv78xx0 linux-headers-3.2.0-2-mv78xx0 linux-image-3.2.0-2-orion5x linux-headers-3.2.0-2-orion5x linux-image-3.2.0-2-versatile linux-headers-3.2.0-2-versatile linux-headers-3.2.0-2-all-armhf kernel-image-3.2.0-2-mx5-di nic-modules-3.2.0-2-mx5-di nic-shared-modules-3.2.0-2-mx5-di scsi-core-modules-3.2.0-2-mx5-di loop-modules-3.2.0-2-mx5-di btrfs-modules-3.2.0-2-mx5-di ext2-modules-3.2.0-2-mx5-di ext3-modules-3.2.0-2-mx5-di ext4-modules-3.2.0-2-mx5-di
 isofs-modules-3.2.0-2-mx5-di jfs-modules-3.2.0-2-mx5-di reiserfs-modules-3.2.0-2-mx5-di fat-modules-3.2.0-2-mx5-di minix-modules-3.2.0-2-mx5-di md-modules-3.2.0-2-mx5-di multipath-modules-3.2.0-2-mx5-di usb-storage-modules-3.2.0-2-mx5-di input-modules-3.2.0-2-mx5-di nic-usb-modules-3.2.0-2-mx5-di core-modules-3.2.0-2-mx5-di crc-modules-3.2.0-2-mx5-di crypto-modules-3.2.0-2-mx5-di crypto-dm-modules-3.2.0-2-mx5-di mmc-modules-3.2.0-2-mx5-di nbd-modules-3.2.0-2-mx5-di squashfs-modules-3.2.0-2-mx5-di uinput-modules-3.2.0-2-mx5-di linux-image-3.2.0-2-mx5 linux-headers-3.2.0-2-mx5 linux-image-3.2.0-2-omap linux-headers-3.2.0-2-omap linux-headers-3.2.0-2-all-hppa kernel-image-3.2.0-2-parisc-di nic-modules-3.2.0-2-parisc-di ppp-modules-3.2.0-2-parisc-di pata-modules-3.2.0-2-parisc-di cdrom-core-modules-3.2.0-2-parisc-di scsi-core-modules-3.2.0-2-parisc-di scsi-modules-3.2.0-2-parisc-di loop-modules-3.2.0-2-parisc-di ipv6-modules-3.2.0-2-parisc-di
 btrfs-modules-3.2.0-2-parisc-di ext3-modules-3.2.0-2-parisc-di ext4-modules-3.2.0-2-parisc-di fat-modules-3.2.0-2-parisc-di md-modules-3.2.0-2-parisc-di multipath-modules-3.2.0-2-parisc-di usb-modules-3.2.0-2-parisc-di usb-storage-modules-3.2.0-2-parisc-di input-modules-3.2.0-2-parisc-di crypto-modules-3.2.0-2-parisc-di crypto-dm-modules-3.2.0-2-parisc-di nbd-modules-3.2.0-2-parisc-di zlib-modules-3.2.0-2-parisc-di kernel-image-3.2.0-2-parisc64-di nic-modules-3.2.0-2-parisc64-di ppp-modules-3.2.0-2-parisc64-di pata-modules-3.2.0-2-parisc64-di cdrom-core-modules-3.2.0-2-parisc64-di scsi-core-modules-3.2.0-2-parisc64-di scsi-modules-3.2.0-2-parisc64-di loop-modules-3.2.0-2-parisc64-di ipv6-modules-3.2.0-2-parisc64-di btrfs-modules-3.2.0-2-parisc64-di ext3-modules-3.2.0-2-parisc64-di ext4-modules-3.2.0-2-parisc64-di fat-modules-3.2.0-2-parisc64-di md-modules-3.2.0-2-parisc64-di multipath-modules-3.2.0-2-parisc64-di usb-modules-3.2.0-2-parisc64-di
 usb-storage-modules-3.2.0-2-parisc64-di input-modules-3.2.0-2-parisc64-di crypto-modules-3.2.0-2-parisc64-di crypto-dm-modules-3.2.0-2-parisc64-di nbd-modules-3.2.0-2-parisc64-di zlib-modules-3.2.0-2-parisc64-di linux-image-3.2.0-2-parisc linux-headers-3.2.0-2-parisc linux-image-3.2.0-2-parisc-smp linux-headers-3.2.0-2-parisc-smp linux-image-3.2.0-2-parisc64 linux-headers-3.2.0-2-parisc64 linux-image-3.2.0-2-parisc64-smp linux-headers-3.2.0-2-parisc64-smp linux-headers-3.2.0-2-all-i386 kernel-image-3.2.0-2-486-di nic-modules-3.2.0-2-486-di nic-extra-modules-3.2.0-2-486-di nic-wireless-modules-3.2.0-2-486-di nic-shared-modules-3.2.0-2-486-di serial-modules-3.2.0-2-486-di usb-serial-modules-3.2.0-2-486-di ppp-modules-3.2.0-2-486-di pata-modules-3.2.0-2-486-di cdrom-core-modules-3.2.0-2-486-di firewire-core-modules-3.2.0-2-486-di scsi-core-modules-3.2.0-2-486-di scsi-modules-3.2.0-2-486-di scsi-common-modules-3.2.0-2-486-di scsi-extra-modules-3.2.0-2-486-di
 plip-modules-3.2.0-2-486-di floppy-modules-3.2.0-2-486-di loop-modules-3.2.0-2-486-di btrfs-modules-3.2.0-2-486-di ext2-modules-3.2.0-2-486-di ext3-modules-3.2.0-2-486-di ext4-modules-3.2.0-2-486-di isofs-modules-3.2.0-2-486-di jfs-modules-3.2.0-2-486-di ntfs-modules-3.2.0-2-486-di reiserfs-modules-3.2.0-2-486-di xfs-modules-3.2.0-2-486-di fat-modules-3.2.0-2-486-di ufs-modules-3.2.0-2-486-di qnx4-modules-3.2.0-2-486-di md-modules-3.2.0-2-486-di multipath-modules-3.2.0-2-486-di usb-modules-3.2.0-2-486-di usb-storage-modules-3.2.0-2-486-di pcmcia-storage-modules-3.2.0-2-486-di fb-modules-3.2.0-2-486-di input-modules-3.2.0-2-486-di event-modules-3.2.0-2-486-di mouse-modules-3.2.0-2-486-di irda-modules-3.2.0-2-486-di parport-modules-3.2.0-2-486-di nic-pcmcia-modules-3.2.0-2-486-di pcmcia-modules-3.2.0-2-486-di nic-usb-modules-3.2.0-2-486-di sata-modules-3.2.0-2-486-di core-modules-3.2.0-2-486-di acpi-modules-3.2.0-2-486-di i2c-modules-3.2.0-2-486-di
 crc-modules-3.2.0-2-486-di crypto-modules-3.2.0-2-486-di crypto-dm-modules-3.2.0-2-486-di efi-modules-3.2.0-2-486-di ata-modules-3.2.0-2-486-di mmc-core-modules-3.2.0-2-486-di mmc-modules-3.2.0-2-486-di nbd-modules-3.2.0-2-486-di squashfs-modules-3.2.0-2-486-di speakup-modules-3.2.0-2-486-di virtio-modules-3.2.0-2-486-di uinput-modules-3.2.0-2-486-di sound-modules-3.2.0-2-486-di zlib-modules-3.2.0-2-486-di kernel-image-3.2.0-2-686-pae-di nic-modules-3.2.0-2-686-pae-di nic-extra-modules-3.2.0-2-686-pae-di nic-wireless-modules-3.2.0-2-686-pae-di nic-shared-modules-3.2.0-2-686-pae-di serial-modules-3.2.0-2-686-pae-di usb-serial-modules-3.2.0-2-686-pae-di ppp-modules-3.2.0-2-686-pae-di pata-modules-3.2.0-2-686-pae-di cdrom-core-modules-3.2.0-2-686-pae-di firewire-core-modules-3.2.0-2-686-pae-di scsi-core-modules-3.2.0-2-686-pae-di scsi-modules-3.2.0-2-686-pae-di scsi-common-modules-3.2.0-2-686-pae-di scsi-extra-modules-3.2.0-2-686-pae-di plip-modules-3.2.0-2-686-pae-di
 floppy-modules-3.2.0-2-686-pae-di loop-modules-3.2.0-2-686-pae-di btrfs-modules-3.2.0-2-686-pae-di ext2-modules-3.2.0-2-686-pae-di ext3-modules-3.2.0-2-686-pae-di ext4-modules-3.2.0-2-686-pae-di isofs-modules-3.2.0-2-686-pae-di jfs-modules-3.2.0-2-686-pae-di ntfs-modules-3.2.0-2-686-pae-di reiserfs-modules-3.2.0-2-686-pae-di xfs-modules-3.2.0-2-686-pae-di fat-modules-3.2.0-2-686-pae-di ufs-modules-3.2.0-2-686-pae-di qnx4-modules-3.2.0-2-686-pae-di md-modules-3.2.0-2-686-pae-di multipath-modules-3.2.0-2-686-pae-di usb-modules-3.2.0-2-686-pae-di usb-storage-modules-3.2.0-2-686-pae-di pcmcia-storage-modules-3.2.0-2-686-pae-di fb-modules-3.2.0-2-686-pae-di input-modules-3.2.0-2-686-pae-di event-modules-3.2.0-2-686-pae-di mouse-modules-3.2.0-2-686-pae-di irda-modules-3.2.0-2-686-pae-di parport-modules-3.2.0-2-686-pae-di nic-pcmcia-modules-3.2.0-2-686-pae-di pcmcia-modules-3.2.0-2-686-pae-di nic-usb-modules-3.2.0-2-686-pae-di sata-modules-3.2.0-2-686-pae-di
 core-modules-3.2.0-2-686-pae-di acpi-modules-3.2.0-2-686-pae-di i2c-modules-3.2.0-2-686-pae-di crc-modules-3.2.0-2-686-pae-di crypto-modules-3.2.0-2-686-pae-di crypto-dm-modules-3.2.0-2-686-pae-di efi-modules-3.2.0-2-686-pae-di ata-modules-3.2.0-2-686-pae-di mmc-core-modules-3.2.0-2-686-pae-di mmc-modules-3.2.0-2-686-pae-di nbd-modules-3.2.0-2-686-pae-di squashfs-modules-3.2.0-2-686-pae-di speakup-modules-3.2.0-2-686-pae-di virtio-modules-3.2.0-2-686-pae-di uinput-modules-3.2.0-2-686-pae-di sound-modules-3.2.0-2-686-pae-di zlib-modules-3.2.0-2-686-pae-di linux-image-3.2.0-2-486 linux-headers-3.2.0-2-486 linux-image-3.2.0-2-686-pae linux-headers-3.2.0-2-686-pae linux-image-3.2.0-2-686-pae-dbg xen-linux-system-3.2.0-2-686-pae linux-image-3.2.0-2-rt-686-pae linux-headers-3.2.0-2-rt-686-pae linux-image-3.2.0-2-rt-686-pae-dbg linux-headers-3.2.0-2-all-ia64 kernel-image-3.2.0-2-itanium-di nic-modules-3.2.0-2-itanium-di nic-shared-modules-3.2.0-2-itanium-di
 serial-modules-3.2.0-2-itanium-di ppp-modules-3.2.0-2-itanium-di ide-modules-3.2.0-2-itanium-di pata-modules-3.2.0-2-itanium-di ide-core-modules-3.2.0-2-itanium-di cdrom-core-modules-3.2.0-2-itanium-di firewire-core-modules-3.2.0-2-itanium-di scsi-core-modules-3.2.0-2-itanium-di scsi-modules-3.2.0-2-itanium-di plip-modules-3.2.0-2-itanium-di loop-modules-3.2.0-2-itanium-di ipv6-modules-3.2.0-2-itanium-di btrfs-modules-3.2.0-2-itanium-di ext2-modules-3.2.0-2-itanium-di ext3-modules-3.2.0-2-itanium-di ext4-modules-3.2.0-2-itanium-di isofs-modules-3.2.0-2-itanium-di jfs-modules-3.2.0-2-itanium-di ntfs-modules-3.2.0-2-itanium-di reiserfs-modules-3.2.0-2-itanium-di xfs-modules-3.2.0-2-itanium-di fat-modules-3.2.0-2-itanium-di ufs-modules-3.2.0-2-itanium-di md-modules-3.2.0-2-itanium-di multipath-modules-3.2.0-2-itanium-di usb-modules-3.2.0-2-itanium-di usb-storage-modules-3.2.0-2-itanium-di fb-modules-3.2.0-2-itanium-di input-modules-3.2.0-2-itanium-di
 event-modules-3.2.0-2-itanium-di mouse-modules-3.2.0-2-itanium-di irda-modules-3.2.0-2-itanium-di parport-modules-3.2.0-2-itanium-di pcmcia-modules-3.2.0-2-itanium-di nic-usb-modules-3.2.0-2-itanium-di sata-modules-3.2.0-2-itanium-di core-modules-3.2.0-2-itanium-di crc-modules-3.2.0-2-itanium-di crypto-modules-3.2.0-2-itanium-di crypto-dm-modules-3.2.0-2-itanium-di efi-modules-3.2.0-2-itanium-di ata-modules-3.2.0-2-itanium-di nbd-modules-3.2.0-2-itanium-di squashfs-modules-3.2.0-2-itanium-di uinput-modules-3.2.0-2-itanium-di zlib-modules-3.2.0-2-itanium-di sn-modules-3.2.0-2-itanium-di linux-image-3.2.0-2-itanium linux-headers-3.2.0-2-itanium linux-image-3.2.0-2-mckinley linux-headers-3.2.0-2-mckinley linux-headers-3.2.0-2-all-m68k kernel-image-3.2.0-2-amiga-di nic-shared-modules-3.2.0-2-amiga-di ppp-modules-3.2.0-2-amiga-di cdrom-core-modules-3.2.0-2-amiga-di scsi-modules-3.2.0-2-amiga-di btrfs-modules-3.2.0-2-amiga-di isofs-modules-3.2.0-2-amiga-di
 jfs-modules-3.2.0-2-amiga-di reiserfs-modules-3.2.0-2-amiga-di fat-modules-3.2.0-2-amiga-di md-modules-3.2.0-2-amiga-di crypto-modules-3.2.0-2-amiga-di nbd-modules-3.2.0-2-amiga-di squashfs-modules-3.2.0-2-amiga-di zlib-modules-3.2.0-2-amiga-di kernel-image-3.2.0-2-atari-di nic-shared-modules-3.2.0-2-atari-di ppp-modules-3.2.0-2-atari-di cdrom-core-modules-3.2.0-2-atari-di scsi-modules-3.2.0-2-atari-di btrfs-modules-3.2.0-2-atari-di isofs-modules-3.2.0-2-atari-di jfs-modules-3.2.0-2-atari-di reiserfs-modules-3.2.0-2-atari-di fat-modules-3.2.0-2-atari-di md-modules-3.2.0-2-atari-di crypto-modules-3.2.0-2-atari-di nbd-modules-3.2.0-2-atari-di squashfs-modules-3.2.0-2-atari-di zlib-modules-3.2.0-2-atari-di kernel-image-3.2.0-2-mac-di nic-shared-modules-3.2.0-2-mac-di ppp-modules-3.2.0-2-mac-di cdrom-core-modules-3.2.0-2-mac-di scsi-modules-3.2.0-2-mac-di btrfs-modules-3.2.0-2-mac-di isofs-modules-3.2.0-2-mac-di jfs-modules-3.2.0-2-mac-di reiserfs-modules-3.2.0-2-mac-di
 fat-modules-3.2.0-2-mac-di md-modules-3.2.0-2-mac-di crypto-modules-3.2.0-2-mac-di nbd-modules-3.2.0-2-mac-di squashfs-modules-3.2.0-2-mac-di zlib-modules-3.2.0-2-mac-di linux-image-3.2.0-2-amiga linux-headers-3.2.0-2-amiga linux-image-3.2.0-2-atari linux-headers-3.2.0-2-atari linux-image-3.2.0-2-bvme6000 linux-headers-3.2.0-2-bvme6000 linux-image-3.2.0-2-mac linux-headers-3.2.0-2-mac linux-image-3.2.0-2-mvme147 linux-headers-3.2.0-2-mvme147 linux-image-3.2.0-2-mvme16x linux-headers-3.2.0-2-mvme16x linux-headers-3.2.0-2-all-mips kernel-image-3.2.0-2-sb1-bcm91250a-di nic-shared-modules-3.2.0-2-sb1-bcm91250a-di ppp-modules-3.2.0-2-sb1-bcm91250a-di cdrom-core-modules-3.2.0-2-sb1-bcm91250a-di scsi-core-modules-3.2.0-2-sb1-bcm91250a-di scsi-modules-3.2.0-2-sb1-bcm91250a-di scsi-common-modules-3.2.0-2-sb1-bcm91250a-di loop-modules-3.2.0-2-sb1-bcm91250a-di ipv6-modules-3.2.0-2-sb1-bcm91250a-di btrfs-modules-3.2.0-2-sb1-bcm91250a-di isofs-modules-3.2.0-2-sb1-bcm91250a-di
 jfs-modules-3.2.0-2-sb1-bcm91250a-di reiserfs-modules-3.2.0-2-sb1-bcm91250a-di xfs-modules-3.2.0-2-sb1-bcm91250a-di fat-modules-3.2.0-2-sb1-bcm91250a-di md-modules-3.2.0-2-sb1-bcm91250a-di multipath-modules-3.2.0-2-sb1-bcm91250a-di usb-modules-3.2.0-2-sb1-bcm91250a-di usb-storage-modules-3.2.0-2-sb1-bcm91250a-di fb-modules-3.2.0-2-sb1-bcm91250a-di input-modules-3.2.0-2-sb1-bcm91250a-di sata-modules-3.2.0-2-sb1-bcm91250a-di crypto-modules-3.2.0-2-sb1-bcm91250a-di crypto-dm-modules-3.2.0-2-sb1-bcm91250a-di nbd-modules-3.2.0-2-sb1-bcm91250a-di squashfs-modules-3.2.0-2-sb1-bcm91250a-di rtc-modules-3.2.0-2-sb1-bcm91250a-di zlib-modules-3.2.0-2-sb1-bcm91250a-di kernel-image-3.2.0-2-r4k-ip22-di nic-shared-modules-3.2.0-2-r4k-ip22-di loop-modules-3.2.0-2-r4k-ip22-di ipv6-modules-3.2.0-2-r4k-ip22-di btrfs-modules-3.2.0-2-r4k-ip22-di isofs-modules-3.2.0-2-r4k-ip22-di jfs-modules-3.2.0-2-r4k-ip22-di reiserfs-modules-3.2.0-2-r4k-ip22-di xfs-modules-3.2.0-2-r4k-ip22-di
 md-modules-3.2.0-2-r4k-ip22-di multipath-modules-3.2.0-2-r4k-ip22-di crypto-modules-3.2.0-2-r4k-ip22-di crypto-dm-modules-3.2.0-2-r4k-ip22-di nbd-modules-3.2.0-2-r4k-ip22-di squashfs-modules-3.2.0-2-r4k-ip22-di zlib-modules-3.2.0-2-r4k-ip22-di kernel-image-3.2.0-2-r5k-ip32-di nic-shared-modules-3.2.0-2-r5k-ip32-di loop-modules-3.2.0-2-r5k-ip32-di ipv6-modules-3.2.0-2-r5k-ip32-di btrfs-modules-3.2.0-2-r5k-ip32-di isofs-modules-3.2.0-2-r5k-ip32-di jfs-modules-3.2.0-2-r5k-ip32-di reiserfs-modules-3.2.0-2-r5k-ip32-di xfs-modules-3.2.0-2-r5k-ip32-di md-modules-3.2.0-2-r5k-ip32-di multipath-modules-3.2.0-2-r5k-ip32-di crypto-modules-3.2.0-2-r5k-ip32-di crypto-dm-modules-3.2.0-2-r5k-ip32-di nbd-modules-3.2.0-2-r5k-ip32-di squashfs-modules-3.2.0-2-r5k-ip32-di zlib-modules-3.2.0-2-r5k-ip32-di kernel-image-3.2.0-2-4kc-malta-di ppp-modules-3.2.0-2-4kc-malta-di cdrom-core-modules-3.2.0-2-4kc-malta-di loop-modules-3.2.0-2-4kc-malta-di ipv6-modules-3.2.0-2-4kc-malta-di
 btrfs-modules-3.2.0-2-4kc-malta-di isofs-modules-3.2.0-2-4kc-malta-di jfs-modules-3.2.0-2-4kc-malta-di reiserfs-modules-3.2.0-2-4kc-malta-di xfs-modules-3.2.0-2-4kc-malta-di fat-modules-3.2.0-2-4kc-malta-di md-modules-3.2.0-2-4kc-malta-di multipath-modules-3.2.0-2-4kc-malta-di usb-modules-3.2.0-2-4kc-malta-di usb-storage-modules-3.2.0-2-4kc-malta-di input-modules-3.2.0-2-4kc-malta-di sata-modules-3.2.0-2-4kc-malta-di crypto-modules-3.2.0-2-4kc-malta-di crypto-dm-modules-3.2.0-2-4kc-malta-di nbd-modules-3.2.0-2-4kc-malta-di squashfs-modules-3.2.0-2-4kc-malta-di virtio-modules-3.2.0-2-4kc-malta-di zlib-modules-3.2.0-2-4kc-malta-di linux-image-3.2.0-2-r4k-ip22 linux-headers-3.2.0-2-r4k-ip22 linux-image-3.2.0-2-r5k-ip32 linux-headers-3.2.0-2-r5k-ip32 linux-image-3.2.0-2-sb1-bcm91250a linux-headers-3.2.0-2-sb1-bcm91250a linux-image-3.2.0-2-sb1a-bcm91480b linux-headers-3.2.0-2-sb1a-bcm91480b linux-image-3.2.0-2-4kc-malta linux-headers-3.2.0-2-4kc-malta
 linux-image-3.2.0-2-5kc-malta linux-headers-3.2.0-2-5kc-malta linux-image-3.2.0-2-octeon linux-headers-3.2.0-2-octeon linux-headers-3.2.0-2-all-mipsel kernel-image-3.2.0-2-r5k-cobalt-di nic-shared-modules-3.2.0-2-r5k-cobalt-di ppp-modules-3.2.0-2-r5k-cobalt-di loop-modules-3.2.0-2-r5k-cobalt-di ipv6-modules-3.2.0-2-r5k-cobalt-di btrfs-modules-3.2.0-2-r5k-cobalt-di jfs-modules-3.2.0-2-r5k-cobalt-di reiserfs-modules-3.2.0-2-r5k-cobalt-di xfs-modules-3.2.0-2-r5k-cobalt-di fat-modules-3.2.0-2-r5k-cobalt-di nfs-modules-3.2.0-2-r5k-cobalt-di md-modules-3.2.0-2-r5k-cobalt-di multipath-modules-3.2.0-2-r5k-cobalt-di crypto-modules-3.2.0-2-r5k-cobalt-di crypto-dm-modules-3.2.0-2-r5k-cobalt-di nbd-modules-3.2.0-2-r5k-cobalt-di squashfs-modules-3.2.0-2-r5k-cobalt-di zlib-modules-3.2.0-2-r5k-cobalt-di kernel-image-3.2.0-2-loongson-2f-di nic-modules-3.2.0-2-loongson-2f-di nic-shared-modules-3.2.0-2-loongson-2f-di ppp-modules-3.2.0-2-loongson-2f-di
 cdrom-core-modules-3.2.0-2-loongson-2f-di loop-modules-3.2.0-2-loongson-2f-di ipv6-modules-3.2.0-2-loongson-2f-di btrfs-modules-3.2.0-2-loongson-2f-di isofs-modules-3.2.0-2-loongson-2f-di jfs-modules-3.2.0-2-loongson-2f-di reiserfs-modules-3.2.0-2-loongson-2f-di xfs-modules-3.2.0-2-loongson-2f-di fat-modules-3.2.0-2-loongson-2f-di md-modules-3.2.0-2-loongson-2f-di multipath-modules-3.2.0-2-loongson-2f-di usb-modules-3.2.0-2-loongson-2f-di usb-storage-modules-3.2.0-2-loongson-2f-di input-modules-3.2.0-2-loongson-2f-di nic-usb-modules-3.2.0-2-loongson-2f-di sata-modules-3.2.0-2-loongson-2f-di crypto-modules-3.2.0-2-loongson-2f-di crypto-dm-modules-3.2.0-2-loongson-2f-di nbd-modules-3.2.0-2-loongson-2f-di squashfs-modules-3.2.0-2-loongson-2f-di virtio-modules-3.2.0-2-loongson-2f-di zlib-modules-3.2.0-2-loongson-2f-di linux-image-3.2.0-2-r5k-cobalt linux-headers-3.2.0-2-r5k-cobalt linux-image-3.2.0-2-loongson-2f linux-headers-3.2.0-2-loongson-2f
 linux-headers-3.2.0-2-all-powerpc kernel-image-3.2.0-2-powerpc-di nic-modules-3.2.0-2-powerpc-di nic-extra-modules-3.2.0-2-powerpc-di nic-shared-modules-3.2.0-2-powerpc-di serial-modules-3.2.0-2-powerpc-di usb-serial-modules-3.2.0-2-powerpc-di ppp-modules-3.2.0-2-powerpc-di pata-modules-3.2.0-2-powerpc-di cdrom-core-modules-3.2.0-2-powerpc-di firewire-core-modules-3.2.0-2-powerpc-di scsi-core-modules-3.2.0-2-powerpc-di scsi-modules-3.2.0-2-powerpc-di scsi-common-modules-3.2.0-2-powerpc-di scsi-extra-modules-3.2.0-2-powerpc-di floppy-modules-3.2.0-2-powerpc-di loop-modules-3.2.0-2-powerpc-di btrfs-modules-3.2.0-2-powerpc-di ext2-modules-3.2.0-2-powerpc-di ext3-modules-3.2.0-2-powerpc-di ext4-modules-3.2.0-2-powerpc-di isofs-modules-3.2.0-2-powerpc-di jfs-modules-3.2.0-2-powerpc-di reiserfs-modules-3.2.0-2-powerpc-di xfs-modules-3.2.0-2-powerpc-di fat-modules-3.2.0-2-powerpc-di hfs-modules-3.2.0-2-powerpc-di affs-modules-3.2.0-2-powerpc-di
 ufs-modules-3.2.0-2-powerpc-di md-modules-3.2.0-2-powerpc-di multipath-modules-3.2.0-2-powerpc-di usb-modules-3.2.0-2-powerpc-di usb-storage-modules-3.2.0-2-powerpc-di pcmcia-storage-modules-3.2.0-2-powerpc-di input-modules-3.2.0-2-powerpc-di event-modules-3.2.0-2-powerpc-di mouse-modules-3.2.0-2-powerpc-di irda-modules-3.2.0-2-powerpc-di nic-pcmcia-modules-3.2.0-2-powerpc-di pcmcia-modules-3.2.0-2-powerpc-di sata-modules-3.2.0-2-powerpc-di core-modules-3.2.0-2-powerpc-di crc-modules-3.2.0-2-powerpc-di crypto-modules-3.2.0-2-powerpc-di crypto-dm-modules-3.2.0-2-powerpc-di ata-modules-3.2.0-2-powerpc-di nbd-modules-3.2.0-2-powerpc-di squashfs-modules-3.2.0-2-powerpc-di virtio-modules-3.2.0-2-powerpc-di uinput-modules-3.2.0-2-powerpc-di zlib-modules-3.2.0-2-powerpc-di kernel-image-3.2.0-2-powerpc64-di nic-modules-3.2.0-2-powerpc64-di nic-extra-modules-3.2.0-2-powerpc64-di nic-shared-modules-3.2.0-2-powerpc64-di serial-modules-3.2.0-2-powerpc64-di
 usb-serial-modules-3.2.0-2-powerpc64-di ppp-modules-3.2.0-2-powerpc64-di pata-modules-3.2.0-2-powerpc64-di cdrom-core-modules-3.2.0-2-powerpc64-di firewire-core-modules-3.2.0-2-powerpc64-di scsi-core-modules-3.2.0-2-powerpc64-di scsi-modules-3.2.0-2-powerpc64-di scsi-common-modules-3.2.0-2-powerpc64-di scsi-extra-modules-3.2.0-2-powerpc64-di floppy-modules-3.2.0-2-powerpc64-di loop-modules-3.2.0-2-powerpc64-di btrfs-modules-3.2.0-2-powerpc64-di ext2-modules-3.2.0-2-powerpc64-di ext3-modules-3.2.0-2-powerpc64-di ext4-modules-3.2.0-2-powerpc64-di isofs-modules-3.2.0-2-powerpc64-di jfs-modules-3.2.0-2-powerpc64-di reiserfs-modules-3.2.0-2-powerpc64-di xfs-modules-3.2.0-2-powerpc64-di fat-modules-3.2.0-2-powerpc64-di hfs-modules-3.2.0-2-powerpc64-di affs-modules-3.2.0-2-powerpc64-di ufs-modules-3.2.0-2-powerpc64-di md-modules-3.2.0-2-powerpc64-di multipath-modules-3.2.0-2-powerpc64-di usb-modules-3.2.0-2-powerpc64-di usb-storage-modules-3.2.0-2-powerpc64-di
 pcmcia-storage-modules-3.2.0-2-powerpc64-di input-modules-3.2.0-2-powerpc64-di event-modules-3.2.0-2-powerpc64-di mouse-modules-3.2.0-2-powerpc64-di irda-modules-3.2.0-2-powerpc64-di nic-pcmcia-modules-3.2.0-2-powerpc64-di pcmcia-modules-3.2.0-2-powerpc64-di sata-modules-3.2.0-2-powerpc64-di core-modules-3.2.0-2-powerpc64-di crc-modules-3.2.0-2-powerpc64-di crypto-modules-3.2.0-2-powerpc64-di crypto-dm-modules-3.2.0-2-powerpc64-di ata-modules-3.2.0-2-powerpc64-di nbd-modules-3.2.0-2-powerpc64-di squashfs-modules-3.2.0-2-powerpc64-di virtio-modules-3.2.0-2-powerpc64-di uinput-modules-3.2.0-2-powerpc64-di hypervisor-modules-3.2.0-2-powerpc64-di fancontrol-modules-3.2.0-2-powerpc64-di linux-image-3.2.0-2-powerpc linux-headers-3.2.0-2-powerpc linux-image-3.2.0-2-powerpc-smp linux-headers-3.2.0-2-powerpc-smp linux-image-3.2.0-2-powerpc64 linux-headers-3.2.0-2-powerpc64 linux-headers-3.2.0-2-all-ppc64 linux-headers-3.2.0-2-all-s390 kernel-image-3.2.0-2-s390x-di
 nic-modules-3.2.0-2-s390x-di scsi-core-modules-3.2.0-2-s390x-di scsi-modules-3.2.0-2-s390x-di ext2-modules-3.2.0-2-s390x-di ext3-modules-3.2.0-2-s390x-di ext4-modules-3.2.0-2-s390x-di xfs-modules-3.2.0-2-s390x-di fat-modules-3.2.0-2-s390x-di md-modules-3.2.0-2-s390x-di multipath-modules-3.2.0-2-s390x-di core-modules-3.2.0-2-s390x-di crypto-modules-3.2.0-2-s390x-di crypto-dm-modules-3.2.0-2-s390x-di nbd-modules-3.2.0-2-s390x-di virtio-modules-3.2.0-2-s390x-di dasd-modules-3.2.0-2-s390x-di dasd-extra-modules-3.2.0-2-s390x-di kernel-image-3.2.0-2-s390x-tape-di linux-image-3.2.0-2-s390x linux-headers-3.2.0-2-s390x linux-image-3.2.0-2-s390x-tape linux-headers-3.2.0-2-all-s390x linux-headers-3.2.0-2-all-sh4 kernel-image-3.2.0-2-sh7751r-di nic-modules-3.2.0-2-sh7751r-di usb-serial-modules-3.2.0-2-sh7751r-di ppp-modules-3.2.0-2-sh7751r-di pata-modules-3.2.0-2-sh7751r-di cdrom-core-modules-3.2.0-2-sh7751r-di firewire-core-modules-3.2.0-2-sh7751r-di
 scsi-core-modules-3.2.0-2-sh7751r-di loop-modules-3.2.0-2-sh7751r-di btrfs-modules-3.2.0-2-sh7751r-di ext2-modules-3.2.0-2-sh7751r-di ext4-modules-3.2.0-2-sh7751r-di isofs-modules-3.2.0-2-sh7751r-di jfs-modules-3.2.0-2-sh7751r-di reiserfs-modules-3.2.0-2-sh7751r-di xfs-modules-3.2.0-2-sh7751r-di fat-modules-3.2.0-2-sh7751r-di ufs-modules-3.2.0-2-sh7751r-di minix-modules-3.2.0-2-sh7751r-di md-modules-3.2.0-2-sh7751r-di multipath-modules-3.2.0-2-sh7751r-di usb-storage-modules-3.2.0-2-sh7751r-di parport-modules-3.2.0-2-sh7751r-di nic-usb-modules-3.2.0-2-sh7751r-di sata-modules-3.2.0-2-sh7751r-di i2c-modules-3.2.0-2-sh7751r-di crc-modules-3.2.0-2-sh7751r-di crypto-modules-3.2.0-2-sh7751r-di crypto-dm-modules-3.2.0-2-sh7751r-di nbd-modules-3.2.0-2-sh7751r-di squashfs-modules-3.2.0-2-sh7751r-di speakup-modules-3.2.0-2-sh7751r-di virtio-modules-3.2.0-2-sh7751r-di sound-modules-3.2.0-2-sh7751r-di zlib-modules-3.2.0-2-sh7751r-di kernel-image-3.2.0-2-sh7785lcr-di
 nic-modules-3.2.0-2-sh7785lcr-di usb-serial-modules-3.2.0-2-sh7785lcr-di ppp-modules-3.2.0-2-sh7785lcr-di pata-modules-3.2.0-2-sh7785lcr-di cdrom-core-modules-3.2.0-2-sh7785lcr-di firewire-core-modules-3.2.0-2-sh7785lcr-di scsi-core-modules-3.2.0-2-sh7785lcr-di loop-modules-3.2.0-2-sh7785lcr-di btrfs-modules-3.2.0-2-sh7785lcr-di ext2-modules-3.2.0-2-sh7785lcr-di ext4-modules-3.2.0-2-sh7785lcr-di isofs-modules-3.2.0-2-sh7785lcr-di jfs-modules-3.2.0-2-sh7785lcr-di reiserfs-modules-3.2.0-2-sh7785lcr-di xfs-modules-3.2.0-2-sh7785lcr-di fat-modules-3.2.0-2-sh7785lcr-di ufs-modules-3.2.0-2-sh7785lcr-di minix-modules-3.2.0-2-sh7785lcr-di md-modules-3.2.0-2-sh7785lcr-di multipath-modules-3.2.0-2-sh7785lcr-di parport-modules-3.2.0-2-sh7785lcr-di nic-usb-modules-3.2.0-2-sh7785lcr-di sata-modules-3.2.0-2-sh7785lcr-di crc-modules-3.2.0-2-sh7785lcr-di crypto-modules-3.2.0-2-sh7785lcr-di crypto-dm-modules-3.2.0-2-sh7785lcr-di nbd-modules-3.2.0-2-sh7785lcr-di
 squashfs-modules-3.2.0-2-sh7785lcr-di speakup-modules-3.2.0-2-sh7785lcr-di sound-modules-3.2.0-2-sh7785lcr-di zlib-modules-3.2.0-2-sh7785lcr-di linux-image-3.2.0-2-sh7751r linux-headers-3.2.0-2-sh7751r linux-image-3.2.0-2-sh7785lcr linux-headers-3.2.0-2-sh7785lcr linux-headers-3.2.0-2-all-sparc kernel-image-3.2.0-2-sparc64-di nic-modules-3.2.0-2-sparc64-di ppp-modules-3.2.0-2-sparc64-di pata-modules-3.2.0-2-sparc64-di cdrom-core-modules-3.2.0-2-sparc64-di scsi-core-modules-3.2.0-2-sparc64-di scsi-modules-3.2.0-2-sparc64-di scsi-common-modules-3.2.0-2-sparc64-di plip-modules-3.2.0-2-sparc64-di btrfs-modules-3.2.0-2-sparc64-di ext2-modules-3.2.0-2-sparc64-di ext3-modules-3.2.0-2-sparc64-di ext4-modules-3.2.0-2-sparc64-di isofs-modules-3.2.0-2-sparc64-di jfs-modules-3.2.0-2-sparc64-di reiserfs-modules-3.2.0-2-sparc64-di xfs-modules-3.2.0-2-sparc64-di fat-modules-3.2.0-2-sparc64-di md-modules-3.2.0-2-sparc64-di multipath-modules-3.2.0-2-sparc64-di
 usb-modules-3.2.0-2-sparc64-di usb-storage-modules-3.2.0-2-sparc64-di input-modules-3.2.0-2-sparc64-di sata-modules-3.2.0-2-sparc64-di core-modules-3.2.0-2-sparc64-di crypto-modules-3.2.0-2-sparc64-di crypto-dm-modules-3.2.0-2-sparc64-di ata-modules-3.2.0-2-sparc64-di nbd-modules-3.2.0-2-sparc64-di squashfs-modules-3.2.0-2-sparc64-di zlib-modules-3.2.0-2-sparc64-di linux-image-3.2.0-2-sparc64 linux-headers-3.2.0-2-sparc64 linux-image-3.2.0-2-sparc64-smp linux-headers-3.2.0-2-sparc64-smp
 linux-headers-3.2.0-2-all-sparc64
Architecture: all source
Version: 3.2.14-1
Distribution: unstable
Urgency: low
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Changed-By: Ben Hutchings <ben@decadent.org.uk>
Closes: 584881 645547 661558 665420 666121 666506
Description: 
 acpi-modules-3.2.0-2-486-di - ACPI support modules (udeb)
 acpi-modules-3.2.0-2-686-pae-di - ACPI support modules (udeb)
 acpi-modules-3.2.0-2-amd64-di - ACPI support modules (udeb)
 affs-modules-3.2.0-2-powerpc64-di - Amiga filesystem support (udeb)
 affs-modules-3.2.0-2-powerpc-di - Amiga filesystem support (udeb)
 ata-modules-3.2.0-2-486-di - ATA disk modules (udeb)
 ata-modules-3.2.0-2-686-pae-di - ATA disk modules (udeb)
 ata-modules-3.2.0-2-amd64-di - ATA disk modules (udeb)
 ata-modules-3.2.0-2-iop32x-di - ATA disk modules (udeb)
 ata-modules-3.2.0-2-itanium-di - ATA disk modules (udeb)
 ata-modules-3.2.0-2-powerpc64-di - ATA disk modules (udeb)
 ata-modules-3.2.0-2-powerpc-di - ATA disk modules (udeb)
 ata-modules-3.2.0-2-sparc64-di - ATA disk modules (udeb)
 btrfs-modules-3.2.0-2-486-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-4kc-malta-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-686-pae-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-amd64-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-amiga-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-atari-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-iop32x-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-itanium-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-kirkwood-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-loongson-2f-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-mac-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-mx5-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-orion5x-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-parisc64-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-parisc-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-powerpc64-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-powerpc-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-r4k-ip22-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-r5k-cobalt-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-r5k-ip32-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-sb1-bcm91250a-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-sh7751r-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-sh7785lcr-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-sparc64-di - BTRFS filesystem support (udeb)
 btrfs-modules-3.2.0-2-versatile-di - BTRFS filesystem support (udeb)
 cdrom-core-modules-3.2.0-2-486-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-4kc-malta-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-686-pae-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-amd64-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-amiga-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-atari-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-iop32x-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-itanium-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-kirkwood-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-loongson-2f-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-mac-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-orion5x-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-parisc64-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-parisc-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-powerpc64-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-powerpc-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-sb1-bcm91250a-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-sh7751r-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-sh7785lcr-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-sparc64-di - CDROM support (udeb)
 cdrom-core-modules-3.2.0-2-versatile-di - CDROM support (udeb)
 core-modules-3.2.0-2-486-di - Core modules (udeb)
 core-modules-3.2.0-2-686-pae-di - Core modules (udeb)
 core-modules-3.2.0-2-amd64-di - Core modules (udeb)
 core-modules-3.2.0-2-iop32x-di - Core modules (udeb)
 core-modules-3.2.0-2-itanium-di - Core modules (udeb)
 core-modules-3.2.0-2-kirkwood-di - Core modules (udeb)
 core-modules-3.2.0-2-mx5-di - Core modules (udeb)
 core-modules-3.2.0-2-orion5x-di - Core modules (udeb)
 core-modules-3.2.0-2-powerpc64-di - Core modules (udeb)
 core-modules-3.2.0-2-powerpc-di - Core modules (udeb)
 core-modules-3.2.0-2-s390x-di - Core modules (udeb)
 core-modules-3.2.0-2-sparc64-di - Core modules (udeb)
 core-modules-3.2.0-2-versatile-di - Core modules (udeb)
 crc-modules-3.2.0-2-486-di - CRC modules (udeb)
 crc-modules-3.2.0-2-686-pae-di - CRC modules (udeb)
 crc-modules-3.2.0-2-amd64-di - CRC modules (udeb)
 crc-modules-3.2.0-2-iop32x-di - CRC modules (udeb)
 crc-modules-3.2.0-2-itanium-di - CRC modules (udeb)
 crc-modules-3.2.0-2-kirkwood-di - CRC modules (udeb)
 crc-modules-3.2.0-2-mx5-di - CRC modules (udeb)
 crc-modules-3.2.0-2-orion5x-di - CRC modules (udeb)
 crc-modules-3.2.0-2-powerpc64-di - CRC modules (udeb)
 crc-modules-3.2.0-2-powerpc-di - CRC modules (udeb)
 crc-modules-3.2.0-2-sh7751r-di - CRC modules (udeb)
 crc-modules-3.2.0-2-sh7785lcr-di - CRC modules (udeb)
 crc-modules-3.2.0-2-versatile-di - CRC modules (udeb)
 crypto-dm-modules-3.2.0-2-486-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-4kc-malta-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-686-pae-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-amd64-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-iop32x-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-itanium-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-kirkwood-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-loongson-2f-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-mx5-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-orion5x-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-parisc64-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-parisc-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-powerpc64-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-powerpc-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-r4k-ip22-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-r5k-cobalt-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-r5k-ip32-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-s390x-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-sb1-bcm91250a-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-sh7751r-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-sh7785lcr-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-sparc64-di - devicemapper crypto module (udeb)
 crypto-dm-modules-3.2.0-2-versatile-di - devicemapper crypto module (udeb)
 crypto-modules-3.2.0-2-486-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-4kc-malta-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-686-pae-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-amd64-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-amiga-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-atari-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-iop32x-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-itanium-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-kirkwood-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-loongson-2f-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-mac-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-mx5-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-orion5x-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-parisc64-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-parisc-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-powerpc64-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-powerpc-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-r4k-ip22-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-r5k-cobalt-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-r5k-ip32-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-s390x-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-sb1-bcm91250a-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-sh7751r-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-sh7785lcr-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-sparc64-di - crypto modules (udeb)
 crypto-modules-3.2.0-2-versatile-di - crypto modules (udeb)
 dasd-extra-modules-3.2.0-2-s390x-di - optional dasd DIAG support (udeb)
 dasd-modules-3.2.0-2-s390x-di - dasd modules (udeb)
 efi-modules-3.2.0-2-486-di - EFI modules (udeb)
 efi-modules-3.2.0-2-686-pae-di - EFI modules (udeb)
 efi-modules-3.2.0-2-amd64-di - EFI modules (udeb)
 efi-modules-3.2.0-2-itanium-di - EFI modules (udeb)
 event-modules-3.2.0-2-486-di - Event support (udeb)
 event-modules-3.2.0-2-686-pae-di - Event support (udeb)
 event-modules-3.2.0-2-amd64-di - Event support (udeb)
 event-modules-3.2.0-2-iop32x-di - Event support (udeb)
 event-modules-3.2.0-2-itanium-di - Event support (udeb)
 event-modules-3.2.0-2-kirkwood-di - Event support (udeb)
 event-modules-3.2.0-2-orion5x-di - Event support (udeb)
 event-modules-3.2.0-2-powerpc64-di - Event support (udeb)
 event-modules-3.2.0-2-powerpc-di - Event support (udeb)
 ext2-modules-3.2.0-2-486-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-686-pae-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-amd64-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-iop32x-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-itanium-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-kirkwood-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-mx5-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-orion5x-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-powerpc64-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-powerpc-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-s390x-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-sh7751r-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-sh7785lcr-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-sparc64-di - EXT2 filesystem support (udeb)
 ext2-modules-3.2.0-2-versatile-di - EXT2 filesystem support (udeb)
 ext3-modules-3.2.0-2-486-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-686-pae-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-amd64-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-iop32x-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-itanium-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-kirkwood-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-mx5-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-orion5x-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-parisc64-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-parisc-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-powerpc64-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-powerpc-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-s390x-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-sparc64-di - EXT3 filesystem support (udeb)
 ext3-modules-3.2.0-2-versatile-di - EXT3 filesystem support (udeb)
 ext4-modules-3.2.0-2-486-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-686-pae-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-amd64-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-iop32x-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-itanium-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-kirkwood-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-mx5-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-orion5x-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-parisc64-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-parisc-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-powerpc64-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-powerpc-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-s390x-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-sh7751r-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-sh7785lcr-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-sparc64-di - EXT4 filesystem support (udeb)
 ext4-modules-3.2.0-2-versatile-di - EXT4 filesystem support (udeb)
 fancontrol-modules-3.2.0-2-powerpc64-di - Apple powermac fancontrol modules (udeb)
 fat-modules-3.2.0-2-486-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-4kc-malta-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-686-pae-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-amd64-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-amiga-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-atari-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-iop32x-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-itanium-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-kirkwood-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-loongson-2f-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-mac-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-mx5-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-orion5x-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-parisc64-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-parisc-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-powerpc64-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-powerpc-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-r5k-cobalt-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-s390x-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-sb1-bcm91250a-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-sh7751r-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-sh7785lcr-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-sparc64-di - FAT filesystem support (udeb)
 fat-modules-3.2.0-2-versatile-di - FAT filesystem support (udeb)
 fb-modules-3.2.0-2-486-di - Frame buffer support (udeb)
 fb-modules-3.2.0-2-686-pae-di - Frame buffer support (udeb)
 fb-modules-3.2.0-2-amd64-di - Frame buffer support (udeb)
 fb-modules-3.2.0-2-itanium-di - Frame buffer support (udeb)
 fb-modules-3.2.0-2-kirkwood-di - Frame buffer support (udeb)
 fb-modules-3.2.0-2-sb1-bcm91250a-di - Frame buffer support (udeb)
 firewire-core-modules-3.2.0-2-486-di - Core FireWire drivers (udeb)
 firewire-core-modules-3.2.0-2-686-pae-di - Core FireWire drivers (udeb)
 firewire-core-modules-3.2.0-2-amd64-di - Core FireWire drivers (udeb)
 firewire-core-modules-3.2.0-2-itanium-di - Core FireWire drivers (udeb)
 firewire-core-modules-3.2.0-2-powerpc64-di - Core FireWire drivers (udeb)
 firewire-core-modules-3.2.0-2-powerpc-di - Core FireWire drivers (udeb)
 firewire-core-modules-3.2.0-2-sh7751r-di - Core FireWire drivers (udeb)
 firewire-core-modules-3.2.0-2-sh7785lcr-di - Core FireWire drivers (udeb)
 floppy-modules-3.2.0-2-486-di - Floppy driver (udeb)
 floppy-modules-3.2.0-2-686-pae-di - Floppy driver (udeb)
 floppy-modules-3.2.0-2-amd64-di - Floppy driver (udeb)
 floppy-modules-3.2.0-2-powerpc64-di - Floppy driver (udeb)
 floppy-modules-3.2.0-2-powerpc-di - Floppy driver (udeb)
 hfs-modules-3.2.0-2-powerpc64-di - HFS filesystem support (udeb)
 hfs-modules-3.2.0-2-powerpc-di - HFS filesystem support (udeb)
 hypervisor-modules-3.2.0-2-powerpc64-di - IBM 64bit hypervisor console modules (udeb)
 i2c-modules-3.2.0-2-486-di - i2c support modules (udeb)
 i2c-modules-3.2.0-2-686-pae-di - i2c support modules (udeb)
 i2c-modules-3.2.0-2-amd64-di - i2c support modules (udeb)
 i2c-modules-3.2.0-2-sh7751r-di - i2c support modules (udeb)
 ide-core-modules-3.2.0-2-itanium-di - IDE support (udeb)
 ide-modules-3.2.0-2-itanium-di - IDE drivers (udeb)
 input-modules-3.2.0-2-486-di - Input devices support (udeb)
 input-modules-3.2.0-2-4kc-malta-di - Input devices support (udeb)
 input-modules-3.2.0-2-686-pae-di - Input devices support (udeb)
 input-modules-3.2.0-2-amd64-di - Input devices support (udeb)
 input-modules-3.2.0-2-itanium-di - Input devices support (udeb)
 input-modules-3.2.0-2-kirkwood-di - Input devices support (udeb)
 input-modules-3.2.0-2-loongson-2f-di - Input devices support (udeb)
 input-modules-3.2.0-2-mx5-di - Input devices support (udeb)
 input-modules-3.2.0-2-parisc64-di - Input devices support (udeb)
 input-modules-3.2.0-2-parisc-di - Input devices support (udeb)
 input-modules-3.2.0-2-powerpc64-di - Input devices support (udeb)
 input-modules-3.2.0-2-powerpc-di - Input devices support (udeb)
 input-modules-3.2.0-2-sb1-bcm91250a-di - Input devices support (udeb)
 input-modules-3.2.0-2-sparc64-di - Input devices support (udeb)
 ipv6-modules-3.2.0-2-4kc-malta-di - IPv6 driver (udeb)
 ipv6-modules-3.2.0-2-itanium-di - IPv6 driver (udeb)
 ipv6-modules-3.2.0-2-loongson-2f-di - IPv6 driver (udeb)
 ipv6-modules-3.2.0-2-parisc64-di - IPv6 driver (udeb)
 ipv6-modules-3.2.0-2-parisc-di - IPv6 driver (udeb)
 ipv6-modules-3.2.0-2-r4k-ip22-di - IPv6 driver (udeb)
 ipv6-modules-3.2.0-2-r5k-cobalt-di - IPv6 driver (udeb)
 ipv6-modules-3.2.0-2-r5k-ip32-di - IPv6 driver (udeb)
 ipv6-modules-3.2.0-2-sb1-bcm91250a-di - IPv6 driver (udeb)
 irda-modules-3.2.0-2-486-di - Infrared devices support (udeb)
 irda-modules-3.2.0-2-686-pae-di - Infrared devices support (udeb)
 irda-modules-3.2.0-2-amd64-di - Infrared devices support (udeb)
 irda-modules-3.2.0-2-itanium-di - Infrared devices support (udeb)
 irda-modules-3.2.0-2-powerpc64-di - Infrared devices support (udeb)
 irda-modules-3.2.0-2-powerpc-di - Infrared devices support (udeb)
 isofs-modules-3.2.0-2-486-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-4kc-malta-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-686-pae-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-amd64-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-amiga-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-atari-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-iop32x-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-itanium-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-kirkwood-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-loongson-2f-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-mac-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-mx5-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-orion5x-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-powerpc64-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-powerpc-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-r4k-ip22-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-r5k-ip32-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-sb1-bcm91250a-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-sh7751r-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-sh7785lcr-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-sparc64-di - ISOFS filesystem support (udeb)
 isofs-modules-3.2.0-2-versatile-di - ISOFS filesystem support (udeb)
 jffs2-modules-3.2.0-2-iop32x-di - JFFS2 filesystem support (udeb)
 jffs2-modules-3.2.0-2-orion5x-di - JFFS2 filesystem support (udeb)
 jfs-modules-3.2.0-2-486-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-4kc-malta-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-686-pae-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-amd64-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-amiga-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-atari-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-iop32x-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-itanium-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-kirkwood-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-loongson-2f-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-mac-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-mx5-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-orion5x-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-powerpc64-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-powerpc-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-r4k-ip22-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-r5k-cobalt-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-r5k-ip32-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-sb1-bcm91250a-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-sh7751r-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-sh7785lcr-di - JFS filesystem support (udeb)
 jfs-modules-3.2.0-2-sparc64-di - JFS filesystem support (udeb)
 kernel-image-3.2.0-2-486-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-4kc-malta-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-686-pae-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-amd64-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-amiga-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-atari-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-iop32x-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-itanium-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-kirkwood-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-loongson-2f-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-mac-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-mx5-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-orion5x-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-parisc64-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-parisc-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-powerpc64-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-powerpc-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-r4k-ip22-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-r5k-cobalt-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-r5k-ip32-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-s390x-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-s390x-tape-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-sb1-bcm91250a-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-sh7751r-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-sh7785lcr-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-sparc64-di - Linux kernel binary image for the Debian installer (udeb)
 kernel-image-3.2.0-2-versatile-di - Linux kernel binary image for the Debian installer (udeb)
 linux-doc-3.2 - Linux kernel specific documentation for version 3.2
 linux-headers-3.2.0-2-486 - Header files for Linux 3.2.0-2-486
 linux-headers-3.2.0-2-4kc-malta - Header files for Linux 3.2.0-2-4kc-malta
 linux-headers-3.2.0-2-5kc-malta - Header files for Linux 3.2.0-2-5kc-malta
 linux-headers-3.2.0-2-686-pae - Header files for Linux 3.2.0-2-686-pae
 linux-headers-3.2.0-2-all - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-alpha - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-amd64 - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-armel - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-armhf - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-hppa - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-i386 - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-ia64 - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-m68k - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-mips - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-mipsel - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-powerpc - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-ppc64 - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-s390 - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-s390x - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-sh4 - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-sparc64 - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-all-sparc - All header files for Linux 3.2 (meta-package)
 linux-headers-3.2.0-2-alpha-generic - Header files for Linux 3.2.0-2-alpha-generic
 linux-headers-3.2.0-2-alpha-legacy - Header files for Linux 3.2.0-2-alpha-legacy
 linux-headers-3.2.0-2-alpha-smp - Header files for Linux 3.2.0-2-alpha-smp
 linux-headers-3.2.0-2-amd64 - Header files for Linux 3.2.0-2-amd64
 linux-headers-3.2.0-2-amiga - Header files for Linux 3.2.0-2-amiga
 linux-headers-3.2.0-2-atari - Header files for Linux 3.2.0-2-atari
 linux-headers-3.2.0-2-bvme6000 - Header files for Linux 3.2.0-2-bvme6000
 linux-headers-3.2.0-2-common - Common header files for Linux 3.2.0-2
 linux-headers-3.2.0-2-common-rt - Common header files for Linux 3.2.0-2-rt
 linux-headers-3.2.0-2-iop32x - Header files for Linux 3.2.0-2-iop32x
 linux-headers-3.2.0-2-itanium - Header files for Linux 3.2.0-2-itanium
 linux-headers-3.2.0-2-ixp4xx - Header files for Linux 3.2.0-2-ixp4xx
 linux-headers-3.2.0-2-kirkwood - Header files for Linux 3.2.0-2-kirkwood
 linux-headers-3.2.0-2-loongson-2f - Header files for Linux 3.2.0-2-loongson-2f
 linux-headers-3.2.0-2-mac - Header files for Linux 3.2.0-2-mac
 linux-headers-3.2.0-2-mckinley - Header files for Linux 3.2.0-2-mckinley
 linux-headers-3.2.0-2-mv78xx0 - Header files for Linux 3.2.0-2-mv78xx0
 linux-headers-3.2.0-2-mvme147 - Header files for Linux 3.2.0-2-mvme147
 linux-headers-3.2.0-2-mvme16x - Header files for Linux 3.2.0-2-mvme16x
 linux-headers-3.2.0-2-mx5 - Header files for Linux 3.2.0-2-mx5
 linux-headers-3.2.0-2-octeon - Header files for Linux 3.2.0-2-octeon
 linux-headers-3.2.0-2-omap - Header files for Linux 3.2.0-2-omap
 linux-headers-3.2.0-2-orion5x - Header files for Linux 3.2.0-2-orion5x
 linux-headers-3.2.0-2-parisc64 - Header files for Linux 3.2.0-2-parisc64
 linux-headers-3.2.0-2-parisc64-smp - Header files for Linux 3.2.0-2-parisc64-smp
 linux-headers-3.2.0-2-parisc - Header files for Linux 3.2.0-2-parisc
 linux-headers-3.2.0-2-parisc-smp - Header files for Linux 3.2.0-2-parisc-smp
 linux-headers-3.2.0-2-powerpc64 - Header files for Linux 3.2.0-2-powerpc64
 linux-headers-3.2.0-2-powerpc - Header files for Linux 3.2.0-2-powerpc
 linux-headers-3.2.0-2-powerpc-smp - Header files for Linux 3.2.0-2-powerpc-smp
 linux-headers-3.2.0-2-r4k-ip22 - Header files for Linux 3.2.0-2-r4k-ip22
 linux-headers-3.2.0-2-r5k-cobalt - Header files for Linux 3.2.0-2-r5k-cobalt
 linux-headers-3.2.0-2-r5k-ip32 - Header files for Linux 3.2.0-2-r5k-ip32
 linux-headers-3.2.0-2-rt-686-pae - Header files for Linux 3.2.0-2-rt-686-pae
 linux-headers-3.2.0-2-rt-amd64 - Header files for Linux 3.2.0-2-rt-amd64
 linux-headers-3.2.0-2-s390x - Header files for Linux 3.2.0-2-s390x
 linux-headers-3.2.0-2-sb1a-bcm91480b - Header files for Linux 3.2.0-2-sb1a-bcm91480b
 linux-headers-3.2.0-2-sb1-bcm91250a - Header files for Linux 3.2.0-2-sb1-bcm91250a
 linux-headers-3.2.0-2-sh7751r - Header files for Linux 3.2.0-2-sh7751r
 linux-headers-3.2.0-2-sh7785lcr - Header files for Linux 3.2.0-2-sh7785lcr
 linux-headers-3.2.0-2-sparc64 - Header files for Linux 3.2.0-2-sparc64
 linux-headers-3.2.0-2-sparc64-smp - Header files for Linux 3.2.0-2-sparc64-smp
 linux-headers-3.2.0-2-versatile - Header files for Linux 3.2.0-2-versatile
 linux-image-3.2.0-2-486 - Linux 3.2 for older PCs
 linux-image-3.2.0-2-4kc-malta - Linux 3.2 for MIPS Malta
 linux-image-3.2.0-2-5kc-malta - Linux 3.2 for MIPS Malta (64-bit)
 linux-image-3.2.0-2-686-pae-dbg - Debugging infos for Linux 3.2.0-2-686-pae
 linux-image-3.2.0-2-686-pae - Linux 3.2 for modern PCs
 linux-image-3.2.0-2-alpha-generic - Linux 3.2 for Alpha
 linux-image-3.2.0-2-alpha-legacy - Linux 3.2 for Alpha Legacy
 linux-image-3.2.0-2-alpha-smp - Linux 3.2 for Alpha SMP
 linux-image-3.2.0-2-amd64-dbg - Debugging infos for Linux 3.2.0-2-amd64
 linux-image-3.2.0-2-amd64 - Linux 3.2 for 64-bit PCs
 linux-image-3.2.0-2-amiga - Linux 3.2 for Amiga
 linux-image-3.2.0-2-atari - Linux 3.2 for Atari
 linux-image-3.2.0-2-bvme6000 - Linux 3.2 for BVM BVME4000 and BVME6000
 linux-image-3.2.0-2-iop32x - Linux 3.2 for IOP32x
 linux-image-3.2.0-2-itanium - Linux 3.2 for Itanium
 linux-image-3.2.0-2-ixp4xx - Linux 3.2 for IXP4xx
 linux-image-3.2.0-2-kirkwood - Linux 3.2 for Marvell Kirkwood
 linux-image-3.2.0-2-loongson-2f - Linux 3.2 for Loongson 2F
 linux-image-3.2.0-2-mac - Linux 3.2 for Macintosh
 linux-image-3.2.0-2-mckinley - Linux 3.2 for Itanium II
 linux-image-3.2.0-2-mv78xx0 - Linux 3.2 for Marvell 78xx0
 linux-image-3.2.0-2-mvme147 - Linux 3.2 for Motorola MVME147
 linux-image-3.2.0-2-mvme16x - Linux 3.2 for Motorola MVME162/6/7, MVME172/7
 linux-image-3.2.0-2-mx5 - Linux 3.2 for Freescale i.MX51
 linux-image-3.2.0-2-octeon - Linux 3.2 for Octeon
 linux-image-3.2.0-2-omap - Linux 3.2 for TI OMAP3+
 linux-image-3.2.0-2-orion5x - Linux 3.2 for Marvell Orion
 linux-image-3.2.0-2-parisc64 - Linux 3.2 for 64-bit PA-RISC
 linux-image-3.2.0-2-parisc64-smp - Linux 3.2 for multiprocessor 64-bit PA-RISC
 linux-image-3.2.0-2-parisc - Linux 3.2 for 32-bit PA-RISC
 linux-image-3.2.0-2-parisc-smp - Linux 3.2 for multiprocessor 32-bit PA-RISC
 linux-image-3.2.0-2-powerpc64 - Linux 3.2 for 64-bit PowerPC
 linux-image-3.2.0-2-powerpc - Linux 3.2 for uniprocessor 32-bit PowerPC
 linux-image-3.2.0-2-powerpc-smp - Linux 3.2 for multiprocessor 32-bit PowerPC
 linux-image-3.2.0-2-r4k-ip22 - Linux 3.2 for SGI IP22
 linux-image-3.2.0-2-r5k-cobalt - Linux 3.2 for Cobalt
 linux-image-3.2.0-2-r5k-ip32 - Linux 3.2 for SGI IP32
 linux-image-3.2.0-2-rt-686-pae-dbg - Debugging infos for Linux 3.2.0-2-rt-686-pae
 linux-image-3.2.0-2-rt-686-pae - Linux 3.2 for modern PCs, PREEMPT_RT
 linux-image-3.2.0-2-rt-amd64-dbg - Debugging infos for Linux 3.2.0-2-rt-amd64
 linux-image-3.2.0-2-rt-amd64 - Linux 3.2 for 64-bit PCs, PREEMPT_RT
 linux-image-3.2.0-2-s390x - Linux 3.2 for IBM zSeries
 linux-image-3.2.0-2-s390x-tape - Linux 3.2 for IBM zSeries, IPL from tape
 linux-image-3.2.0-2-sb1a-bcm91480b - Linux 3.2 for BCM91480B
 linux-image-3.2.0-2-sb1-bcm91250a - Linux 3.2 for BCM91250A
 linux-image-3.2.0-2-sh7751r - Linux 3.2 for sh7751r
 linux-image-3.2.0-2-sh7785lcr - Linux 3.2 for sh7785lcr
 linux-image-3.2.0-2-sparc64 - Linux 3.2 for uniprocessor 64-bit UltraSPARC
 linux-image-3.2.0-2-sparc64-smp - Linux 3.2 for multiprocessor 64-bit UltraSPARC
 linux-image-3.2.0-2-versatile - Linux 3.2 for Versatile
 linux-libc-dev - Linux support headers for userspace development
 linux-manual-3.2 - Linux kernel API manual pages for version 3.2
 linux-patch-debian-3.2 - Debian patches to version 3.2 of the Linux kernel
 linux-source-3.2 - Linux kernel source for version 3.2 with Debian patches
 linux-support-3.2.0-2 - Support files for Linux 3.2
 loop-modules-3.2.0-2-486-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-4kc-malta-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-686-pae-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-amd64-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-iop32x-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-itanium-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-kirkwood-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-loongson-2f-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-mx5-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-orion5x-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-parisc64-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-parisc-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-powerpc64-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-powerpc-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-r4k-ip22-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-r5k-cobalt-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-r5k-ip32-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-sb1-bcm91250a-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-sh7751r-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-sh7785lcr-di - Loopback filesystem support (udeb)
 loop-modules-3.2.0-2-versatile-di - Loopback filesystem support (udeb)
 md-modules-3.2.0-2-486-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-4kc-malta-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-686-pae-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-amd64-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-amiga-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-atari-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-iop32x-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-itanium-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-kirkwood-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-loongson-2f-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-mac-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-mx5-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-orion5x-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-parisc64-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-parisc-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-powerpc64-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-powerpc-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-r4k-ip22-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-r5k-cobalt-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-r5k-ip32-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-s390x-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-sb1-bcm91250a-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-sh7751r-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-sh7785lcr-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-sparc64-di - RAID and LVM support (udeb)
 md-modules-3.2.0-2-versatile-di - RAID and LVM support (udeb)
 minix-modules-3.2.0-2-kirkwood-di - Minix filesystem support (udeb)
 minix-modules-3.2.0-2-mx5-di - Minix filesystem support (udeb)
 minix-modules-3.2.0-2-orion5x-di - Minix filesystem support (udeb)
 minix-modules-3.2.0-2-sh7751r-di - Minix filesystem support (udeb)
 minix-modules-3.2.0-2-sh7785lcr-di - Minix filesystem support (udeb)
 mmc-core-modules-3.2.0-2-486-di - MMC/SD/SDIO core modules (udeb)
 mmc-core-modules-3.2.0-2-686-pae-di - MMC/SD/SDIO core modules (udeb)
 mmc-core-modules-3.2.0-2-amd64-di - MMC/SD/SDIO core modules (udeb)
 mmc-modules-3.2.0-2-486-di - MMC/SD card modules (udeb)
 mmc-modules-3.2.0-2-686-pae-di - MMC/SD card modules (udeb)
 mmc-modules-3.2.0-2-amd64-di - MMC/SD card modules (udeb)
 mmc-modules-3.2.0-2-kirkwood-di - MMC/SD card modules (udeb)
 mmc-modules-3.2.0-2-mx5-di - MMC/SD card modules (udeb)
 mouse-modules-3.2.0-2-486-di - Mouse support (udeb)
 mouse-modules-3.2.0-2-686-pae-di - Mouse support (udeb)
 mouse-modules-3.2.0-2-amd64-di - Mouse support (udeb)
 mouse-modules-3.2.0-2-itanium-di - Mouse support (udeb)
 mouse-modules-3.2.0-2-kirkwood-di - Mouse support (udeb)
 mouse-modules-3.2.0-2-powerpc64-di - Mouse support (udeb)
 mouse-modules-3.2.0-2-powerpc-di - Mouse support (udeb)
 multipath-modules-3.2.0-2-486-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-4kc-malta-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-686-pae-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-amd64-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-iop32x-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-itanium-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-kirkwood-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-loongson-2f-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-mx5-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-orion5x-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-parisc64-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-parisc-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-powerpc64-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-powerpc-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-r4k-ip22-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-r5k-cobalt-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-r5k-ip32-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-s390x-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-sb1-bcm91250a-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-sh7751r-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-sh7785lcr-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-sparc64-di - Multipath support (udeb)
 multipath-modules-3.2.0-2-versatile-di - Multipath support (udeb)
 nbd-modules-3.2.0-2-486-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-4kc-malta-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-686-pae-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-amd64-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-amiga-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-atari-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-iop32x-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-itanium-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-kirkwood-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-loongson-2f-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-mac-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-mx5-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-orion5x-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-parisc64-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-parisc-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-powerpc64-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-powerpc-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-r4k-ip22-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-r5k-cobalt-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-r5k-ip32-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-s390x-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-sb1-bcm91250a-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-sh7751r-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-sh7785lcr-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-sparc64-di - Network Block Device modules (udeb)
 nbd-modules-3.2.0-2-versatile-di - Network Block Device modules (udeb)
 nfs-modules-3.2.0-2-r5k-cobalt-di - NFS filesystem support (udeb)
 nic-extra-modules-3.2.0-2-486-di - Rare NIC drivers (udeb)
 nic-extra-modules-3.2.0-2-686-pae-di - Rare NIC drivers (udeb)
 nic-extra-modules-3.2.0-2-amd64-di - Rare NIC drivers (udeb)
 nic-extra-modules-3.2.0-2-powerpc64-di - Rare NIC drivers (udeb)
 nic-extra-modules-3.2.0-2-powerpc-di - Rare NIC drivers (udeb)
 nic-modules-3.2.0-2-486-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-686-pae-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-amd64-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-iop32x-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-itanium-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-kirkwood-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-loongson-2f-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-mx5-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-orion5x-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-parisc64-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-parisc-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-powerpc64-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-powerpc-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-s390x-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-sh7751r-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-sh7785lcr-di - Common NIC drivers (udeb)
 nic-modules-3.2.0-2-sparc64-di - Network card modules for Sparc kernels (udeb)
 nic-modules-3.2.0-2-versatile-di - Common NIC drivers (udeb)
 nic-pcmcia-modules-3.2.0-2-486-di - Common PCMCIA NIC drivers (udeb)
 nic-pcmcia-modules-3.2.0-2-686-pae-di - Common PCMCIA NIC drivers (udeb)
 nic-pcmcia-modules-3.2.0-2-amd64-di - Common PCMCIA NIC drivers (udeb)
 nic-pcmcia-modules-3.2.0-2-powerpc64-di - Common PCMCIA NIC drivers (udeb)
 nic-pcmcia-modules-3.2.0-2-powerpc-di - Common PCMCIA NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-486-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-686-pae-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-amd64-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-amiga-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-atari-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-iop32x-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-itanium-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-kirkwood-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-loongson-2f-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-mac-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-mx5-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-orion5x-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-powerpc64-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-powerpc-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-r4k-ip22-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-r5k-cobalt-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-r5k-ip32-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-sb1-bcm91250a-di - Shared NIC drivers (udeb)
 nic-shared-modules-3.2.0-2-versatile-di - Shared NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-486-di - USB NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-686-pae-di - USB NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-amd64-di - USB NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-iop32x-di - USB NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-itanium-di - USB NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-kirkwood-di - USB NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-loongson-2f-di - USB NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-mx5-di - USB NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-orion5x-di - USB NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-sh7751r-di - USB NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-sh7785lcr-di - USB NIC drivers (udeb)
 nic-usb-modules-3.2.0-2-versatile-di - USB NIC drivers (udeb)
 nic-wireless-modules-3.2.0-2-486-di - Wireless NIC drivers (udeb)
 nic-wireless-modules-3.2.0-2-686-pae-di - Wireless NIC drivers (udeb)
 nic-wireless-modules-3.2.0-2-amd64-di - Wireless NIC drivers (udeb)
 ntfs-modules-3.2.0-2-486-di - NTFS filesystem support (udeb)
 ntfs-modules-3.2.0-2-686-pae-di - NTFS filesystem support (udeb)
 ntfs-modules-3.2.0-2-amd64-di - NTFS filesystem support (udeb)
 ntfs-modules-3.2.0-2-itanium-di - NTFS filesystem support (udeb)
 parport-modules-3.2.0-2-486-di - Parallel port support (udeb)
 parport-modules-3.2.0-2-686-pae-di - Parallel port support (udeb)
 parport-modules-3.2.0-2-amd64-di - Parallel port support (udeb)
 parport-modules-3.2.0-2-itanium-di - Parallel port support (udeb)
 parport-modules-3.2.0-2-sh7751r-di - Parallel port support (udeb)
 parport-modules-3.2.0-2-sh7785lcr-di - Parallel port support (udeb)
 pata-modules-3.2.0-2-486-di - PATA drivers (udeb)
 pata-modules-3.2.0-2-686-pae-di - PATA drivers (udeb)
 pata-modules-3.2.0-2-amd64-di - PATA drivers (udeb)
 pata-modules-3.2.0-2-iop32x-di - PATA drivers (udeb)
 pata-modules-3.2.0-2-itanium-di - PATA drivers (udeb)
 pata-modules-3.2.0-2-parisc64-di - PATA drivers (udeb)
 pata-modules-3.2.0-2-parisc-di - PATA drivers (udeb)
 pata-modules-3.2.0-2-powerpc64-di - PATA drivers (udeb)
 pata-modules-3.2.0-2-powerpc-di - PATA drivers (udeb)
 pata-modules-3.2.0-2-sh7751r-di - PATA drivers (udeb)
 pata-modules-3.2.0-2-sh7785lcr-di - PATA drivers (udeb)
 pata-modules-3.2.0-2-sparc64-di - PATA drivers (udeb)
 pcmcia-modules-3.2.0-2-486-di - Common PCMCIA drivers (udeb)
 pcmcia-modules-3.2.0-2-686-pae-di - Common PCMCIA drivers (udeb)
 pcmcia-modules-3.2.0-2-amd64-di - Common PCMCIA drivers (udeb)
 pcmcia-modules-3.2.0-2-itanium-di - Common PCMCIA drivers (udeb)
 pcmcia-modules-3.2.0-2-powerpc64-di - Common PCMCIA drivers (udeb)
 pcmcia-modules-3.2.0-2-powerpc-di - Common PCMCIA drivers (udeb)
 pcmcia-storage-modules-3.2.0-2-486-di - PCMCIA storage drivers (udeb)
 pcmcia-storage-modules-3.2.0-2-686-pae-di - PCMCIA storage drivers (udeb)
 pcmcia-storage-modules-3.2.0-2-amd64-di - PCMCIA storage drivers (udeb)
 pcmcia-storage-modules-3.2.0-2-powerpc64-di - PCMCIA storage drivers (udeb)
 pcmcia-storage-modules-3.2.0-2-powerpc-di - PCMCIA storage drivers (udeb)
 plip-modules-3.2.0-2-486-di - PLIP drivers (udeb)
 plip-modules-3.2.0-2-686-pae-di - PLIP drivers (udeb)
 plip-modules-3.2.0-2-amd64-di - PLIP drivers (udeb)
 plip-modules-3.2.0-2-itanium-di - PLIP drivers (udeb)
 plip-modules-3.2.0-2-sparc64-di - PLIP drivers (udeb)
 ppp-modules-3.2.0-2-486-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-4kc-malta-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-686-pae-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-amd64-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-amiga-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-atari-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-iop32x-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-itanium-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-kirkwood-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-loongson-2f-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-mac-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-orion5x-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-parisc64-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-parisc-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-powerpc64-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-powerpc-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-r5k-cobalt-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-sb1-bcm91250a-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-sh7751r-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-sh7785lcr-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-sparc64-di - PPP drivers (udeb)
 ppp-modules-3.2.0-2-versatile-di - PPP drivers (udeb)
 qnx4-modules-3.2.0-2-486-di - QNX4 filesystem support (udeb)
 qnx4-modules-3.2.0-2-686-pae-di - QNX4 filesystem support (udeb)
 qnx4-modules-3.2.0-2-amd64-di - QNX4 filesystem support (udeb)
 reiserfs-modules-3.2.0-2-486-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-4kc-malta-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-686-pae-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-amd64-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-amiga-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-atari-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-iop32x-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-itanium-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-kirkwood-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-loongson-2f-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-mac-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-mx5-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-orion5x-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-powerpc64-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-powerpc-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-r4k-ip22-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-r5k-cobalt-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-r5k-ip32-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-sb1-bcm91250a-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-sh7751r-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-sh7785lcr-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-sparc64-di - Reiser filesystem support (udeb)
 reiserfs-modules-3.2.0-2-versatile-di - Reiser filesystem support (udeb)
 rtc-modules-3.2.0-2-sb1-bcm91250a-di - RTC modules (udeb)
 sata-modules-3.2.0-2-486-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-4kc-malta-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-686-pae-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-amd64-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-iop32x-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-itanium-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-kirkwood-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-loongson-2f-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-orion5x-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-powerpc64-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-powerpc-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-sb1-bcm91250a-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-sh7751r-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-sh7785lcr-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-sparc64-di - SATA drivers (udeb)
 sata-modules-3.2.0-2-versatile-di - SATA drivers (udeb)
 scsi-common-modules-3.2.0-2-486-di - Very common SCSI drivers (udeb)
 scsi-common-modules-3.2.0-2-686-pae-di - Very common SCSI drivers (udeb)
 scsi-common-modules-3.2.0-2-amd64-di - Very common SCSI drivers (udeb)
 scsi-common-modules-3.2.0-2-powerpc64-di - Very common SCSI drivers (udeb)
 scsi-common-modules-3.2.0-2-powerpc-di - Very common SCSI drivers (udeb)
 scsi-common-modules-3.2.0-2-sb1-bcm91250a-di - Very common SCSI drivers (udeb)
 scsi-common-modules-3.2.0-2-sparc64-di - Very common SCSI drivers (udeb)
 scsi-common-modules-3.2.0-2-versatile-di - Very common SCSI drivers (udeb)
 scsi-core-modules-3.2.0-2-486-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-686-pae-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-amd64-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-iop32x-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-itanium-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-kirkwood-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-mx5-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-orion5x-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-parisc64-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-parisc-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-powerpc64-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-powerpc-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-s390x-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-sb1-bcm91250a-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-sh7751r-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-sh7785lcr-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-sparc64-di - Core SCSI subsystem (udeb)
 scsi-core-modules-3.2.0-2-versatile-di - Core SCSI subsystem (udeb)
 scsi-extra-modules-3.2.0-2-486-di - Uncommon SCSI drivers (udeb)
 scsi-extra-modules-3.2.0-2-686-pae-di - Uncommon SCSI drivers (udeb)
 scsi-extra-modules-3.2.0-2-amd64-di - Uncommon SCSI drivers (udeb)
 scsi-extra-modules-3.2.0-2-powerpc64-di - Uncommon SCSI drivers (udeb)
 scsi-extra-modules-3.2.0-2-powerpc-di - Uncommon SCSI drivers (udeb)
 scsi-modules-3.2.0-2-486-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-686-pae-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-amd64-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-amiga-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-atari-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-itanium-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-mac-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-parisc64-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-parisc-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-powerpc64-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-powerpc-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-s390x-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-sb1-bcm91250a-di - SCSI drivers (udeb)
 scsi-modules-3.2.0-2-sparc64-di - SCSI drivers (udeb)
 serial-modules-3.2.0-2-486-di - Serial drivers (udeb)
 serial-modules-3.2.0-2-686-pae-di - Serial drivers (udeb)
 serial-modules-3.2.0-2-amd64-di - Serial drivers (udeb)
 serial-modules-3.2.0-2-itanium-di - Serial drivers (udeb)
 serial-modules-3.2.0-2-powerpc64-di - Serial drivers (udeb)
 serial-modules-3.2.0-2-powerpc-di - Serial drivers (udeb)
 sn-modules-3.2.0-2-itanium-di - SN modules (udeb)
 sound-modules-3.2.0-2-486-di - sound support (udeb)
 sound-modules-3.2.0-2-686-pae-di - sound support (udeb)
 sound-modules-3.2.0-2-amd64-di - sound support (udeb)
 sound-modules-3.2.0-2-sh7751r-di - sound support (udeb)
 sound-modules-3.2.0-2-sh7785lcr-di - sound support (udeb)
 speakup-modules-3.2.0-2-486-di - speakup modules (udeb)
 speakup-modules-3.2.0-2-686-pae-di - speakup modules (udeb)
 speakup-modules-3.2.0-2-amd64-di - speakup modules (udeb)
 speakup-modules-3.2.0-2-sh7751r-di - speakup modules (udeb)
 speakup-modules-3.2.0-2-sh7785lcr-di - speakup modules (udeb)
 squashfs-modules-3.2.0-2-486-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-4kc-malta-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-686-pae-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-amd64-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-amiga-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-atari-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-iop32x-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-itanium-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-kirkwood-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-loongson-2f-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-mac-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-mx5-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-orion5x-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-powerpc64-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-powerpc-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-r4k-ip22-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-r5k-cobalt-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-r5k-ip32-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-sb1-bcm91250a-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-sh7751r-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-sh7785lcr-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-sparc64-di - squashfs modules (udeb)
 squashfs-modules-3.2.0-2-versatile-di - squashfs modules (udeb)
 ufs-modules-3.2.0-2-486-di - UFS filesystem support (udeb)
 ufs-modules-3.2.0-2-686-pae-di - UFS filesystem support (udeb)
 ufs-modules-3.2.0-2-amd64-di - UFS filesystem support (udeb)
 ufs-modules-3.2.0-2-itanium-di - UFS filesystem support (udeb)
 ufs-modules-3.2.0-2-powerpc64-di - UFS filesystem support (udeb)
 ufs-modules-3.2.0-2-powerpc-di - UFS filesystem support (udeb)
 ufs-modules-3.2.0-2-sh7751r-di - UFS filesystem support (udeb)
 ufs-modules-3.2.0-2-sh7785lcr-di - UFS filesystem support (udeb)
 uinput-modules-3.2.0-2-486-di - uinput support (udeb)
 uinput-modules-3.2.0-2-686-pae-di - uinput support (udeb)
 uinput-modules-3.2.0-2-amd64-di - uinput support (udeb)
 uinput-modules-3.2.0-2-itanium-di - uinput support (udeb)
 uinput-modules-3.2.0-2-kirkwood-di - uinput support (udeb)
 uinput-modules-3.2.0-2-mx5-di - uinput support (udeb)
 uinput-modules-3.2.0-2-powerpc64-di - uinput support (udeb)
 uinput-modules-3.2.0-2-powerpc-di - uinput support (udeb)
 usb-modules-3.2.0-2-486-di - USB support (udeb)
 usb-modules-3.2.0-2-4kc-malta-di - USB support (udeb)
 usb-modules-3.2.0-2-686-pae-di - USB support (udeb)
 usb-modules-3.2.0-2-amd64-di - USB support (udeb)
 usb-modules-3.2.0-2-iop32x-di - USB support (udeb)
 usb-modules-3.2.0-2-itanium-di - USB support (udeb)
 usb-modules-3.2.0-2-kirkwood-di - USB support (udeb)
 usb-modules-3.2.0-2-loongson-2f-di - USB support (udeb)
 usb-modules-3.2.0-2-orion5x-di - USB support (udeb)
 usb-modules-3.2.0-2-parisc64-di - USB support (udeb)
 usb-modules-3.2.0-2-parisc-di - USB support (udeb)
 usb-modules-3.2.0-2-powerpc64-di - USB support (udeb)
 usb-modules-3.2.0-2-powerpc-di - USB support (udeb)
 usb-modules-3.2.0-2-sb1-bcm91250a-di - USB support (udeb)
 usb-modules-3.2.0-2-sparc64-di - USB support (udeb)
 usb-modules-3.2.0-2-versatile-di - USB support (udeb)
 usb-serial-modules-3.2.0-2-486-di - USB serial drivers (udeb)
 usb-serial-modules-3.2.0-2-686-pae-di - USB serial drivers (udeb)
 usb-serial-modules-3.2.0-2-amd64-di - USB serial drivers (udeb)
 usb-serial-modules-3.2.0-2-iop32x-di - USB serial drivers (udeb)
 usb-serial-modules-3.2.0-2-kirkwood-di - USB serial drivers (udeb)
 usb-serial-modules-3.2.0-2-orion5x-di - USB serial drivers (udeb)
 usb-serial-modules-3.2.0-2-powerpc64-di - USB serial drivers (udeb)
 usb-serial-modules-3.2.0-2-powerpc-di - USB serial drivers (udeb)
 usb-serial-modules-3.2.0-2-sh7751r-di - USB serial drivers (udeb)
 usb-serial-modules-3.2.0-2-sh7785lcr-di - USB serial drivers (udeb)
 usb-serial-modules-3.2.0-2-versatile-di - USB serial drivers (udeb)
 usb-storage-modules-3.2.0-2-486-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-4kc-malta-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-686-pae-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-amd64-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-iop32x-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-itanium-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-kirkwood-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-loongson-2f-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-mx5-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-orion5x-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-parisc64-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-parisc-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-powerpc64-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-powerpc-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-sb1-bcm91250a-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-sh7751r-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-sparc64-di - USB storage support (udeb)
 usb-storage-modules-3.2.0-2-versatile-di - USB storage support (udeb)
 virtio-modules-3.2.0-2-486-di - virtio modules (udeb)
 virtio-modules-3.2.0-2-4kc-malta-di - virtio modules (udeb)
 virtio-modules-3.2.0-2-686-pae-di - virtio modules (udeb)
 virtio-modules-3.2.0-2-amd64-di - virtio modules (udeb)
 virtio-modules-3.2.0-2-loongson-2f-di - virtio modules (udeb)
 virtio-modules-3.2.0-2-powerpc64-di - virtio modules (udeb)
 virtio-modules-3.2.0-2-powerpc-di - virtio modules (udeb)
 virtio-modules-3.2.0-2-s390x-di - virtio modules (udeb)
 virtio-modules-3.2.0-2-sh7751r-di - virtio modules (udeb)
 virtio-modules-3.2.0-2-versatile-di - virtio modules (udeb)
 xen-linux-system-3.2.0-2-686-pae - Xen system with Linux 3.2 on modern PCs (meta-package)
 xen-linux-system-3.2.0-2-amd64 - Xen system with Linux 3.2 on 64-bit PCs (meta-package)
 xfs-modules-3.2.0-2-486-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-4kc-malta-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-686-pae-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-amd64-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-itanium-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-loongson-2f-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-powerpc64-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-powerpc-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-r4k-ip22-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-r5k-cobalt-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-r5k-ip32-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-s390x-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-sb1-bcm91250a-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-sh7751r-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-sh7785lcr-di - XFS filesystem support (udeb)
 xfs-modules-3.2.0-2-sparc64-di - XFS filesystem support (udeb)
 zlib-modules-3.2.0-2-486-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-4kc-malta-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-686-pae-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-amd64-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-amiga-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-atari-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-iop32x-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-itanium-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-loongson-2f-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-mac-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-orion5x-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-parisc64-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-parisc-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-powerpc-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-r4k-ip22-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-r5k-cobalt-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-r5k-ip32-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-sb1-bcm91250a-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-sh7751r-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-sh7785lcr-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-sparc64-di - zlib modules (udeb)
 zlib-modules-3.2.0-2-versatile-di - zlib modules (udeb)
Changes: 
 linux-2.6 (3.2.14-1) unstable; urgency=low
 .
   * New upstream stable update:
     http://www.kernel.org/pub/linux/kernel/v3.x/ChangeLog-3.2.14
     - mm: thp: fix pmd_bad() triggering in code paths holding mmap_sem
       read mode (CVE-2012-1179)
     - hugetlbfs: avoid taking i_mutex from hugetlbfs_read()
     - md/bitmap: ensure to load bitmap when creating via sysfs
       (Closes: #661558)
     - md: dont set md arrays to readonly on shutdown
     - md/raid1,raid10: avoid deadlock during resync/recovery (Closes: #584881)
     - md: fix clearing of the changed flags for the bad blocks list
     - xfs: fix inode lookup race
     - sysctl: protect poll() in entries that may go away
     - NFSv4: Rate limit the state manager warning messages (Closes: #666121)
     - jbd2: clear BH_Delay & BH_Unwritten in journal_unmap_buffer
     - ext4: ignore EXT4_INODE_JOURNAL_DATA flag with delalloc
     - ext4: fix race between sync and completed io work
     - ext4: check for zero length extent
     - vfs: fix d_ancestor() case in d_materialize_unique
     - udf: Fix deadlock in udf_release_file()
     - dm crypt: add missing error handling
     - dm thin: fix stacked bi_next usage
     - xfs: Fix oops on IO error during xlog_recover_process_iunlinks()
     - NFSv4: Fix two infinite loops in the mount code
     - drm/i915: suspend fbdev device around suspend/hibernate
       (Closes: #645547)
     - net: fix a potential rcu_read_lock() imbalance in rt6_fill_node()
     - [x86] tls: Off by one limit check
     - PCI: ASPM: Fix pcie devices with non-pcie children (Closes: #665420)
 .
   [ Jonathan Nieder ]
   * ata: Enable PATA_IT8213 as module (Closes: #666506)
Checksums-Sha1: 
 a3dad31d8489c48be9ae4ea569bbab79161ddc76 93822 linux-2.6_3.2.14-1.dsc
 d290b26c48354a78c65882fd7bd4e9d4f7d8236e 98469236 linux-2.6_3.2.14.orig.tar.gz
 26454096689b14c1a85325c4a984605b5b28ed0e 5756357 linux-2.6_3.2.14-1.diff.gz
 1def7644661beced7145b7820d5fb119d6009ad9 189602 linux-support-3.2.0-2_3.2.14-1_all.deb
 c9d8097ef60e51ba904da964b7c65780c09131ef 745052 linux-patch-debian-3.2_3.2.14-1_all.deb
 13602f4ced89128a2aaa8017bdbda2b0245db7eb 76804164 linux-source-3.2_3.2.14-1_all.deb
 8bd3fcffc24e729b90ccd81f1826bf397e172e50 6346310 linux-doc-3.2_3.2.14-1_all.deb
 c0df8e96cd172a0ff199504677d3051058ed4965 2893498 linux-manual-3.2_3.2.14-1_all.deb
Checksums-Sha256: 
 e337af6dead83e581d9a2ee0d278007117b4fd9064064341693d8538f5fee780 93822 linux-2.6_3.2.14-1.dsc
 fc40acd185c0b4a2455c7bb3a05c994f040ae07eb80aae0b4dd0897b9df406a1 98469236 linux-2.6_3.2.14.orig.tar.gz
 4896969152d1ebc57112bfd01a5fa5db5809fafb1c50cda9076604471dfcca80 5756357 linux-2.6_3.2.14-1.diff.gz
 38ff9551c5754de6767c855fc4f4bcb48ef218add6550ad6b09d6e1a1a01d43f 189602 linux-support-3.2.0-2_3.2.14-1_all.deb
 bb0510d3d317fa2c7b9ba247456ae9addc228cd38665580aba58671e18fc913c 745052 linux-patch-debian-3.2_3.2.14-1_all.deb
 e5879b402267fd6a606d490bab7b935d763d81661d6b4289fa8b6cb59d133044 76804164 linux-source-3.2_3.2.14-1_all.deb
 67782f98a7d69201a38d08abb5661f1056b9d1ac867612292e7bd31e43d4d33e 6346310 linux-doc-3.2_3.2.14-1_all.deb
 5b7a00da772b7653598aabfe96130c31b52a1054d57db12e0a3dbbdcda3a25b8 2893498 linux-manual-3.2_3.2.14-1_all.deb
Files: 
 1faa328aee468e1b03cfd459c8fcd09c 93822 kernel optional linux-2.6_3.2.14-1.dsc
 ee140bf80a0cd9afcae59ce504321c44 98469236 kernel optional linux-2.6_3.2.14.orig.tar.gz
 9606173936e3e600482a35c68a0c6a71 5756357 kernel optional linux-2.6_3.2.14-1.diff.gz
 535b46cf70840b2a4c6efa632239b15f 189602 devel optional linux-support-3.2.0-2_3.2.14-1_all.deb
 86bc08e3bd37e3e5c7f59a0b486874a8 745052 kernel optional linux-patch-debian-3.2_3.2.14-1_all.deb
 b15e5ab2276c6f412f05902872cb02ae 76804164 kernel optional linux-source-3.2_3.2.14-1_all.deb
 be8ca347a7e1f3738d22feaaf5e864dc 6346310 doc optional linux-doc-3.2_3.2.14-1_all.deb
 566082ae40438b321ba7a5bff41ab4b3 2893498 doc optional linux-manual-3.2_3.2.14-1_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIVAwUBT35Pp+e/yOyVhhEJAQrixRAAvulHPlOyhnOG8FlBZzC1dmTBct5hH1Sg
AZhFW+AeWXYZ3+94HaoU4wO51xXD3YMhBpJ/hihtBbxBpCIqb4wTQlR0pTU4hHZu
Xvpbj7zs0CH7FRixrLZjO6s4WlaIXk+Uo3fl6u52vCk/CTawnsjkkKUJtD2xn0wK
o5zKTHLkq+v19y2/PnswRWvsD3L+MV/IwGWDGP0Wt5ywI5Kv7jXss83FbUXfYBYK
OjDRW4Vw7lf4Tc3bDw3qM9tId6YuiaX2VWiS61ouIltruS9F3VewDAezh3mpp02z
tp+CtEE6sMpsdwtf5b34VnVgT77W6cshJBvY56HeQg/fb7JGsC+tO1DEnrC2ERmT
5xNeDwKTPeienF7WtTLALHehZk6pr9EEUbONaKIoUr12PIX3HFEIG9QxCj8gEKy9
hRCbVs2Ugr67+G+p/9xsbnSd2zeeoXE0J8gKPJZehUEp27gzgEQcbn8ik9FK5biw
yNDTWRO1vuv07QwEayW4gqD9HjJQq86PBgw+3cPPA1c6ywZFS9phChnDw+xnwvaw
bpDMve/omQw9cfPfuxtqJ3vOhNFadyn9Pvuodq55SSZ4FvWzfd8eLnhD4TB+/PLG
/QT9Y8Alkzp5r45AxuGTtdu/jku1MhbNNkLGNA4agiOQjD5tNCsDxJYI1qMVDMqt
/1/XXIrBI8k=
=rzHi
-----END PGP SIGNATURE-----





Added tag(s) pending. Request was from Ben Hutchings <benh@alioth.debian.org> to control@bugs.debian.org. (Fri, 06 Apr 2012 21:03:11 GMT) Full text and rfc822 format available.

Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Sun, 08 Jul 2012 08:00:48 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Fri Apr 18 19:11:39 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.