Debian Bug report logs - #604453
reproducible way to crash the system with kernel.function("*@*/*.c")

Package: systemtap; Maintainer for systemtap is Ritesh Raj Sarraf <rrs@debian.org>; Source for systemtap is src:systemtap.

Reported by: Timo Juhani Lindfors <timo.lindfors@iki.fi>

Date: Mon, 22 Nov 2010 10:48:08 UTC

Severity: minor

Forwarded to http://sourceware.org/bugzilla/show_bug.cgi?id=2725

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#604453; Package linux-2.6. (Mon, 22 Nov 2010 10:48:11 GMT) Full text and rfc822 format available.

Acknowledgement sent to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
New Bug report received and forwarded. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Mon, 22 Nov 2010 10:48:11 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Timo Juhani Lindfors <timo.lindfors@iki.fi>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: linux-2.6: xen domU hangs with systemtap kernel.function("*@*/*.c")
Date: Mon, 22 Nov 2010 12:46:36 +0200
Package: linux-2.6
Version: 2.6.32-27
Severity: minor

[ Note that I have applied a patch from #602273 so this is not a
completely unmodified linux-2.6 package.]

Steps to reproduce:
1) cat > mmap-trace.stp <<EOF
global trace

function trace(entry_p) {
  if(tid() in trace)
    printf("%s%s%s\n",
      thread_indent(entry_p),
      (entry_p>0?"->":"<-"),
      probefunc())
}

probe syscall.execve {
#  printf("%s\n", filename);
  if (filename != "./mmap") next
  trace[tid()] = 1
  trace(1)
}
#probe syscall.execve.return {
#   trace(-1)
#   delete trace[tid()]
#}
#probe kernel.function("*@mm/*.c").call { trace(1) }
#probe kernel.function("*@mm/*.c").return { trace(-1) }

probe kernel.function("*@*/*.c").call { trace(1) }
probe kernel.function("*@*/*.c").return { trace(-1) }

probe begin {
    printf("begin\n");
}
EOF
2) sudo stap ./mmap-trace.stp

Expected results:
2) The system does not crash

Actual results:
2) The system hangs and does not respond to ping.

More info:
1) dmesg from a crasdump shows

[   15.529018] br0: no IPv6 routers present
[   15.669525] eth0: no IPv6 routers present
[  216.773795] stap_db2bea815fe5469cb8137996bcbc994c_6678582: systemtap: 1.2/0.148, base: ffffffffa012c000, memory: 13010data/22text/64ctx/13net/377alloc kb, probes: 31451
[  216.774296] ------------[ cut here ]------------
[  216.774312] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:508 kprobe_exceptions_notify+0x21d/0x5f8()
[  216.774324] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.774368] Pid: 1936, comm: stapio Not tainted 2.6.32-5+lindi.5-amd64 #1
[  216.774375] Call Trace:
[  216.774382]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.774389]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.774398]  [<ffffffff8104cd0c>] ? warn_slowpath_common+0x77/0xa3
[  216.774407]  [<ffffffff8100e1f2>] ? check_events+0x12/0x20
[  216.774414]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.774422]  [<ffffffff812f59ab>] ? notifier_call_chain+0x29/0x4c
[  216.774431]  [<ffffffff81067444>] ? notify_die+0x2d/0x32
[  216.774438]  [<ffffffff812f3dbf>] ? do_int3+0x2f/0x98
[  216.774447]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.774454]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.774462]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.774470]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.774477]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.774484]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.774493]  [<ffffffff8100cb88>] ? xen_mc_entry+0x2e/0x33
[  216.774500]  [<ffffffff8100cd08>] ? xen_flush_tlb_single+0x19/0x6d
[  216.774508]  [<ffffffff8100cdcb>] ? xen_set_pte+0x12/0x1b
[  216.774516]  [<ffffffff81030971>] ? set_pte_vaddr_pud+0x41/0x4b
[  216.774524]  [<ffffffff81034a46>] ? __native_set_fixmap+0x24/0x2c
[  216.774532]  [<ffffffff8100c9c5>] ? xen_set_fixmap+0x86/0xb5
[  216.774539]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.774546]  [<ffffffff812f45ef>] ? text_poke+0x13d/0x19b
[  216.774554]  [<ffffffff812f5449>] ? arch_arm_kprobe+0x1c/0x21
[  216.774561]  [<ffffffff812f6b0d>] ? register_kprobe+0x452/0x4ad
[  216.774569]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.774578]  [<ffffffffa0131575>] ? _stp_ctl_write_cmd+0x770/0x84e [stap_db2bea815fe5469cb8137996bcbc994c_6678582]
[  216.774589]  [<ffffffff810ed68e>] ? vfs_write+0xa9/0x102
[  216.774596]  [<ffffffff810ed7a3>] ? sys_write+0x45/0x6e
[  216.774604]  [<ffffffff81010e63>] ? stub_clone+0x13/0x20
[  216.774611]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[  216.774617] ---[ end trace 6100392551b72adc ]---
[  216.774625] int3: 0000 [#1] SMP 
[  216.774633] last sysfs file: /sys/module/xen_netfront/sections/__bug_table
[  216.774639] CPU 0 
[  216.774644] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.774687] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1 
[  216.774694] RIP: e030:[<ffffffff8100b0a9>]  [<ffffffff8100b0a9>] xen_get_debugreg+0x1/0x9
[  216.774704] RSP: e02b:ffff88003778dc20  EFLAGS: 00000086
[  216.774710] RAX: 000000000000c570 RBX: 0000000000000001 RCX: 000000008101154e
[  216.774717] RDX: ffff8800020a0000 RSI: 0000000000000000 RDI: 0000000000000006
[  216.774724] RBP: ffff88003778dc68 R08: 0000000000000001 R09: 0000000000000040
[  216.774731] R10: ffffffff8100c157 R11: 0000000000000040 R12: ffff88003ee04d00
[  216.774738] R13: 00000000000302d8 R14: 0000000000000000 R15: 0000160000000000
[  216.774746] ------------[ cut here ]------------
[  216.774753] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:508 kprobe_exceptions_notify+0x21d/0x5f8()
[  216.774764] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.774805] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.774812] Call Trace:
[  216.774818]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.774826]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.774833]  [<ffffffff8104cd0c>] ? warn_slowpath_common+0x77/0xa3
[  216.774841]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.774849]  [<ffffffff812f59ab>] ? notifier_call_chain+0x29/0x4c
[  216.774857]  [<ffffffff81067444>] ? notify_die+0x2d/0x32
[  216.774864]  [<ffffffff812f3dbf>] ? do_int3+0x2f/0x98
[  216.774871]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.774879]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.774886]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.774894]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.774901]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.774909]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.774917]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.774925]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.774932]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.774939]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.774946]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.774953]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.774960]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.774968]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.774975]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.774982]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.774989]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.774998]  [<ffffffff8100cb88>] ? xen_mc_entry+0x2e/0x33
[  216.775005]  [<ffffffff8100cd08>] ? xen_flush_tlb_single+0x19/0x6d
[  216.775012]  [<ffffffff8100cdcb>] ? xen_set_pte+0x12/0x1b
[  216.775019]  [<ffffffff81030971>] ? set_pte_vaddr_pud+0x41/0x4b
[  216.775027]  [<ffffffff81034a46>] ? __native_set_fixmap+0x24/0x2c
[  216.775035]  [<ffffffff8100c9c5>] ? xen_set_fixmap+0x86/0xb5
[  216.775042]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.775049]  [<ffffffff812f45ef>] ? text_poke+0x13d/0x19b
[  216.775056]  [<ffffffff812f5449>] ? arch_arm_kprobe+0x1c/0x21
[  216.775064]  [<ffffffff812f6b0d>] ? register_kprobe+0x452/0x4ad
[  216.775071]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.775079]  [<ffffffffa0131575>] ? _stp_ctl_write_cmd+0x770/0x84e [stap_db2bea815fe5469cb8137996bcbc994c_6678582]
[  216.775089]  [<ffffffff810ed68e>] ? vfs_write+0xa9/0x102
[  216.775096]  [<ffffffff810ed7a3>] ? sys_write+0x45/0x6e
[  216.775103]  [<ffffffff81010e63>] ? stub_clone+0x13/0x20
[  216.775110]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[  216.775116] ---[ end trace 6100392551b72add ]---
[  216.775122] int3: 0000 [#2] SMP 
[  216.775129] last sysfs file: /sys/module/xen_netfront/sections/__bug_table
[  216.775135] CPU 0 
[  216.775140] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.775181] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1 
[  216.775188] RIP: e030:[<ffffffff8100b0d7>]  [<ffffffff8100b0d7>] xen_read_cr0+0x1/0x1b
[  216.776533] ------------[ cut here ]------------
[  216.776544] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:508 kprobe_exceptions_notify+0x21d/0x5f8()
[  216.776549] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776576] Pid: 0, comm: swapper Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.776579] Call Trace:
[  216.776585]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776591]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776597]  [<ffffffff8104cd0c>] ? warn_slowpath_common+0x77/0xa3
[  216.776603]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776608]  [<ffffffff812f59ab>] ? notifier_call_chain+0x29/0x4c
[  216.776615]  [<ffffffff81067444>] ? notify_die+0x2d/0x32
[  216.776620]  [<ffffffff812f3dbf>] ? do_int3+0x2f/0x98
[  216.776626]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776633]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776639]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.776644]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.776649]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.776655]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776661]  [<ffffffff8100feb1>] ? cpu_idle+0xa2/0xda
[  216.776667]  [<ffffffff8100e199>] ? xen_irq_enable_direct_end+0x0/0x7
[  216.776673]  [<ffffffff812e7d8e>] ? cpu_bringup+0x6d/0x72
[  216.776677] ---[ end trace 6100392551b72ade ]---
[  216.792045] ------------[ cut here ]------------
[  216.792050] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:508 kprobe_exceptions_notify+0x21d/0x5f8()
[  216.792052] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.792068] Pid: 0, comm: swapper Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.792070] Call Trace:
[  216.792073] ---[ end trace 6100392551b72adf ]---
[  216.855381] ------------[ cut here ]------------
[  216.855386] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:508 kprobe_exceptions_notify+0x21d/0x5f8()
[  216.855389] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.855405] Pid: 0, comm: swapper Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.855408] Call Trace:
[  216.855412]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.855415]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.855419]  [<ffffffff8104cd0c>] ? warn_slowpath_common+0x77/0xa3
[  216.855423]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.855428]  [<ffffffff811924b2>] ? vsnprintf+0x9e/0x449
[  216.855431]  [<ffffffff812f59ab>] ? notifier_call_chain+0x29/0x4c
[  216.855435]  [<ffffffff81067444>] ? notify_die+0x2d/0x32
[  216.855438]  [<ffffffff812f3dbf>] ? do_int3+0x2f/0x98
[  216.855442]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.855446]  [<ffffffff8100e10c>] ? xen_vcpuop_set_next_event+0x0/0x60
[  216.855450]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.855453]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.855456]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.855460]  [<ffffffff8100e10c>] ? xen_vcpuop_set_next_event+0x0/0x60
[  216.855464]  [<ffffffff8100ceb8>] ? __xen_write_cr3+0x49/0xc4
[  216.855469]  [<ffffffff81040a7d>] ? pick_next_task_fair+0xca/0xd6
[  216.855472]  [<ffffffff8100cf72>] ? xen_write_cr3+0x3f/0x7a
[  216.855476]  [<ffffffff8104641e>] ? switch_mm+0x5d/0xcb
[  216.855480]  [<ffffffff812f2031>] ? schedule+0x75e/0x79f
[  216.855483]  [<ffffffff8100e10c>] ? xen_vcpuop_set_next_event+0x0/0x60
[  216.855487]  [<ffffffff8106683e>] ? hrtimer_try_to_cancel+0x3a/0x43
[  216.855491]  [<ffffffff8100fee7>] ? cpu_idle+0xd8/0xda
[  216.855495]  [<ffffffff8100e199>] ? xen_irq_enable_direct_end+0x0/0x7
[  216.855498]  [<ffffffff812e7d8e>] ? cpu_bringup+0x6d/0x72
[  216.855501] ---[ end trace 6100392551b72ae0 ]---
[  216.776009] RSP: e02b:ffff88003778d950  EFLAGS: 00000002
[  216.776009] RAX: 0000000000000000 RBX: ffff88003778da7c RCX: 0000000000000000
[  216.776009] RDX: 0000000000000000 RSI: ffff88003778da7c RDI: 0000000000000000
[  216.776009] RBP: 00007f7f31eb5700 R08: 0000000000004bde R09: 000000000000000a
[  216.776009] R10: 0000000000000000 R11: ffffffff811b4174 R12: ffffffff813c2cc1
[  216.776009] R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000000
[  216.776009] ------------[ cut here ]------------
[  216.776009] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:508 kprobe_exceptions_notify+0x21d/0x5f8()
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.776009] Call Trace:
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff8104cd0c>] ? warn_slowpath_common+0x77/0xa3
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f59ab>] ? notifier_call_chain+0x29/0x4c
[  216.776009]  [<ffffffff81067444>] ? notify_die+0x2d/0x32
[  216.776009]  [<ffffffff812f3dbf>] ? do_int3+0x2f/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.776009]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.776009]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100cb88>] ? xen_mc_entry+0x2e/0x33
[  216.776009]  [<ffffffff8100cd08>] ? xen_flush_tlb_single+0x19/0x6d
[  216.776009]  [<ffffffff8100cdcb>] ? xen_set_pte+0x12/0x1b
[  216.776009]  [<ffffffff81030971>] ? set_pte_vaddr_pud+0x41/0x4b
[  216.776009]  [<ffffffff81034a46>] ? __native_set_fixmap+0x24/0x2c
[  216.776009]  [<ffffffff8100c9c5>] ? xen_set_fixmap+0x86/0xb5
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffff812f45ef>] ? text_poke+0x13d/0x19b
[  216.776009]  [<ffffffff812f5449>] ? arch_arm_kprobe+0x1c/0x21
[  216.776009]  [<ffffffff812f6b0d>] ? register_kprobe+0x452/0x4ad
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffffa0131575>] ? _stp_ctl_write_cmd+0x770/0x84e [stap_db2bea815fe5469cb8137996bcbc994c_6678582]
[  216.776009]  [<ffffffff810ed68e>] ? vfs_write+0xa9/0x102
[  216.776009]  [<ffffffff810ed7a3>] ? sys_write+0x45/0x6e
[  216.776009]  [<ffffffff81010e63>] ? stub_clone+0x13/0x20
[  216.776009]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[  216.776009] ---[ end trace 6100392551b72ae1 ]---
[  216.776009] int3: 0000 [#3] SMP 
[  216.776009] last sysfs file: /sys/module/xen_netfront/sections/__bug_table
[  216.776009] CPU 0 
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1 
[  216.776009] RIP: e030:[<ffffffff8100b0d7>]  [<ffffffff8100b0d7>] xen_read_cr0+0x1/0x1b
[  216.776009] RSP: e02b:ffff88003778d680  EFLAGS: 00000002
[  216.776009] RAX: 0000000000000000 RBX: ffff88003778d7ac RCX: 0000000000000000
[  216.776009] RDX: 0000000000000000 RSI: ffff88003778d7ac RDI: 0000000000000000
[  216.776009] RBP: 00007f7f31eb5700 R08: 0000000000006f19 R09: 000000000000000a
[  216.776009] R10: 0000000000000000 R11: ffffffff811b4174 R12: ffffffff813c2cc1
[  216.776009] R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000000
[  216.776009] ------------[ cut here ]------------
[  216.776009] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:508 kprobe_exceptions_notify+0x21d/0x5f8()
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.776009] Call Trace:
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff8104cd0c>] ? warn_slowpath_common+0x77/0xa3
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f59ab>] ? notifier_call_chain+0x29/0x4c
[  216.776009]  [<ffffffff81067444>] ? notify_die+0x2d/0x32
[  216.776009]  [<ffffffff812f3dbf>] ? do_int3+0x2f/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.776009]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.776009]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100cb88>] ? xen_mc_entry+0x2e/0x33
[  216.776009]  [<ffffffff8100cd08>] ? xen_flush_tlb_single+0x19/0x6d
[  216.776009]  [<ffffffff8100cdcb>] ? xen_set_pte+0x12/0x1b
[  216.776009]  [<ffffffff81030971>] ? set_pte_vaddr_pud+0x41/0x4b
[  216.776009]  [<ffffffff81034a46>] ? __native_set_fixmap+0x24/0x2c
[  216.776009]  [<ffffffff8100c9c5>] ? xen_set_fixmap+0x86/0xb5
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffff812f45ef>] ? text_poke+0x13d/0x19b
[  216.776009]  [<ffffffff812f5449>] ? arch_arm_kprobe+0x1c/0x21
[  216.776009]  [<ffffffff812f6b0d>] ? register_kprobe+0x452/0x4ad
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffffa0131575>] ? _stp_ctl_write_cmd+0x770/0x84e [stap_db2bea815fe5469cb8137996bcbc994c_6678582]
[  216.776009]  [<ffffffff810ed68e>] ? vfs_write+0xa9/0x102
[  216.776009]  [<ffffffff810ed7a3>] ? sys_write+0x45/0x6e
[  216.776009]  [<ffffffff81010e63>] ? stub_clone+0x13/0x20
[  216.776009]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[  216.776009] ---[ end trace 6100392551b72ae2 ]---
[  216.776009] int3: 0000 [#4] SMP 
[  216.776009] last sysfs file: /sys/module/xen_netfront/sections/__bug_table
[  216.776009] CPU 0 
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1 
[  216.776009] RIP: e030:[<ffffffff8100b0d7>]  [<ffffffff8100b0d7>] xen_read_cr0+0x1/0x1b
[  216.776009] RSP: e02b:ffff88003778d3b0  EFLAGS: 00000002
[  216.776009] RAX: 0000000000000000 RBX: ffff88003778d4dc RCX: 0000000000000000
[  216.776009] RDX: 0000000000000000 RSI: ffff88003778d4dc RDI: 0000000000000000
[  216.776009] RBP: 00007f7f31eb5700 R08: 00000000000082fe R09: 000000000000000a
[  216.776009] R10: 0000000000000000 R11: ffffffff811b4174 R12: ffffffff813c2cc1
[  216.776009] R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000000
[  216.776009] ------------[ cut here ]------------
[  216.776009] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:508 kprobe_exceptions_notify+0x21d/0x5f8()
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.776009] Call Trace:
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff8104cd0c>] ? warn_slowpath_common+0x77/0xa3
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f59ab>] ? notifier_call_chain+0x29/0x4c
[  216.776009]  [<ffffffff81067444>] ? notify_die+0x2d/0x32
[  216.776009]  [<ffffffff812f3dbf>] ? do_int3+0x2f/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.776009]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.776009]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100cb88>] ? xen_mc_entry+0x2e/0x33
[  216.776009]  [<ffffffff8100cd08>] ? xen_flush_tlb_single+0x19/0x6d
[  216.776009]  [<ffffffff8100cdcb>] ? xen_set_pte+0x12/0x1b
[  216.776009]  [<ffffffff81030971>] ? set_pte_vaddr_pud+0x41/0x4b
[  216.776009]  [<ffffffff81034a46>] ? __native_set_fixmap+0x24/0x2c
[  216.776009]  [<ffffffff8100c9c5>] ? xen_set_fixmap+0x86/0xb5
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffff812f45ef>] ? text_poke+0x13d/0x19b
[  216.776009]  [<ffffffff812f5449>] ? arch_arm_kprobe+0x1c/0x21
[  216.776009]  [<ffffffff812f6b0d>] ? register_kprobe+0x452/0x4ad
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffffa0131575>] ? _stp_ctl_write_cmd+0x770/0x84e [stap_db2bea815fe5469cb8137996bcbc994c_6678582]
[  216.776009]  [<ffffffff810ed68e>] ? vfs_write+0xa9/0x102
[  216.776009]  [<ffffffff810ed7a3>] ? sys_write+0x45/0x6e
[  216.776009]  [<ffffffff81010e63>] ? stub_clone+0x13/0x20
[  216.776009]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[  216.776009] ---[ end trace 6100392551b72ae3 ]---
[  216.776009] int3: 0000 [#5] SMP 
[  216.776009] last sysfs file: /sys/module/xen_netfront/sections/__bug_table
[  216.776009] CPU 0 
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1 
[  216.776009] RIP: e030:[<ffffffff8100b0d7>]  [<ffffffff8100b0d7>] xen_read_cr0+0x1/0x1b
[  216.776009] RSP: e02b:ffff88003778d0e0  EFLAGS: 00000002
[  216.776009] RAX: 0000000000000000 RBX: ffff88003778d20c RCX: 0000000000000000
[  216.776009] RDX: 0000000000000000 RSI: ffff88003778d20c RDI: 0000000000000000
[  216.776009] RBP: 00007f7f31eb5700 R08: 000000000000999a R09: 000000000000000a
[  216.776009] R10: 0000000000000000 R11: ffffffff811b4174 R12: ffffffff813c2cc1
[  216.776009] R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000000
[  216.776009] ------------[ cut here ]------------
[  216.776009] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:508 kprobe_exceptions_notify+0x21d/0x5f8()
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.776009] Call Trace:
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff8104cd0c>] ? warn_slowpath_common+0x77/0xa3
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f59ab>] ? notifier_call_chain+0x29/0x4c
[  216.776009]  [<ffffffff81067444>] ? notify_die+0x2d/0x32
[  216.776009]  [<ffffffff812f3dbf>] ? do_int3+0x2f/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.776009]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.776009]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100cb88>] ? xen_mc_entry+0x2e/0x33
[  216.776009]  [<ffffffff8100cd08>] ? xen_flush_tlb_single+0x19/0x6d
[  216.776009]  [<ffffffff8100cdcb>] ? xen_set_pte+0x12/0x1b
[  216.776009]  [<ffffffff81030971>] ? set_pte_vaddr_pud+0x41/0x4b
[  216.776009]  [<ffffffff81034a46>] ? __native_set_fixmap+0x24/0x2c
[  216.776009]  [<ffffffff8100c9c5>] ? xen_set_fixmap+0x86/0xb5
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffff812f45ef>] ? text_poke+0x13d/0x19b
[  216.776009]  [<ffffffff812f5449>] ? arch_arm_kprobe+0x1c/0x21
[  216.776009]  [<ffffffff812f6b0d>] ? register_kprobe+0x452/0x4ad
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffffa0131575>] ? _stp_ctl_write_cmd+0x770/0x84e [stap_db2bea815fe5469cb8137996bcbc994c_6678582]
[  216.776009]  [<ffffffff810ed68e>] ? vfs_write+0xa9/0x102
[  216.776009]  [<ffffffff810ed7a3>] ? sys_write+0x45/0x6e
[  216.776009]  [<ffffffff81010e63>] ? stub_clone+0x13/0x20
[  216.776009]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[  216.776009] ---[ end trace 6100392551b72ae4 ]---
[  216.776009] int3: 0000 [#6] SMP 
[  216.776009] last sysfs file: /sys/module/xen_netfront/sections/__bug_table
[  216.776009] CPU 0 
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1 
[  216.776009] RIP: e030:[<ffffffff8100b0d7>]  [<ffffffff8100b0d7>] xen_read_cr0+0x1/0x1b
[  216.776009] RSP: e02b:ffff88003778ce10  EFLAGS: 00000002
[  216.776009] RAX: 0000000000000000 RBX: ffff88003778cf3c RCX: 0000000000000000
[  216.776009] RDX: 0000000000000000 RSI: ffff88003778cf3c RDI: 0000000000000000
[  216.776009] RBP: 00007f7f31eb5700 R08: 000000000000b2ed R09: 000000000000000a
[  216.776009] R10: 0000000000000000 R11: ffffffff811b4174 R12: ffffffff813c2cc1
[  216.776009] R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000000
[  216.776009] ------------[ cut here ]------------
[  216.776009] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:508 kprobe_exceptions_notify+0x21d/0x5f8()
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.776009] Call Trace:
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff8104cd0c>] ? warn_slowpath_common+0x77/0xa3
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f59ab>] ? notifier_call_chain+0x29/0x4c
[  216.776009]  [<ffffffff81067444>] ? notify_die+0x2d/0x32
[  216.776009]  [<ffffffff812f3dbf>] ? do_int3+0x2f/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.776009]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.776009]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100cb88>] ? xen_mc_entry+0x2e/0x33
[  216.776009]  [<ffffffff8100cd08>] ? xen_flush_tlb_single+0x19/0x6d
[  216.776009]  [<ffffffff8100cdcb>] ? xen_set_pte+0x12/0x1b
[  216.776009]  [<ffffffff81030971>] ? set_pte_vaddr_pud+0x41/0x4b
[  216.776009]  [<ffffffff81034a46>] ? __native_set_fixmap+0x24/0x2c
[  216.776009]  [<ffffffff8100c9c5>] ? xen_set_fixmap+0x86/0xb5
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffff812f45ef>] ? text_poke+0x13d/0x19b
[  216.776009]  [<ffffffff812f5449>] ? arch_arm_kprobe+0x1c/0x21
[  216.776009]  [<ffffffff812f6b0d>] ? register_kprobe+0x452/0x4ad
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffffa0131575>] ? _stp_ctl_write_cmd+0x770/0x84e [stap_db2bea815fe5469cb8137996bcbc994c_6678582]
[  216.776009]  [<ffffffff810ed68e>] ? vfs_write+0xa9/0x102
[  216.776009]  [<ffffffff810ed7a3>] ? sys_write+0x45/0x6e
[  216.776009]  [<ffffffff81010e63>] ? stub_clone+0x13/0x20
[  216.776009]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[  216.776009] ---[ end trace 6100392551b72ae5 ]---
[  216.776009] int3: 0000 [#7] SMP 
[  216.776009] last sysfs file: /sys/module/xen_netfront/sections/__bug_table
[  216.776009] CPU 0 
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1 
[  216.776009] RIP: e030:[<ffffffff8100b0d7>]  [<ffffffff8100b0d7>] xen_read_cr0+0x1/0x1b
[  216.776009] RSP: e02b:ffff88003778cb40  EFLAGS: 00000002
[  216.776009] RAX: 0000000000000000 RBX: ffff88003778cc6c RCX: 0000000000000000
[  216.776009] RDX: 0000000000000000 RSI: ffff88003778cc6c RDI: 0000000000000000
[  216.776009] RBP: 00007f7f31eb5700 R08: 000000000000cef7 R09: 000000000000000a
[  216.776009] R10: 0000000000000000 R11: ffffffff811b4174 R12: ffffffff813c2cc1
[  216.776009] R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000000
[  216.776009] ------------[ cut here ]------------
[  216.776009] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:508 kprobe_exceptions_notify+0x21d/0x5f8()
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.776009] Call Trace:
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff8104cd0c>] ? warn_slowpath_common+0x77/0xa3
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f59ab>] ? notifier_call_chain+0x29/0x4c
[  216.776009]  [<ffffffff81067444>] ? notify_die+0x2d/0x32
[  216.776009]  [<ffffffff812f3dbf>] ? do_int3+0x2f/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.776009]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.776009]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100cb88>] ? xen_mc_entry+0x2e/0x33
[  216.776009]  [<ffffffff8100cd08>] ? xen_flush_tlb_single+0x19/0x6d
[  216.776009]  [<ffffffff8100cdcb>] ? xen_set_pte+0x12/0x1b
[  216.776009]  [<ffffffff81030971>] ? set_pte_vaddr_pud+0x41/0x4b
[  216.776009]  [<ffffffff81034a46>] ? __native_set_fixmap+0x24/0x2c
[  216.776009]  [<ffffffff8100c9c5>] ? xen_set_fixmap+0x86/0xb5
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffff812f45ef>] ? text_poke+0x13d/0x19b
[  216.776009]  [<ffffffff812f5449>] ? arch_arm_kprobe+0x1c/0x21
[  216.776009]  [<ffffffff812f6b0d>] ? register_kprobe+0x452/0x4ad
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffffa0131575>] ? _stp_ctl_write_cmd+0x770/0x84e [stap_db2bea815fe5469cb8137996bcbc994c_6678582]
[  216.776009]  [<ffffffff810ed68e>] ? vfs_write+0xa9/0x102
[  216.776009]  [<ffffffff810ed7a3>] ? sys_write+0x45/0x6e
[  216.776009]  [<ffffffff81010e63>] ? stub_clone+0x13/0x20
[  216.776009]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[  216.776009] ---[ end trace 6100392551b72ae6 ]---
[  216.776009] BUG: scheduling while atomic: stapio/1936/0x1000006d
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.776009] Call Trace:
[  216.776009]  [<ffffffff81046fb1>] ? __schedule_bug+0x40/0x54
[  216.776009]  [<ffffffff812f1998>] ? schedule+0xc5/0x79f
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff812f1570>] ? dump_stack+0x69/0x6f
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff8104cd26>] ? warn_slowpath_common+0x91/0xa3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff810497a2>] ? __cond_resched+0x1d/0x26
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff812f2277>] ? _cond_resched+0x24/0x30
[  216.776009]  [<ffffffff8101340c>] ? is_valid_bugaddr+0x9/0x22
[  216.776009]  [<ffffffff8118b6a3>] ? report_bug+0xf/0xba
[  216.776009]  [<ffffffff81014253>] ? die+0x33/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.776009]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.776009]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100cb88>] ? xen_mc_entry+0x2e/0x33
[  216.776009]  [<ffffffff8100cd08>] ? xen_flush_tlb_single+0x19/0x6d
[  216.776009]  [<ffffffff8100cdcb>] ? xen_set_pte+0x12/0x1b
[  216.776009]  [<ffffffff81030971>] ? set_pte_vaddr_pud+0x41/0x4b
[  216.776009]  [<ffffffff81034a46>] ? __native_set_fixmap+0x24/0x2c
[  216.776009]  [<ffffffff8100c9c5>] ? xen_set_fixmap+0x86/0xb5
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffff812f45ef>] ? text_poke+0x13d/0x19b
[  216.776009]  [<ffffffff812f5449>] ? arch_arm_kprobe+0x1c/0x21
[  216.776009]  [<ffffffff812f6b0d>] ? register_kprobe+0x452/0x4ad
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffffa0131575>] ? _stp_ctl_write_cmd+0x770/0x84e [stap_db2bea815fe5469cb8137996bcbc994c_6678582]
[  216.776009]  [<ffffffff810ed68e>] ? vfs_write+0xa9/0x102
[  216.776009]  [<ffffffff810ed7a3>] ? sys_write+0x45/0x6e
[  216.776009]  [<ffffffff81010e63>] ? stub_clone+0x13/0x20
[  216.776009]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[  216.776009] ------------[ cut here ]------------
[  216.776009] WARNING: at /local/lindi/debian/linux-2.6-2.6.32/debian/build/source_amd64_none/kernel/sched_fair.c:979 hrtick_start_fair+0x3d/0x15b()
[  216.776009] Modules linked in: stap_db2bea815fe5469cb8137996bcbc994c_6678582 binfmt_misc nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp fuse loop evdev pcspkr ext3 jbd mbcache dm_mod xen_netfront xen_blkfront
[  216.776009] Pid: 1936, comm: stapio Tainted: G        W  2.6.32-5+lindi.5-amd64 #1
[  216.776009] Call Trace:
[  216.776009]  [<ffffffff81040895>] ? hrtick_start_fair+0x3d/0x15b
[  216.776009]  [<ffffffff81040895>] ? hrtick_start_fair+0x3d/0x15b
[  216.776009]  [<ffffffff8104cd0c>] ? warn_slowpath_common+0x77/0xa3
[  216.776009]  [<ffffffff81040895>] ? hrtick_start_fair+0x3d/0x15b
[  216.776009]  [<ffffffff81040a7d>] ? pick_next_task_fair+0xca/0xd6
[  216.776009]  [<ffffffff810398d8>] ? pick_next_task+0x21/0x3d
[  216.776009]  [<ffffffff812f1ebc>] ? schedule+0x5e9/0x79f
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff812f1570>] ? dump_stack+0x69/0x6f
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff812f5052>] ? kprobe_exceptions_notify+0x21d/0x5f8
[  216.776009]  [<ffffffff8104cd26>] ? warn_slowpath_common+0x91/0xa3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff810497a2>] ? __cond_resched+0x1d/0x26
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff812f2277>] ? _cond_resched+0x24/0x30
[  216.776009]  [<ffffffff8101340c>] ? is_valid_bugaddr+0x9/0x22
[  216.776009]  [<ffffffff8118b6a3>] ? report_bug+0xf/0xba
[  216.776009]  [<ffffffff81014253>] ? die+0x33/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff811b4174>] ? dummycon_dummy+0x0/0x3
[  216.776009]  [<ffffffff8100b0d7>] ? xen_read_cr0+0x1/0x1b
[  216.776009]  [<ffffffff8100fc72>] ? __show_regs+0x1e0/0x2ee
[  216.776009]  [<ffffffff8100e1df>] ? xen_restore_fl_direct_end+0x0/0x1
[  216.776009]  [<ffffffff8104d887>] ? vprintk+0x321/0x364
[  216.776009]  [<ffffffff812f1705>] ? printk+0x4e/0x59
[  216.776009]  [<ffffffff81013573>] ? show_registers+0x44/0x1f8
[  216.776009]  [<ffffffff812f4233>] ? __die+0x90/0xdb
[  216.776009]  [<ffffffff81014261>] ? die+0x41/0x5c
[  216.776009]  [<ffffffff812f3e04>] ? do_int3+0x74/0x98
[  216.776009]  [<ffffffff812f377b>] ? xen_int3+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100b0a9>] ? xen_get_debugreg+0x1/0x9
[  216.776009]  [<ffffffff812f3b38>] ? do_debug+0x25/0x13e
[  216.776009]  [<ffffffff812f375b>] ? xen_debug+0x1b/0x20
[  216.776009]  [<ffffffff8100c157>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[  216.776009]  [<ffffffff8100cb88>] ? xen_mc_entry+0x2e/0x33
[  216.776009]  [<ffffffff8100cd08>] ? xen_flush_tlb_single+0x19/0x6d
[  216.776009]  [<ffffffff8100cdcb>] ? xen_set_pte+0x12/0x1b
[  216.776009]  [<ffffffff81030971>] ? set_pte_vaddr_pud+0x41/0x4b
[  216.776009]  [<ffffffff81034a46>] ? __native_set_fixmap+0x24/0x2c
[  216.776009]  [<ffffffff8100c9c5>] ? xen_set_fixmap+0x86/0xb5
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffff812f45ef>] ? text_poke+0x13d/0x19b
[  216.776009]  [<ffffffff812f5449>] ? arch_arm_kprobe+0x1c/0x21
[  216.776009]  [<ffffffff812f6b0d>] ? register_kprobe+0x452/0x4ad
[  216.776009]  [<ffffffff8100c070>] ? __xen_mc_entry+0x0/0xb8
[  216.776009]  [<ffffffffa0131575>] ? _stp_ctl_write_cmd+0x770/0x84e [stap_db2bea815fe5469cb8137996bcbc994c_6678582]
[  216.776009]  [<ffffffff810ed68e>] ? vfs_write+0xa9/0x102
[  216.776009]  [<ffffffff810ed7a3>] ? sys_write+0x45/0x6e
[  216.776009]  [<ffffffff81010e63>] ? stub_clone+0x13/0x20
[  216.776009]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[  216.776009] ---[ end trace 6100392551b72ae7 ]---

3) crash shows

please wait... (gathering task table data)
WARNING: active task ffff88003ee04d00 on cpu 0: corrupt cpu value: 4294967295

please wait... (determining panic task)
WARNING: multiple active tasks have called die

foreach: inconsistent active task indications for CPU 2:
             runqueue: ffff88003fe38700 "events/2" (default)
         current_task: ffff88003fdc1500 "swapper"

foreach: inconsistent active task indications for CPU 3:
             runqueue: ffff880037133f00 "nagios3" (default)
         current_task: ffff88003fdc1c00 "swapper"

      KERNEL: /usr/lib/debug/boot/vmlinux-2.6.32-5+lindi.5-amd64
    DUMPFILE: systemtap-wildcard1.dump
        CPUS: 4
        DATE: Mon Nov 22 12:04:30 2010
      UPTIME: 00:03:39
LOAD AVERAGE: 0.20, 0.07, 0.02
       TASKS: 206
    NODENAME: lindi1
     RELEASE: 2.6.32-5+lindi.5-amd64
     VERSION: #1 SMP Tue Nov 9 12:54:48 EET 2010
     MACHINE: x86_64  (2394 Mhz)
      MEMORY: 1 GB
       PANIC: ""
         PID: 1936
     COMMAND: "stapio"
        TASK: ffff88003ee04d00  [THREAD_INFO: ffff88003778c000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)



PID: 1936   TASK: ffff88003ee04d00  CPU: 0   COMMAND: "stapio"
 #0 [ffff88003778d878] schedule at ffffffff812f2072
 #1 [ffff88003778d880] do_int3 at ffffffff812f3e04
 #2 [ffff88003778d8a0] xen_int3 at ffffffff812f377b
    [exception RIP: xen_read_cr0+1]
    RIP: ffffffff8100b0d7  RSP: ffff88003778d950  RFLAGS: 00000002
    RAX: 0000000000000000  RBX: ffff88003778da7c  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffff88003778da7c  RDI: 0000000000000000
    RBP: 00007f7f31eb5700   R8: 0000000000004bde   R9: 000000000000000a
    R10: 0000000000000000  R11: ffffffff811b4174  R12: ffffffff813c2cc1
    R13: 0000000000000002  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 10000e030  SS: e02b
 #3 [ffff88003778d950] __show_regs at ffffffff8100fc72
 #4 [ffff88003778dab0] show_registers at ffffffff81013573
 #5 [ffff88003778db00] __die at ffffffff812f4233
 #6 [ffff88003778db20] die at ffffffff81014261
 #7 [ffff88003778db50] do_int3 at ffffffff812f3e04
 #8 [ffff88003778db70] xen_int3 at ffffffff812f377b
    [exception RIP: xen_get_debugreg+1]
    RIP: ffffffff8100b0a9  RSP: ffff88003778dc20  RFLAGS: 00000086
    RAX: 000000000000c570  RBX: 0000000000000001  RCX: 000000008101154e
    RDX: ffff8800020a0000  RSI: 0000000000000000  RDI: 0000000000000006
    RBP: ffff88003778dc68   R8: 0000000000000001   R9: 0000000000000040
    R10: ffffffff8100c157  R11: 0000000000000040  R12: ffff88003ee04d00
    R13: 00000000000302d8  R14: 0000000000000000  R15: 0000160000000000
    ORIG_RAX: ffffffffffffffff  CS: 10000e030  SS: e02b
 #9 [ffff88003778dc20] do_debug at ffffffff812f3b38
#10 [ffff88003778dc60] xen_debug at ffffffff812f375b
#11 [ffff88003778dd20] xen_mc_entry at ffffffff8100cb88


-- System Information:
Debian Release: squeeze/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5+lindi.5-amd64 (SMP w/4 CPU cores)
Locale: LANG=C, LC_CTYPE=fi_FI (charmap=ISO-8859-1)
Shell: /bin/sh linked to /bin/bash




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#604453; Package linux-2.6. (Mon, 22 Nov 2010 11:27:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Mon, 22 Nov 2010 11:27:03 GMT) Full text and rfc822 format available.

Message #10 received at 604453@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: Timo Juhani Lindfors <timo.lindfors@iki.fi>, 604453@bugs.debian.org
Subject: Re: Bug#604453: linux-2.6: xen domU hangs with systemtap kernel.function("*@*/*.c")
Date: Mon, 22 Nov 2010 11:23:25 +0000
[Message part 1 (text/plain, inline)]
On Mon, 2010-11-22 at 12:46 +0200, Timo Juhani Lindfors wrote:
> Package: linux-2.6
> Version: 2.6.32-27
> Severity: minor
> 
> [ Note that I have applied a patch from #602273 so this is not a
> completely unmodified linux-2.6 package.]
[...]

Also, SystemTap inserts modules into the kernel, so my first thought is
to suspect SystemTap.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
[signature.asc (application/pgp-signature, inline)]

Bug reassigned from package 'linux-2.6' to 'systemtap'. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Mon, 22 Nov 2010 11:27:05 GMT) Full text and rfc822 format available.

Bug No longer marked as found in versions 2.6.32-27. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Mon, 22 Nov 2010 11:27:05 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Mon, 22 Nov 2010 16:18:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Frank Ch. Eigler" <fche@redhat.com>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Mon, 22 Nov 2010 16:18:03 GMT) Full text and rfc822 format available.

Message #19 received at 604453@bugs.debian.org (full text, mbox):

From: "Frank Ch. Eigler" <fche@redhat.com>
To: 604453@bugs.debian.org
Cc: systemtap@sourceware.org
Subject: stap hangs with broad wildcards
Date: Mon, 22 Nov 2010 11:15:11 -0500
This problem sounds like <http://sourceware.org/PR2725>, and usually
represents a latent kernel problem (in the kprobes or related APIs)
rather than a bug in systemtap.

With newer kernels, the "perf probe" interface can be used to set
kprobes at the same place where systemtap would place them, and the
same sorts of hangs/crashes tend to occur.  With older kernels such
as 2.6.32, some more work is needed to test a systemtap-free
configuration:

  git clone git://sourceware.org/git/systemtap.git
  cd systemtap/scripts/kprobes_test
  sh gen_code_all.sh
  insmod kprobe_module.ko
  <bang>

- FChE




Set Bug forwarded-to-address to 'http://sourceware.org/bugzilla/show_bug.cgi?id=2725'. Request was from Ritesh Raj Sarraf <rrs@debian.org> to control@bugs.debian.org. (Mon, 22 Nov 2010 19:51:08 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Tue, 23 Nov 2010 00:03:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Tue, 23 Nov 2010 00:03:04 GMT) Full text and rfc822 format available.

Message #26 received at 604453@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: "Frank Ch. Eigler" <fche@redhat.com>
Cc: Ritesh Raj Sarraf <rrs@debian.org>, 604453@bugs.debian.org
Subject: Re: Processed: reassign 604453 to systemtap
Date: Mon, 22 Nov 2010 23:59:22 +0000
[Message part 1 (text/plain, inline)]
On Mon, 2010-11-22 at 08:26 -0500, Frank Ch. Eigler wrote:
> > Bug #604453 [linux-2.6] linux-2.6: xen domU hangs with systemtap kernel.function("*@*/*.c")
> > Bug reassigned from package 'linux-2.6' to 'systemtap'.
> > Bug No longer marked as found in versions 2.6.32-27.
> 
> This is a known problem, also known as http://sourceware.org/PR2725.
> They appear to be latent kernel (kprobes layer?) bugs, as indicated by
> "perf probe ..." (or git systemtap's -DSTP_ALIBI mode) also causing
> similar kernel failures when probing large subsets of the kernel.

So do you think I should assign this back to the kernel?  I note the bug
you reference remains assigned to systemtap.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Tue, 23 Nov 2010 01:27:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Frank Ch. Eigler" <fche@redhat.com>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Tue, 23 Nov 2010 01:27:03 GMT) Full text and rfc822 format available.

Message #31 received at 604453@bugs.debian.org (full text, mbox):

From: "Frank Ch. Eigler" <fche@redhat.com>
To: Ben Hutchings <ben@decadent.org.uk>
Cc: Ritesh Raj Sarraf <rrs@debian.org>, 604453@bugs.debian.org
Subject: Re: Processed: reassign 604453 to systemtap
Date: Mon, 22 Nov 2010 20:24:11 -0500
Hi -

On Mon, Nov 22, 2010 at 11:59:22PM +0000, Ben Hutchings wrote:
> [...]
> > This is a known problem, also known as http://sourceware.org/PR2725.
> > [...]

> So do you think I should assign this back to the kernel? 

I guess it depends on whether you folks are likely to have time to
improve the kernel side of the situation.

On the systemtap side, we only have a blacklist of some known-bad
areas (that the kernel is not known to put into its own kprobes
blacklist).  We add entries to that list periodically as testing
identifies specific vulnerable areas.  It is an available workaround,
but it is by definition incomplete, and can only protect systemtap
users.

> I note the bug you reference remains assigned to systemtap.

(We track some known non-systemtap problems in that same database.)


- FChE




Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Tue, 23 Nov 2010 01:48:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Tue, 23 Nov 2010 01:48:03 GMT) Full text and rfc822 format available.

Message #36 received at 604453@bugs.debian.org (full text, mbox):

From: Ben Hutchings <ben@decadent.org.uk>
To: "Frank Ch. Eigler" <fche@redhat.com>
Cc: Ritesh Raj Sarraf <rrs@debian.org>, 604453@bugs.debian.org
Subject: Re: Processed: reassign 604453 to systemtap
Date: Tue, 23 Nov 2010 01:45:04 +0000
[Message part 1 (text/plain, inline)]
On Mon, 2010-11-22 at 20:24 -0500, Frank Ch. Eigler wrote:
> Hi -
> 
> On Mon, Nov 22, 2010 at 11:59:22PM +0000, Ben Hutchings wrote:
> > [...]
> > > This is a known problem, also known as http://sourceware.org/PR2725.
> > > [...]
> 
> > So do you think I should assign this back to the kernel? 
> 
> I guess it depends on whether you folks are likely to have time to
> improve the kernel side of the situation.
[...]

We're happy to apply/backport bug fixes that are accepted upstream.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Wed, 10 Aug 2011 18:39:08 GMT) Full text and rfc822 format available.

Acknowledgement sent to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Wed, 10 Aug 2011 18:39:08 GMT) Full text and rfc822 format available.

Message #41 received at 604453@bugs.debian.org (full text, mbox):

From: Timo Juhani Lindfors <timo.lindfors@iki.fi>
To: 604453@bugs.debian.org
Cc: "Frank Ch. Eigler" <fche@redhat.com>, Ritesh Raj Sarraf <rrs@debian.org>, 604453-submitter@bugs.debian.org, Ben Hutchings <ben@decadent.org.uk>
Subject: Re: bug #604453: linux-2.6: xen domU hangs with systemtap kernel.function("*@*/*.c")
Date: Wed, 10 Aug 2011 21:35:11 +0300
Hi,

just an update: the bug is still fully reproducible with

systemtap       1.6-1
linux-image-3.0.0-1-amd64       3.0.0-1

under xen:

[   59.871088] Disabling lock debugging due to kernel taint
[   59.911184] Unrecoverable kprobe detected at ffffffff81003127.
[   59.911200] Dumping kprobe:
[   59.911203] Name: (null)
[   59.911203] Address: ffffffff81003127
[   59.911204] Offset: 0
[   59.911212] ------------[ cut here ]------------
[   59.911216] kernel BUG at /tmp/buildd/linux-2.6-3.0.0/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:523!
[   59.911221] invalid opcode: 0000 [#1] SMP 
[   59.911226] CPU 0 
[   59.911227] Modules linked in: stap_38863ea43e74c670f3a28dfd93ac8e7f_76_1646(O) fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc loop evdev snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr ext3 jbd mbcache dm_mod virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront
[   59.911262] 
[   59.911266] Pid: 1646, comm: stapio Tainted: G           O 3.0.0-1-amd64 #1  
[   59.911271] RIP: e030:[<ffffffff81338a03>]  [<ffffffff81338a03>] kprobe_exceptions_notify+0xbf/0x3ad
[   59.911280] RSP: e02b:ffff88003c51ba78  EFLAGS: 00010092
[   59.911283] RAX: 000000000000006a RBX: ffff88003c51bb58 RCX: 000000000000021d
[   59.911287] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
[   59.911291] RBP: ffffffffa0a7ed70 R08: 0000000000000000 R09: 0720072007200720
[   59.911294] R10: 0720072007200720 R11: 0720072007200720 R12: ffff88003fd6ae60
[   59.911298] R13: ffffffff81003127 R14: ffffffff81003128 R15: 6db6db6db6db6db7
[   59.911304] Unrecoverable kprobe detected at ffffffff810032a0.
[   59.911307] Dumping kprobe:
[   59.911309] Name: (null)
[   59.911310] Address: ffffffff810032a0
[   59.911311] Offset: 0
[   59.911315] ------------[ cut here ]------------
[   59.911319] kernel BUG at /tmp/buildd/linux-2.6-3.0.0/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:523!
[   59.911323] invalid opcode: 0000 [#2] SMP 
[   59.911327] CPU 0 
[   59.911329] Modules linked in: stap_38863ea43e74c670f3a28dfd93ac8e7f_76_1646(O) fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc loop evdev snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr ext3 jbd mbcache dm_mod virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront
[   59.911361] 
[   59.911363] Pid: 1646, comm: stapio Tainted: G           O 3.0.0-1-amd64 #1  
[   59.911368] RIP: e030:[<ffffffff81338a03>]  [<ffffffff81338a03>] kprobe_exceptions_notify+0xbf/0x3ad
[   59.911375] RSP: e02b:ffff88003c51b658  EFLAGS: 00010096
[   59.911378] RAX: 000000000000006a RBX: ffff88003c51b738 RCX: 0000000000000289
[   59.911381] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
[   59.911385] RBP: ffff88003cc218c0 R08: 0000000000000000 R09: 0720072007200720
[   59.911389] R10: 0720072007200720 R11: 0720072007200720 R12: ffff88003fd6ae60
[   59.911393] R13: ffffffff810032a0 R14: ffffffff810032a1 R15: 6db6db6db6db6db7
[   59.911398] Unrecoverable kprobe detected at ffffffff810032a0.
[   59.911401] Dumping kprobe:
[   59.911403] Name: (null)
[   59.911404] Address: ffffffff810032a0
[   59.911404] Offset: 0
[   59.911409] ------------[ cut here ]------------
[   59.911413] kernel BUG at /tmp/buildd/linux-2.6-3.0.0/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:523!
[   59.911417] invalid opcode: 0000 [#3] SMP 
[   59.911421] CPU 0 
[   59.911423] Modules linked in: stap_38863ea43e74c670f3a28dfd93ac8e7f_76_1646(O) fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc loop evdev snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr ext3 jbd mbcache dm_mod virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront
[   59.911454] 
[   59.911457] Pid: 1646, comm: stapio Tainted: G           O 3.0.0-1-amd64 #1  
[   59.911462] RIP: e030:[<ffffffff81338a03>]  [<ffffffff81338a03>] kprobe_exceptions_notify+0xbf/0x3ad
[   59.911468] RSP: e02b:ffff88003c51b238  EFLAGS: 00010096
[   59.911471] RAX: 000000000000006a RBX: ffff88003c51b318 RCX: 00000000000002f5
[   59.911475] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
[   59.911479] RBP: ffff88003cc218c0 R08: 0000000000000000 R09: 0720072007200720
[   59.911483] R10: 0720072007200720 R11: 0720072007200720 R12: ffff88003fd6ae60
[   59.911486] R13: ffffffff810032a0 R14: ffffffff810032a1 R15: 6db6db6db6db6db7
[   59.911491] Unrecoverable kprobe detected at ffffffff810032a0.
[   59.911494] Dumping kprobe:
[   59.911496] Name: (null)
[   59.911496] Address: ffffffff810032a0
[   59.911497] Offset: 0
[   59.911502] ------------[ cut here ]------------
[   59.911505] kernel BUG at /tmp/buildd/linux-2.6-3.0.0/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:523!
[   59.911510] invalid opcode: 0000 [#4] SMP 
[   59.911514] CPU 0 
[   59.911515] Modules linked in: stap_38863ea43e74c670f3a28dfd93ac8e7f_76_1646(O) fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc loop evdev snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr ext3 jbd mbcache dm_mod virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront
[   59.911547] 
[   59.911549] Pid: 1646, comm: stapio Tainted: G           O 3.0.0-1-amd64 #1  
[   59.911554] RIP: e030:[<ffffffff81338a03>]  [<ffffffff81338a03>] kprobe_exceptions_notify+0xbf/0x3ad
[   59.911560] RSP: e02b:ffff88003c51ae18  EFLAGS: 00010092
[   59.911564] RAX: 000000000000006a RBX: ffff88003c51aef8 RCX: 0000000000000361
[   59.911567] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
[   59.911571] RBP: ffff88003cc218c0 R08: 0000000000000000 R09: 0720072007200720
[   59.911575] R10: 0720072007200720 R11: 0720072007200720 R12: ffff88003fd6ae60
[   59.911579] R13: ffffffff810032a0 R14: ffffffff810032a1 R15: 6db6db6db6db6db7
[   59.911583] Unrecoverable kprobe detected at ffffffff810032a0.
[   59.911586] Dumping kprobe:
[   59.911588] Name: (null)
[   59.911589] Address: ffffffff810032a0
[   59.911589] Offset: 0
[   59.911594] ------------[ cut here ]------------
[   59.911597] kernel BUG at /tmp/buildd/linux-2.6-3.0.0/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:523!
[   59.911602] invalid opcode: 0000 [#5] SMP 
[   59.911606] CPU 0 
[   59.911607] Modules linked in: stap_38863ea43e74c670f3a28dfd93ac8e7f_76_1646(O) fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc loop evdev snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr ext3 jbd mbcache dm_mod virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront
[   59.911639] 
[   59.911641] Pid: 1646, comm: stapio Tainted: G           O 3.0.0-1-amd64 #1  
[   59.911646] RIP: e030:[<ffffffff81338a03>]  [<ffffffff81338a03>] kprobe_exceptions_notify+0xbf/0x3ad
[   59.911652] RSP: e02b:ffff88003c51a9f8  EFLAGS: 00010096
[   59.911655] RAX: 000000000000006a RBX: ffff88003c51aad8 RCX: 00000000000003cd
[   59.911659] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
[   59.911662] RBP: ffff88003cc218c0 R08: 0000000000000000 R09: 0720072007200720
[   59.911666] R10: 0720072007200720 R11: 0720072007200720 R12: ffff88003fd6ae60
[   59.911670] R13: ffffffff810032a0 R14: ffffffff810032a1 R15: 6db6db6db6db6db7
[   59.911674] Unrecoverable kprobe detected at ffffffff810032a0.
[   59.911677] Dumping kprobe:
[   59.911679] Name: (null)
[   59.911680] Address: ffffffff810032a0
[   59.911680] Offset: 0
[   59.911685] ------------[ cut here ]------------
[   59.911689] kernel BUG at /tmp/buildd/linux-2.6-3.0.0/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:523!
[   59.911696] invalid opcode: 0000 [#6] SMP 
[   59.911700] CPU 0 
[   59.911702] Modules linked in: stap_38863ea43e74c670f3a28dfd93ac8e7f_76_1646(O) fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc loop evdev snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr ext3 jbd mbcache dm_mod virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront
[   59.911733] 
[   59.911735] Pid: 1646, comm: stapio Tainted: G           O 3.0.0-1-amd64 #1  
[   59.911740] RIP: e030:[<ffffffff81338a03>]  [<ffffffff81338a03>] kprobe_exceptions_notify+0xbf/0x3ad
[   59.911746] RSP: e02b:ffff88003c51a5d8  EFLAGS: 00010092
[   59.911749] RAX: 000000000000006a RBX: ffff88003c51a6b8 RCX: 0000000000000439
[   59.911753] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
[   60.008062] Unrecoverable kprobe detected at ffffffff81003127.
[   60.008067] Unrecoverable kprobe detected at ffffffff81003127.
[   60.008072] Unrecoverable kprobe detected at ffffffff81003127.
[   60.008075] Dumping kprobe:
[   60.008077] Dumping kprobe:
[   60.008080] Dumping kprobe:
[   60.008082] Name: (null)
[   60.008084] Address: ffffffff81003127
[   60.008085] Offset: 0
[   60.008087] Name: (null)
[   60.008089] Address: ffffffff81003127
[   60.008090] Offset: 0
[   60.008092] Name: (null)
[   60.008093] Address: ffffffff81003127
[   60.008094] Offset: 0
[   59.912022] RBP: ffff88003cc218c0 R08: 0000000000000000 R09: 0720072007200720
[   59.912022] R10: 0720072007200720 R11: 0720072007200720 R12: ffff88003fd6ae60
[   59.912022] R13: ffffffff810032a0 R14: ffffffff810032a1 R15: 6db6db6db6db6db7
[   59.912022] Unrecoverable kprobe detected at ffffffff810032a0.
[   59.912022] Dumping kprobe:
[   59.912022] Name: (null)
[   59.912022] Address: ffffffff810032a0
[   59.912022] Offset: 0
[   59.912022] ------------[ cut here ]------------
[   59.912022] kernel BUG at /tmp/buildd/linux-2.6-3.0.0/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:523!
[   59.912022] invalid opcode: 0000 [#7] SMP 
[   59.912022] CPU 0 
[   59.912022] Modules linked in: stap_38863ea43e74c670f3a28dfd93ac8e7f_76_1646(O) fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc loop evdev snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr ext3 jbd mbcache dm_mod virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront
[   59.912022] 
[   59.912022] Pid: 1646, comm: stapio Tainted: G           O 3.0.0-1-amd64 #1  
[   59.912022] RIP: e030:[<ffffffff81338a03>]  [<ffffffff81338a03>] kprobe_exceptions_notify+0xbf/0x3ad
[   59.912022] RSP: e02b:ffff88003c51a1b8  EFLAGS: 00010092
[   59.912022] RAX: 000000000000006a RBX: ffff88003c51a298 RCX: 00000000000004ae
[   59.912022] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
[   59.912022] RBP: ffff88003cc218c0 R08: 0000000000000000 R09: 0720072007200720
[   59.912022] R10: 0720072007200720 R11: 0720072007200720 R12: ffff88003fd6ae60
[   59.912022] R13: ffffffff810032a0 R14: ffffffff810032a1 R15: 6db6db6db6db6db7
[   59.912022] Unrecoverable kprobe detected at ffffffff810032a0.
[   59.912022] Dumping kprobe:
[   59.912022] Name: (null)
[   59.912022] Address: ffffffff810032a0
[   59.912022] Offset: 0
[   59.912022] ------------[ cut here ]------------
[   59.912022] kernel BUG at /tmp/buildd/linux-2.6-3.0.0/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:523!
[   59.912022] invalid opcode: 0000 [#8] SMP 
[   59.912022] CPU 0 
[   59.912022] Modules linked in: stap_38863ea43e74c670f3a28dfd93ac8e7f_76_1646(O) fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc loop evdev snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr ext3 jbd mbcache dm_mod virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront
[   59.912022] 
[   59.912022] Pid: 1646, comm: stapio Tainted: G           O 3.0.0-1-amd64 #1  
[   59.912022] RIP: e030:[<ffffffff81338a03>]  [<ffffffff81338a03>] kprobe_exceptions_notify+0xbf/0x3ad
[   59.912022] RSP: e02b:ffff88003c519d98  EFLAGS: 00010096
[   59.912022] RAX: 000000000000006a RBX: ffff88003c519e78 RCX: 000000000000051a
[   59.912022] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
[   59.912022] RBP: ffff88003cc218c0 R08: 0000000000000000 R09: 0720072007200720
[   59.912022] R10: 0720072007200720 R11: 0720072007200720 R12: ffff88003fd6ae60
[   59.912022] R13: ffffffff810032a0 R14: ffffffff810032a1 R15: 6db6db6db6db6db7
[   59.912022] Unrecoverable kprobe detected at ffffffff810032a0.
[   59.912022] Dumping kprobe:
[   59.912022] Name: (null)
[   59.912022] Address: ffffffff810032a0
[   59.912022] Offset: 0
[   59.912022] ------------[ cut here ]------------
[   59.912022] kernel BUG at /tmp/buildd/linux-2.6-3.0.0/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:523!
[   59.912022] invalid opcode: 0000 [#9] SMP 
[   59.912022] CPU 0 
[   59.912022] Modules linked in: stap_38863ea43e74c670f3a28dfd93ac8e7f_76_1646(O) fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc loop evdev snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr ext3 jbd mbcache dm_mod virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront
[   59.912022] 
[   59.912022] Pid: 1646, comm: stapio Tainted: G           O 3.0.0-1-amd64 #1  
[   59.912022] RIP: e030:[<ffffffff81338a03>]  [<ffffffff81338a03>] kprobe_exceptions_notify+0xbf/0x3ad
[   59.912022] RSP: e02b:ffff88003c519978  EFLAGS: 00010092
[   59.912022] RAX: 000000000000006a RBX: ffff88003c519a58 RCX: 0000000000000586
[   59.912022] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000004
[   59.912022] RBP: ffff88003cc218c0 R08: 0000000000000000 R09: 0720072007200720
[   59.912022] R10: 0720072007200720 R11: 0720072007200720 R12: ffff88003fd6ae60
[   59.912022] R13: ffffffff810032a0 R14: ffffffff810032a1 R15: 6db6db6db6db6db7
[   59.912022] Unrecoverable kprobe detected at ffffffff810032a0.
[   59.912022] Dumping kprobe:
[   59.912022] Name: (null)
[   59.912022] Address: ffffffff810032a0
[   59.912022] Offset: 0
[   59.912022] ------------[ cut here ]------------
[   59.912022] kernel BUG at /tmp/buildd/linux-2.6-3.0.0/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:523!
[   59.912022] invalid opcode: 0000 [#10] SMP 
[   59.912022] CPU 0 
[   59.912022] Modules linked in: stap_38863ea43e74c670f3a28dfd93ac8e7f_76_1646(O) fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc loop evdev snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr ext3 jbd mbcache dm_mod virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront


-Timo




Message sent on to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Bug#604453. (Wed, 10 Aug 2011 18:39:18 GMT) Full text and rfc822 format available.

Changed Bug title to 'reproducible way to crash the system with kernel.function("*@*/*.c")' from 'linux-2.6: xen domU hangs with systemtap kernel.function("*@*/*.c")' Request was from Timo Juhani Lindfors <timo.lindfors@iki.fi> to control@bugs.debian.org. (Thu, 11 Aug 2011 12:01:09 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Thu, 11 Aug 2011 13:12:22 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Frank Ch. Eigler" <fche@redhat.com>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Thu, 11 Aug 2011 13:12:24 GMT) Full text and rfc822 format available.

Message #51 received at 604453@bugs.debian.org (full text, mbox):

From: "Frank Ch. Eigler" <fche@redhat.com>
To: 604453@bugs.debian.org
Subject: Re: linux-2.6: xen domU hangs with systemtap kernel.function("*@*/*.c")
Date: Thu, 11 Aug 2011 09:09:52 -0400
Hi, Timo -

We have generally found that such problems reflect bugs in the kernel
kprobes implementation rather than in systemtap.  For one possible
alibi procedure, see https://bugzilla.redhat.com/show_bug.cgi?id=655904

- FChE




Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Fri, 12 Aug 2011 11:56:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Fri, 12 Aug 2011 11:56:11 GMT) Full text and rfc822 format available.

Message #56 received at 604453@bugs.debian.org (full text, mbox):

From: Timo Juhani Lindfors <timo.lindfors@iki.fi>
To: "Frank Ch. Eigler" <fche@redhat.com>
Cc: 604453@bugs.debian.org, 604453-submitter@bugs.debian.org
Subject: Re: linux-2.6: xen domU hangs with systemtap kernel.function("*@*/*.c")
Date: Fri, 12 Aug 2011 14:14:01 +0300
Hi Frank,

> alibi procedure, see https://bugzilla.redhat.com/show_bug.cgi?id=655904

Thanks. I can confirm that these instructions crash

linux-image-3.0.0-1-amd64  3.0.0-1

at least under xen. The README in the same directory talks about
generating a whitelist of safe probe points. I 

1) commented out

-untested "$TESTNAME is disabled"
-return
+#untested "$TESTNAME is disabled"
+#return

2) removed probe points that don't seem to be found anymore:

         kprocess.*, 
-        process.*, 
         nfs.fop.*, 
         nfs.aop.*,
         nfsd.proc.*, 
-        nfsd.*, 
         nfs.proc.*, 


3) fixed rc.local path for debian:

 set startup_line_in_RCLOCAL "cd $env(PWD); runtest whitelist.exp&"
-set RCLOCAL "/etc/rc.d/rc.local"
+set RCLOCAL "/etc/rc.local"
 

4) and finally had to comment out a check that I don't fully understand:

-if {[info procs installtest_p] != "" && ![installtest_p]} { 
-    untested $TESTNAME; 
-    return 
-}
+#if {[info procs installtest_p] != "" && ![installtest_p]} { 
+#    untested $TESTNAME; 
+#    return 
+#}


After this I get the following output:



$ sudo runtest whitelist.exp
WARNING: No tool specified
Test Run By root on Fri Aug 12 14:04:11 2011
Native configuration is x86_64-unknown-linux-gnu

                     ===  tests ===

Schedule of variations:
    unix

Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target.
Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
Using ./config/unix.exp as tool-and-target-specific interface file.
kernel location: 
kernel version: 3.0.0-1-amd64
systemtap location: /usr/bin/stap
systemtap version: version 1.6/0.152 non-git sources
gcc location: /usr/bin/gcc
gcc version: gcc (Debian 4.6.1-4) 4.6.1

Host: Linux lindi2 3.0.0-1-amd64 #1 SMP Sun Jul 24 02:24:44 UTC 2011 x86_64 GNU/Linux
Snapshot: version 1.6/0.152 non-git sources
GCC: 4.6.1 [gcc (Debian 4.6.1-4) 4.6.1]
Distro: Debian GNU/Linux testing (wheezy)

Running ./systemtap.stress/whitelist.exp ...
Start a fresh stp_genwhitelist test.
current_size_const is initialized as 107
Current size of probes.pending is 963
Start a probe test...
ERROR no match in probe_ok() L1_Grp_kernel.function("idle_balance@/tm... no match
Completed one probe test.
Current size of probes.pending is 856
Start a probe test...
ERROR no match in probe_ok() L1_Grp_kernel.function("compat_sys_newfs... no match
Completed one probe test.
Current size of probes.pending is 749
Start a probe test...
ERROR no match in probe_ok() L1_Grp_kernel.function("sys_mprotect@/tm... no match
Completed one probe test.
Current size of probes.pending is 642
Start a probe test...
ERROR no match in probe_ok() L1_Grp_kernel.function("sys_setpriority@... no match
Completed one probe test.
Current size of probes.pending is 535
Start a probe test...
ERROR no match in probe_ok() L1_Grp_kernel.function("sys_chown@/tmp/b... no match
Completed one probe test.
Current size of probes.pending is 428
Start a probe test...
ERROR no match in probe_ok() L1_Grp_kernel.function("sys_getpgrp@/tmp... no match
Completed one probe test.
Current size of probes.pending is 321
Start a probe test...
ERROR no match in probe_ok() L1_Grp_kernel.function("sys_openat@/tmp/... no match
Completed one probe test.
Current size of probes.pending is 214
Start a probe test...
ERROR no match in probe_ok() L1_Grp_kernel.function("sys_signalfd4@/t... no match
Completed one probe test.
Current size of probes.pending is 107
Start a probe test...
ERROR no match in probe_ok() L1_Grp_kernel.function("do_sigaction@/tm... no match
Completed one probe test.
Current size of probes.pending is 0
Running level increased to 2
Current running level is 2
Append probes.untriggered to probes.pending
current_size_const is set as 19 now
Current size of probes.pending is 963
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("idle_balance@/tm... no match
Completed one probe test.
Current size of probes.pending is 944
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_accept4@/tmp... no match
Completed one probe test.
Current size of probes.pending is 925
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_clock_gettim... no match
Completed one probe test.
Current size of probes.pending is 906
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("compat_sys_msgct... no match
Completed one probe test.
Current size of probes.pending is 887
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_epoll_create... no match
Completed one probe test.
Current size of probes.pending is 868
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("compat_sys_fcntl... no match
Completed one probe test.
Current size of probes.pending is 849
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_fsync@/tmp/b... no match
Completed one probe test.
Current size of probes.pending is 830
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_gethostname@... no match
Completed one probe test.
Current size of probes.pending is 811
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_getsockopt@/... no match
Completed one probe test.
Current size of probes.pending is 792
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_ioperm@/tmp/... no match
Completed one probe test.
Current size of probes.pending is 773
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_lremovexattr... no match
Completed one probe test.
Current size of probes.pending is 754
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_modify_ldt@/... no match
Completed one probe test.
Current size of probes.pending is 735
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_msgget@/tmp/... no match
Completed one probe test.
Current size of probes.pending is 716
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_pivot_root@/... no match
Completed one probe test.
Current size of probes.pending is 697
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_recvmmsg@/tm... no match
Completed one probe test.
Current size of probes.pending is 678
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_rt_sigtimedw... no match
Completed one probe test.
Current size of probes.pending is 659
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_sendmsg@/tmp... no match
Completed one probe test.
Current size of probes.pending is 640
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_setregid16@/... no match
Completed one probe test.
Current size of probes.pending is 621
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_shmdt@/tmp/b... no match
Completed one probe test.
Current size of probes.pending is 602
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("compat_sys_statf... no match
Completed one probe test.
Current size of probes.pending is 583
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("compat_sys_time@... no match
Completed one probe test.
Current size of probes.pending is 564
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_uselib@/tmp/... no match
Completed one probe test.
Current size of probes.pending is 545
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys32_alarm@/tmp... no match
Completed one probe test.
Current size of probes.pending is 526
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("compat_sys_clock... no match
Completed one probe test.
Current size of probes.pending is 507
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("compat_sys_recvm... no match
Completed one probe test.
Current size of probes.pending is 488
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_epoll_create... no match
Completed one probe test.
Current size of probes.pending is 469
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_fcntl@/tmp/b... no match
Completed one probe test.
Current size of probes.pending is 450
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_ftruncate@/t... no match
Completed one probe test.
Current size of probes.pending is 431
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_getitimer@/t... no match
Completed one probe test.
Current size of probes.pending is 412
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_gettid@/tmp/... no match
Completed one probe test.
Current size of probes.pending is 393
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_ioperm@/tmp/... no match
Completed one probe test.
Current size of probes.pending is 374
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_lremovexattr... no match
Completed one probe test.
Current size of probes.pending is 355
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_modify_ldt@/... no match
Completed one probe test.
Current size of probes.pending is 336
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_msgget@/tmp/... no match
Completed one probe test.
Current size of probes.pending is 317
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_pipe2@/tmp/b... no match
Completed one probe test.
Current size of probes.pending is 298
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_recv@/tmp/bu... no match
Completed one probe test.
Current size of probes.pending is 279
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_rt_sigsuspen... no match
Completed one probe test.
Current size of probes.pending is 260
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_sendfile@/tm... no match
Completed one probe test.
Current size of probes.pending is 241
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_setpriority@... no match
Completed one probe test.
Current size of probes.pending is 222
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_shmat@/tmp/b... no match
Completed one probe test.
Current size of probes.pending is 203
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_newstat@/tmp... no match
Completed one probe test.
Current size of probes.pending is 184
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_syslog@/tmp/... no match
Completed one probe test.
Current size of probes.pending is 165
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sys_unlink@/tmp/... no match
Completed one probe test.
Current size of probes.pending is 146
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("generic_make_req... no match
Completed one probe test.
Current size of probes.pending is 127
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("kmem_cache_alloc... no match
Completed one probe test.
Current size of probes.pending is 108
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("check_kill_permi... no match
Completed one probe test.
Current size of probes.pending is 89
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("sigprocmask@/tmp... no match
Completed one probe test.
Current size of probes.pending is 70
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("tcp_disconnect@/... no match
Completed one probe test.
Current size of probes.pending is 51
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("do_sync_write@/t... no match
Completed one probe test.
Current size of probes.pending is 32
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("generic_file_spl... no match
Completed one probe test.
Current size of probes.pending is 13
Start a probe test...
ERROR no match in probe_ok() L2_Grp_kernel.function("do_sync_read@/tm... no match
Completed one probe test.
Current size of probes.pending is 0
Running level increased to 3
Exceed max running level limit.
Remove all temporary files, unregister the service and return.

           ===  Summary ===

# of expected passes            1
lindi2:~/tmp/systemtap/testsuite(master)$ 

-Timo









Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Fri, 12 Aug 2011 11:56:30 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Frank Ch. Eigler" <fche@redhat.com>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Fri, 12 Aug 2011 11:56:36 GMT) Full text and rfc822 format available.

Message #61 received at 604453@bugs.debian.org (full text, mbox):

From: "Frank Ch. Eigler" <fche@redhat.com>
To: Timo Juhani Lindfors <timo.lindfors@iki.fi>
Cc: 604453@bugs.debian.org, 604453-submitter@bugs.debian.org, dsmith@redhat.com
Subject: Re: Fwd: Fix the uprobes.ko path when used with remotes
Date: Fri, 12 Aug 2011 07:27:30 -0400
Hi -

On Fri, Aug 12, 2011 at 02:14:01PM +0300, Timo Juhani Lindfors wrote:
> > alibi procedure, see https://bugzilla.redhat.com/show_bug.cgi?id=655904
> 
> Thanks. I can confirm that these instructions crash
> linux-image-3.0.0-1-amd64  3.0.0-1
>
> at least under xen.

OK.  So that confirms kernel bugs.  Perhaps you should fork this bug
against the kernel, and hope for the best.


> The README in the same directory talks about
> generating a whitelist of safe probe points.  [...]
> After this I get the following output:
> [...]
> $ sudo runtest whitelist.exp
> Running ./systemtap.stress/whitelist.exp ...
> Start a fresh stp_genwhitelist test.
> current_size_const is initialized as 107
> Current size of probes.pending is 963
> Start a probe test...
> [...]
> Completed one probe test.
> Current size of probes.pending is 0
> Running level increased to 3
> Exceed max running level limit.
> Remove all temporary files, unregister the service and return.

David, can you advise to what extent the results of this whitelist
test could be useful, for example to populate our blacklist?  The
number of non-inlined kernel functions is far greater than 963, so
perhaps this default test run just looked at those functions listed by
the various tapsets.  If so, one'd need to add gross patterns like
kernel.function("*").call to give the kernel a manlier workout.

- FChE




Message sent on to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Bug#604453. (Fri, 12 Aug 2011 12:02:12 GMT) Full text and rfc822 format available.

Message sent on to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Bug#604453. (Fri, 12 Aug 2011 12:02:40 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Mon, 15 Aug 2011 15:48:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to David Smith <dsmith@redhat.com>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Mon, 15 Aug 2011 15:48:03 GMT) Full text and rfc822 format available.

Message #72 received at 604453@bugs.debian.org (full text, mbox):

From: David Smith <dsmith@redhat.com>
To: "Frank Ch. Eigler" <fche@redhat.com>
Cc: Timo Juhani Lindfors <timo.lindfors@iki.fi>, 604453@bugs.debian.org, 604453-submitter@bugs.debian.org
Subject: Re: Fwd: Fix the uprobes.ko path when used with remotes
Date: Mon, 15 Aug 2011 10:45:11 -0500
On 08/12/2011 06:27 AM, Frank Ch. Eigler wrote:
> David, can you advise to what extent the results of this whitelist
> test could be useful, for example to populate our blacklist?  The
> number of non-inlined kernel functions is far greater than 963, so
> perhaps this default test run just looked at those functions listed by
> the various tapsets.  If so, one'd need to add gross patterns like
> kernel.function("*").call to give the kernel a manlier workout.

The whitelist test is pretty old, and should probably be removed.  The
stuff in scripts/kprobes_test is much closer to what I think you want,
but of course it doesn't use systemtap, it tests "raw" kprobes.  It
probably wouldn't be that difficult to modify
scripts/kprobes_test/gen_code.py to produce a systemtap module instead
of a "raw" kprobes module.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)




Message sent on to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Bug#604453. (Mon, 15 Aug 2011 15:48:06 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Wed, 17 Aug 2011 12:51:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Wed, 17 Aug 2011 12:51:10 GMT) Full text and rfc822 format available.

Message #80 received at 604453@bugs.debian.org (full text, mbox):

From: Timo Juhani Lindfors <timo.lindfors@iki.fi>
To: David Smith <dsmith@redhat.com>
Cc: "Frank Ch. Eigler" <fche@redhat.com>, 604453@bugs.debian.org, 604453-submitter@bugs.debian.org
Subject: Re: Fwd: Fix the uprobes.ko path when used with remotes
Date: Wed, 17 Aug 2011 15:48:20 +0300
Hi David and Frank,

David Smith <dsmith@redhat.com> writes:
> The whitelist test is pretty old, and should probably be removed.  The
> stuff in scripts/kprobes_test is much closer to what I think you want,
> but of course it doesn't use systemtap, it tests "raw" kprobes.  It
> probably wouldn't be that difficult to modify
> scripts/kprobes_test/gen_code.py to produce a systemtap module instead
> of a "raw" kprobes module.

thanks for the pointer. I forked gen_code_all.sh to a version that
incrementally adds new probes to the set until the kernel crashes. The
log looks like:

round 1: adding run_init_process to the set
round 2: adding _stext to the set
round 3: adding hypercall_page to the set
round 4: adding arch_local_save_flags to the set
round 5: adding arch_local_irq_disable to the set
round 6: adding arch_local_irq_enable to the set
round 7: adding run_init_process to the set
round 1: adding _stext to the set
round 2: adding hypercall_page to the set
round 3: adding arch_local_save_flags to the set
round 4: adding arch_local_irq_disable to the set
round 5: adding arch_local_irq_enable to the set
round 6: adding run_init_process to the set
round 7: adding init_post to the set
round 8: adding do_one_initcall to the set
round 9: adding match_dev_by_uuid to the set
round 10: adding name_to_dev_t to the set
round 11: adding arch_local_irq_restore to the set
round 12: adding arch_local_irq_disable to the set
round 13: adding native_read_cr4 to the set
round 14: adding native_read_cr4_safe to the set
round 15: adding native_wbinvd to the set
round 16: adding native_read_msr_safe to the set
round 17: adding native_read_pmc to the set
round 18: adding cpuid to the set
round 19: adding pte_pfn to the set
round 20: adding pfn_pte to the set
round 21: adding native_store_gdt to the set
round 22: adding native_store_idt to the set
round 23: adding xen_mc_batch to the set
round 24: adding clamp_max_cpus to the set
round 25: adding xen_cpuid to the set
round 26: adding xen_set_debugreg to the set

After that I get

[440390.108932] Unrecoverable kprobe detected at ffffffff8100311e.
[440390.108936] Dumping kprobe:
[440390.108939] Name: xen_set_debugreg
[440390.108940] Address: ffffffff8100311e
[440390.108940] Offset: 0
[440390.108949] ------------[ cut here ]------------
[440390.108952] kernel BUG at /tmp/buildd/linux-2.6-3.0.0/debian/build/source_amd64_none/arch/x86/kernel/kprobes.c:523!

and the machine is dead.

Can we please blacklist these in Linux or systemtap?

-Timo




Message sent on to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Bug#604453. (Wed, 17 Aug 2011 12:52:42 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Thu, 18 Aug 2011 17:51:11 GMT) Full text and rfc822 format available.

Acknowledgement sent to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Thu, 18 Aug 2011 17:51:11 GMT) Full text and rfc822 format available.

Message #88 received at 604453@bugs.debian.org (full text, mbox):

From: Timo Juhani Lindfors <timo.lindfors@iki.fi>
To: David Smith <dsmith@redhat.com>
Cc: "Frank Ch. Eigler" <fche@redhat.com>, 604453@bugs.debian.org, 604453-submitter@bugs.debian.org
Subject: Re: Bug#604453: systemtap: reproducible way to crash the system with kernel.function("*@*/*.c")
Date: Thu, 18 Aug 2011 20:49:44 +0300
[Message part 1 (text/plain, inline)]
Timo Juhani Lindfors <timo.lindfors@iki.fi> writes:
> thanks for the pointer. I forked gen_code_all.sh to a version that
> incrementally adds new probes to the set until the kernel crashes.

This process revealed only 17 problematic functions:

xen_set_debugreg
xen_get_debugreg
xen_irq_enable_direct_reloc
xen_irq_disable_direct
xen_irq_disable_direct_end
xen_save_fl_direct
xen_restore_fl_direct_reloc
check_events
xen_adjust_exception_frame
xen_iret
xen_iret_reloc
xen_sysret64
xen_sysret64_reloc
test_ti_thread_flag.clone.16
hash_ptr.clone.12
inat_get_opcode_attribute
system_call_after_swapgs

I blacklisted these and I can now load the huge module generated by
gen_code_all.sh succesfully:

$ lsmod | grep kprobe_module
kprobe_module        3432584  -Timo

I then blacklisted these in dwflpp.cxx with the attached patch and
voila, the original bug does not occur anymore :-)

lindi2:~/tmp$ cat > mmap-trace.stp <<EOF
> global trace
> 
> function trace(entry_p) {
>   if(tid() in trace)
>     printf("%s%s%s\n",
>       thread_indent(entry_p),
>       (entry_p>0?"->":"<-"),
>       probefunc())
> }
> 
> probe syscall.execve {
> #  printf("%s\n", filename);
>   if (filename != "./mmap") next
>   trace[tid()] = 1
>   trace(1)
> }
> #probe syscall.execve.return {
> #   trace(-1)
> #   delete trace[tid()]
> #}
> #probe kernel.function("*@mm/*.c").call { trace(1) }
> #probe kernel.function("*@mm/*.c").return { trace(-1) }
> 
> probe kernel.function("*@*/*.c").call { trace(1) }
> probe kernel.function("*@*/*.c").return { trace(-1) }
> 
> probe begin {
>     printf("begin\n");
> }
> EOF
lindi2:~/tmp$ sudo stap ./mmap-trace.stp
begin

How should we continue from here? If systemtap upstream accepts this
should somebody then try to get the missing __kprobe annotations also to
Linux?

-Timo
[0001-Add-few-more-entries-to-kprobe-blacklist.patch (text/x-diff, attachment)]

Message sent on to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Bug#604453. (Thu, 18 Aug 2011 17:51:13 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Thu, 18 Aug 2011 18:15:07 GMT) Full text and rfc822 format available.

Acknowledgement sent to David Smith <dsmith@redhat.com>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Thu, 18 Aug 2011 18:15:07 GMT) Full text and rfc822 format available.

Message #96 received at 604453@bugs.debian.org (full text, mbox):

From: David Smith <dsmith@redhat.com>
To: Timo Juhani Lindfors <timo.lindfors@iki.fi>
Cc: "Frank Ch. Eigler" <fche@redhat.com>, 604453@bugs.debian.org, 604453-submitter@bugs.debian.org
Subject: Re: Bug#604453: systemtap: reproducible way to crash the system with kernel.function("*@*/*.c")
Date: Thu, 18 Aug 2011 13:14:21 -0500
On 08/18/2011 12:49 PM, Timo Juhani Lindfors wrote:
> Timo Juhani Lindfors <timo.lindfors@iki.fi> writes:
> How should we continue from here? If systemtap upstream accepts this
> should somebody then try to get the missing __kprobe annotations also to
> Linux?

Now you need to figure out if this is a systemtap problem or a
kernel/kprobes problem.  To do that, go back to the original gen_code.py
to generate a module that uses straight kprobes (no systemtap).  If the
problem still occurs, then the kernel folks should listen to your
request to add the missing __kprobe annotations.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)




Message sent on to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Bug#604453. (Thu, 18 Aug 2011 18:15:12 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Thu, 18 Aug 2011 18:24:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Thu, 18 Aug 2011 18:24:03 GMT) Full text and rfc822 format available.

Message #104 received at 604453@bugs.debian.org (full text, mbox):

From: Timo Juhani Lindfors <timo.lindfors@iki.fi>
To: David Smith <dsmith@redhat.com>
Cc: "Frank Ch. Eigler" <fche@redhat.com>, 604453@bugs.debian.org, 604453-submitter@bugs.debian.org
Subject: Re: Bug#604453: systemtap: reproducible way to crash the system with kernel.function("*@*/*.c")
Date: Thu, 18 Aug 2011 21:20:51 +0300
David Smith <dsmith@redhat.com> writes:
> Now you need to figure out if this is a systemtap problem or a
> kernel/kprobes problem.  To do that, go back to the original gen_code.py
> to generate a module that uses straight kprobes (no systemtap).

That's what I did. I modified gen_code_all.sh to first find my the buggy
functions and then to verify that everything works when they are not
used. Only after that I modified the systemtap blacklist.

> If the problem still occurs, then the kernel folks should listen to
> your request to add the missing __kprobe annotations.

Ok.

-Timo




Message sent on to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Bug#604453. (Thu, 18 Aug 2011 18:24:05 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Mon, 06 Feb 2012 10:36:19 GMT) Full text and rfc822 format available.

Acknowledgement sent to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Mon, 06 Feb 2012 10:36:27 GMT) Full text and rfc822 format available.

Message #112 received at 604453@bugs.debian.org (full text, mbox):

From: Timo Juhani Lindfors <timo.lindfors@iki.fi>
To: 604453@bugs.debian.org
Cc: "Frank Ch. Eigler" <fche@redhat.com>, 604453-submitter@bugs.debian.org, David Smith <dsmith@redhat.com>
Subject: Re: Bug#604453: systemtap: reproducible way to crash the system with kernel.function("*@*/*.c")
Date: Mon, 06 Feb 2012 12:35:08 +0200
Hi,

here's a related upstream bug:

http://sourceware.org/bugzilla/show_bug.cgi?id=13659

-Timo




Message sent on to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Bug#604453. (Mon, 06 Feb 2012 10:36:43 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Ritesh Raj Sarraf <rrs@debian.org>:
Bug#604453; Package systemtap. (Thu, 15 Aug 2013 09:27:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Timo Juhani Lindfors <timo.lindfors@iki.fi>:
Extra info received and forwarded to list. Copy sent to Ritesh Raj Sarraf <rrs@debian.org>. (Thu, 15 Aug 2013 09:27:04 GMT) Full text and rfc822 format available.

Message #120 received at 604453@bugs.debian.org (full text, mbox):

From: Timo Juhani Lindfors <timo.lindfors@iki.fi>
To: 604453@bugs.debian.org
Subject: #604453 reproducible way to crash the system with kernel.function("*@*/*.c")
Date: Thu, 15 Aug 2013 12:02:05 +0300
Hi,

just a status update, the following combination still crashes:

 systemtap 2.3-1
 linux-image-3.10-2-amd64 3.10.5-1

at least under xen with the following messages:
 
[333215.296319] BUG: unable to handle kernel NULL pointer dereference at 0000000000000050
[333215.296704] IP: [<ffffffff8138ad42>] vmalloc_fault+0x85/0x218
[333215.297156] PGD 0 
[333215.297429] Oops: 0000 [#1] SMP 
[333215.297750] Modules linked in: stap_fd7ea93b5dd2667b9c0675e91657cd59_7_27055(O) zlib_deflate mtd parport_pc ppdev lp parport cuse binfmt_misc fuse rpcsec_gss_krb5 nfsv4 nfsd auth_rpcgss oid_registry nfs_acl nfs lockd dns_resolver fscache sunrpc loop evdev snd_pcm_oss snd_mixer_oss snd_pcm acpi_cpufreq mperf processor thermal_sys snd_page_alloc snd_timer snd soundcore pcspkr ext3 mbcache jbd virtio_rng rng_core virtio_net virtio_blk virtio_balloon virtio_pci virtio_ring virtio xen_netfront xen_blkfront [last unloaded: stap_181ee4f37ff7ffc8f1678b7334b79c6e_21956]
[333215.297772] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G           O 3.10-2-amd64 #1 Debian 3.10.5-1
[333215.297772] task: ffff88003e20f040 ti: ffff88003e23e000 task.ti: ffff88003e23e000
[333215.297772] RIP: e030:[<ffffffff8138ad42>]  [<ffffffff8138ad42>] vmalloc_fault+0x85/0x218
[333215.297772] RSP: e02b:ffff88003c7dd4c0  EFLAGS: 00010086
[333215.297772] RAX: 0000000000000000 RBX: ffffe8ffffd00000 RCX: ffffffff8138846c
[333215.297772] RDX: 00001fffffd00000 RSI: 0000000000000000 RDI: 0000000000000e88
[333215.297772] RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff8163ad30
[333215.297772] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff8160ce88
[333215.297772] R13: ffff88003e20f040 R14: 0000000000000028 R15: 0000000000000000
[333215.297772] FS:  00007f2647e1a700(0000) GS:ffff88003f900000(0000) knlGS:0000000000000000
[333215.297772] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[333215.297772] CR2: 0000000000000050 CR3: 000000003c76f000 CR4: 0000000000000660
[333215.297772] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[333215.297772] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[333215.297772] Stack:
[333215.297772]  ffff88003c7dd5e8 0000000000000000 ffffe8ffffd00000 ffff88003e20f040
[333215.297772]  0000000000000028 ffffffff8138b044 ffffffff811b002f 0000000000000068
[333215.297772]  0000000000000000 ffffffff81389984 0000000000000000 ffff88003c7dd608
[333215.297772] Call Trace:
[333215.297772] Code: 38 30 00 01 48 89 df 4c 8b 25 4b 7f 2b 00 65 48 8b 04 25 c0 c7 00 00 48 c1 ef 24 48 8b 80 d8 01 00 00 81 e7 f8 0f 00 00 49 01 fc <48> 8b 68 50 49 8b 34 24 48 01 fd 48 85 f6 0f 84 73 01 00 00 48 
[333215.297772] RIP  [<ffffffff8138ad42>] vmalloc_fault+0x85/0x218
[333215.297772]  RSP <ffff88003c7dd4c0>
[333215.297772] CR2: 0000000000000050
[333215.297772] ---[ end trace b5a343cbf72903fa ]---
[333215.297772] Kernel panic - not syncing: Attempted to kill the idle task!

-Timo



Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Sat Apr 19 03:02:23 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.