Debian Bug report logs - #634149
KVM string PIO emulation slow since Linux 2.6.39

version graph

Package: src:linux-2.6; Maintainer for src:linux-2.6 is Debian Kernel Team <debian-kernel@lists.debian.org>;

Reported by: Svante Signell <srs@kth.se>

Date: Mon, 11 Jul 2011 21:51:02 UTC

Severity: wishlist

Tags: confirmed, wontfix

Merged with 633589

Found in version linux-2.6/2.6.39-2

Done: Moritz Mühlenhoff <jmm@inutil.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, unknown-package@qa.debian.org:
Bug#633589; Package linux-image. (Mon, 11 Jul 2011 21:51:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Svante Signell <srs@kth.se>:
New Bug report received and forwarded. Copy sent to unknown-package@qa.debian.org. (Mon, 11 Jul 2011 21:51:04 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Svante Signell <srs@kth.se>
To: Debian bug reporting <submit@bugs.debian.org>
Subject: kvm extremely __slow__ under 2.6.39-2-amd64 compared to 2.6.32-5-amd64
Date: Mon, 11 Jul 2011 22:55:29 +0200
Package: linux-image 
Version: 2.6.39-2-amd64
Severity: important

Since kernel version 2.6.{37-39}-x-amd64 running GNU/Hurd under kvm
(with hardware support, kvm_intel and kvm kernel modules loaded) runs at
crawling speed compared to the latest fully working
linux-kernel-2.6.32-5-amd64. With this speed kvm is totally unusable. 

Has this been reported for other OSes/archs running under kvm? 
In case this is not known in Debian where can I report this upstream?





Bug reassigned from package 'linux-image' to 'linux-image-2.6.39-2-amd64'. Request was from Gergely Nagy <algernon@madhouse-project.org> to control@bugs.debian.org. (Mon, 11 Jul 2011 22:33:04 GMT) Full text and rfc822 format available.

Bug No longer marked as found in versions 2.6.39-2-amd64. Request was from Gergely Nagy <algernon@madhouse-project.org> to control@bugs.debian.org. (Mon, 11 Jul 2011 22:33:05 GMT) Full text and rfc822 format available.

Bug Marked as found in versions linux-2.6/2.6.39-2. Request was from Gergely Nagy <algernon@madhouse-project.org> to control@bugs.debian.org. (Mon, 11 Jul 2011 22:33:05 GMT) Full text and rfc822 format available.

Bug reassigned from package 'linux-image-2.6.39-2-amd64' to 'src:linux-2.6'. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Mon, 11 Jul 2011 23:48:06 GMT) Full text and rfc822 format available.

Bug No longer marked as found in versions linux-2.6/2.6.39-2. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Mon, 11 Jul 2011 23:48:07 GMT) Full text and rfc822 format available.

Bug Marked as found in versions linux-2.6/2.6.39-2. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Mon, 11 Jul 2011 23:48:08 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#633589; Package src:linux-2.6. (Tue, 12 Jul 2011 10:12:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to srs@kth.se:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Tue, 12 Jul 2011 10:12:06 GMT) Full text and rfc822 format available.

Message #22 received at 633589@bugs.debian.org (full text, mbox):

From: Svante Signell <srs@kth.se>
To: 633589@bugs.debian.org
Subject: Slowdown confirmed also for AMD
Date: Tue, 12 Jul 2011 11:51:37 +0200
I have another AMD-box with HW acceleration via insertion of kvm and
kvm_amd modules. I can confirm that the slowdown is also present on that
box (2.6.32 is blazingly fast compared to 2.6.39!!)







Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#633589; Package src:linux-2.6. (Sun, 17 Jul 2011 08:54:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to srs@kth.se:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Sun, 17 Jul 2011 08:54:03 GMT) Full text and rfc822 format available.

Message #27 received at 633589@bugs.debian.org (full text, mbox):

From: Svante Signell <srs@kth.se>
To: 633589@bugs.debian.org
Cc: control@bugs.debian.org
Subject: Some sysbench numbers.
Date: Sun, 17 Jul 2011 10:51:18 +0200
clone 633589 -1
reassign -1 qemu-kvm
thanks

Some numbers from sysbench:
srs: sysbench sequential read 2.6.32: 12.97Mb/sec, 2.6.39: 3.38Mb/sec.
srs: sysbench sequential write: 2.6.32: 10.08Mb/sec. 2.6.39: Waited more
than an hour before rebooting!

Version: qemu-kvm 0.14.1+dfsg-2





Bug 633589 cloned as bug 634149. Request was from Svante Signell <srs@kth.se> to control@bugs.debian.org. (Sun, 17 Jul 2011 09:15:31 GMT) Full text and rfc822 format available.

Bug reassigned from package 'src:linux-2.6' to 'qemu-kvm'. Request was from Svante Signell <srs@kth.se> to control@bugs.debian.org. (Sun, 17 Jul 2011 09:15:35 GMT) Full text and rfc822 format available.

Bug No longer marked as found in versions linux-2.6/2.6.39-2. Request was from Svante Signell <srs@kth.se> to control@bugs.debian.org. (Sun, 17 Jul 2011 09:15:35 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Jan Lübbe <jluebbe@debian.org>:
Bug#634149; Package qemu-kvm. (Mon, 18 Jul 2011 07:06:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Michael Tokarev <mjt@tls.msk.ru>:
Extra info received and forwarded to list. Copy sent to Jan Lübbe <jluebbe@debian.org>. (Mon, 18 Jul 2011 07:06:04 GMT) Full text and rfc822 format available.

Message #38 received at 634149@bugs.debian.org (full text, mbox):

From: Michael Tokarev <mjt@tls.msk.ru>
To: 634149@bugs.debian.org
Cc: Svante Signell <srs@kth.se>
Subject: Re: Bug#634149: kvm extremely __slow__ under 2.6.39-2-amd64 compared to 2.6.32-5-amd64
Date: Mon, 18 Jul 2011 11:03:31 +0400
tags 634149 + moreinfo
thanks

There's almost no information in this bugreport.
You never specified which command line options did you
use.  You never stated what kind of guest is that, ie,
what one should do to reproduce this.  You didn't tell
us what kind of storage do you have.

One more note: you cloned this bugreport to qemu-kvm
package, even if that package appears to be unaffected
(since the same version works differently with different
kernels).  Why?

Please provide some information to reproduce the problem.


Thanks,

/mjt




Added tag(s) moreinfo. Request was from Michael Tokarev <mjt@tls.msk.ru> to control@bugs.debian.org. (Mon, 18 Jul 2011 07:06:06 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Jan Lübbe <jluebbe@debian.org>:
Bug#634149; Package qemu-kvm. (Mon, 18 Jul 2011 08:09:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to srs@kth.se:
Extra info received and forwarded to list. Copy sent to Jan Lübbe <jluebbe@debian.org>. (Mon, 18 Jul 2011 08:09:06 GMT) Full text and rfc822 format available.

Message #45 received at 634149@bugs.debian.org (full text, mbox):

From: Svante Signell <srs@kth.se>
To: Michael Tokarev <mjt@tls.msk.ru>
Cc: 634149@bugs.debian.org
Subject: Re: Bug#634149: kvm extremely __slow__ under 2.6.39-2-amd64 compared to 2.6.32-5-amd64
Date: Mon, 18 Jul 2011 09:45:22 +0200
On Mon, 2011-07-18 at 11:03 +0400, Michael Tokarev wrote:
> tags 634149 + moreinfo
> thanks
> 
> There's almost no information in this bugreport.
> You never specified which command line options did you
> use.  You never stated what kind of guest is that, ie,
> what one should do to reproduce this.  You didn't tell
> us what kind of storage do you have.

Host: Linux AMD + Intel
Guest: GNU/Hurd
sysbench numbers: From the AMD box.
see below for more information.

> One more note: you cloned this bugreport to qemu-kvm
> package, even if that package appears to be unaffected
> (since the same version works differently with different
> kernels).  Why?

Maybe that was a mistake. Somebody on #kvm mentioned that later Linux
kernels opens up more features, that kvm uses. Unfortunately these
features are not HW-accelerated.

> Please provide some information to reproduce the problem.

Commandline: (both Intel and AMD boxes)

kvm -m 1024 -net nic,model=rtl8139 -net user,hostfwd=tcp::5556-:22
-drive cache=writeback,index=0,media=disk,file=hurd.img -cdrom
netinst.iso

Intel box:
==========
RAM: 4GB

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
CPU(s):                4
Thread(s) per core:    1
Core(s) per socket:    4
CPU socket(s):         1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 15
Stepping:              11
CPU MHz:               1596.000
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K

cat /proc/cpuinfo + 3 more CPUs
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
stepping	: 11
cpu MHz		: 2400.000
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm
constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64
monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow vnmi
flexpriority
bogomips	: 4800.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

Loaded modules:
kvm_intel              38146  6 
kvm                   214248  1 kvm_intel


Storage:
[    1.112138] scsi 0:0:0:0: Direct-Access     ATA      ST3500630AS
3.CH PQ: 0 ANSI: 5
[    1.123907] sd 0:0:0:0: [sda] 976773168 512-byte logical blocks: (500
GB/465 GiB)
[    1.123966] sd 0:0:0:0: [sda] Write Protect is off
[    1.123968] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    1.123988] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA

AMD-box: 
========
(that box is shut down right now, more info abut the box can be found in
bug #619034.)

RAM: 2GB

CPU:
[  186.838814] kvm: Nested Virtualization enabled
[  188.350222] powernow-k8: Found 1 AMD Athlon(tm) 64 X2 Dual Core
Processor 6000+ (2 cpu cores) (version 2.20.00)

Board info:
sys_vendor: MICRO-STAR INTERNATIONAL CO., LTD
product_name: MS-7253
product_version: 1.0
chassis_vendor: MICRO-STAR INTERNATIONAL CO., LTD
chassis_version:  
bios_vendor: Phoenix Technologies, LTD
bios_version: V1.6
board_vendor: MICRO-STAR INTERNATIONAL CO., LTD
board_name: MS-7253
board_version: 1.0

Loaded modules:
kvm_amd                42175  0 
kvm                   255441  1 kvm_amd

Storage:
Similar as the Intel box, 500GB SATA






Information forwarded to debian-bugs-dist@lists.debian.org, Jan Lübbe <jluebbe@debian.org>:
Bug#634149; Package qemu-kvm. (Mon, 18 Jul 2011 10:27:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Michael Tokarev <mjt@tls.msk.ru>:
Extra info received and forwarded to list. Copy sent to Jan Lübbe <jluebbe@debian.org>. (Mon, 18 Jul 2011 10:27:08 GMT) Full text and rfc822 format available.

Message #50 received at 634149@bugs.debian.org (full text, mbox):

From: Michael Tokarev <mjt@tls.msk.ru>
To: srs@kth.se
Cc: 634149@bugs.debian.org
Subject: Re: Bug#634149: kvm extremely __slow__ under 2.6.39-2-amd64 compared to 2.6.32-5-amd64
Date: Mon, 18 Jul 2011 14:24:09 +0400
18.07.2011 11:45, Svante Signell wrote:
> On Mon, 2011-07-18 at 11:03 +0400, Michael Tokarev wrote:
[]
> kvm -m 1024 -net nic,model=rtl8139 -net user,hostfwd=tcp::5556-:22
> -drive cache=writeback,index=0,media=disk,file=hurd.img -cdrom
> netinst.iso

Ok.  What's hurd.img and what's netinst.iso?  How did you install
whatever is in hurd.img?

The thing is: I need information to reproduce the issue.  If I'll
use the above command line locally it will complain for sure --
complain about missing hurd.img and netinst.iso, because I don't
have these here, so I need a way to get/create them.

For the rest, that's basically ok.  Only missing info is the guest
bits.

btw, it was me with whom you spoke in #kvm.

/mjt




Information forwarded to debian-bugs-dist@lists.debian.org, Jan Lübbe <jluebbe@debian.org>:
Bug#634149; Package qemu-kvm. (Tue, 19 Jul 2011 08:51:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Michael Tokarev <mjt@tls.msk.ru>:
Extra info received and forwarded to list. Copy sent to Jan Lübbe <jluebbe@debian.org>. (Tue, 19 Jul 2011 08:51:07 GMT) Full text and rfc822 format available.

Message #55 received at 634149@bugs.debian.org (full text, mbox):

From: Michael Tokarev <mjt@tls.msk.ru>
To: Michael Tokarev <mjt@tls.msk.ru>, 634149@bugs.debian.org
Cc: srs@kth.se
Subject: Re: Bug#634149: kvm extremely __slow__ under 2.6.39-2-amd64 compared to 2.6.32-5-amd64
Date: Tue, 19 Jul 2011 12:49:31 +0400
reassign 634149 src:linux-2.6
tags 634149 - moreinfo + confirmed
merge 633589 634149
thanks

18.07.2011 14:24, Michael Tokarev wrote:
[..]

So, the problem is that with this kernel commit:

commit 7972995b0c346de76fe260ce0fd6bcc8ffab724a
Author: Gleb Natapov <gleb@redhat.com>
Date:   Thu Mar 18 15:20:24 2010 +0200

    KVM: x86 emulator: Move string pio emulation into emulator.c

    Currently emulation is done outside of emulator so things like doing
    ins/outs to/from mmio are broken it also makes it hard (if not impossible)
    to implement single stepping in the future. The implementation in this
    patch is not efficient since it exits to userspace for each IO while
    previous implementation did 'ins' in batches. Further patch that
    implements pio in string read ahead address this problem.

    Signed-off-by: Gleb Natapov <gleb@redhat.com>
    Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

which went into mainline with 2.6.35, there has been
quite some changes in PIO emulation handling in kvm
which resulted in correct but slow (as opposed by
fast but incorrect) emulation.  This slowed down
guests that use PIO to access disks. Among these
are WinXP (it switches from DMA to PIO after some
I/O errors), WinNT (ditto) and - apparently - Hurd.

Someone needs to investigate why Hurd does not use
DMA in this case - PIO is long obsolete technology.

I'd mark this as "notabug" (not possible with BTS)
or "wontfix", but it's definitely possible to
optimize the new code further to speed things up.
If it's worth the effort is another question.

/mjt




Bug reassigned from package 'qemu-kvm' to 'src:linux-2.6'. Request was from Michael Tokarev <mjt@tls.msk.ru> to control@bugs.debian.org. (Tue, 19 Jul 2011 08:51:15 GMT) Full text and rfc822 format available.

Removed tag(s) moreinfo. Request was from Michael Tokarev <mjt@tls.msk.ru> to control@bugs.debian.org. (Tue, 19 Jul 2011 08:51:19 GMT) Full text and rfc822 format available.

Added tag(s) confirmed. Request was from Michael Tokarev <mjt@tls.msk.ru> to control@bugs.debian.org. (Tue, 19 Jul 2011 08:51:23 GMT) Full text and rfc822 format available.

Merged 633589 634149. Request was from Michael Tokarev <mjt@tls.msk.ru> to control@bugs.debian.org. (Tue, 19 Jul 2011 08:51:27 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#634149; Package src:linux-2.6. (Tue, 19 Jul 2011 09:09:54 GMT) Full text and rfc822 format available.

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Tue, 19 Jul 2011 09:10:31 GMT) Full text and rfc822 format available.

Message #68 received at 634149@bugs.debian.org (full text, mbox):

From: Jonathan Nieder <jrnieder@gmail.com>
To: Michael Tokarev <mjt@tls.msk.ru>
Cc: 634149@bugs.debian.org, srs@kth.se
Subject: Re: kvm extremely __slow__ under 2.6.39-2-amd64 compared to 2.6.32-5-amd64
Date: Tue, 19 Jul 2011 04:05:46 -0500
severity 634149 wishlist
quit

Hi,

Michael Tokarev wrote:

> So, the problem is that with this kernel commit:
>
> commit 7972995b0c346de76fe260ce0fd6bcc8ffab724a
> Author: Gleb Natapov <gleb@redhat.com>
> Date:   Thu Mar 18 15:20:24 2010 +0200
>
>     KVM: x86 emulator: Move string pio emulation into emulator.c
[...]
> which went into mainline with 2.6.35, there has been
> quite some changes in PIO emulation handling in kvm
> which resulted in correct but slow (as opposed by
> fast but incorrect) emulation.  This slowed down
> guests that use PIO to access disks. Among these
> are WinXP (it switches from DMA to PIO after some
> I/O errors), WinNT (ditto) and - apparently - Hurd.

Thanks for a nice explanation.

[...]
> I'd mark this as "notabug" (not possible with BTS)
> or "wontfix", but it's definitely possible to
> optimize the new code further to speed things up.
> If it's worth the effort is another question.

That means "wishlist", and probably not worth tracking unless we want
to document it somewhere or someone is interested in working on it,
right?

Svante, if I were in your shoes, I'd come up with some words of
warning for a qemu-performance.txt file and send them (ideally as a
patch against the qemu or qemu-kvm package) to kvm@vger.kernel.org,
cc-ing this bug.  And if you are looking for a low-level Hurd coding
project, perhaps look into teaching the Hurd some performance tricks
--- it would help on raw hardware, too. :)  Investigating how to
improve kvm's PIO emulation might also be interesting.

Hope that helps,
Jonathan




Severity set to 'wishlist' from 'important' Request was from Jonathan Nieder <jrnieder@gmail.com> to control@bugs.debian.org. (Tue, 19 Jul 2011 09:10:58 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Kernel Team <debian-kernel@lists.debian.org>:
Bug#634149; Package src:linux-2.6. (Tue, 19 Jul 2011 09:16:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <debian-kernel@lists.debian.org>. (Tue, 19 Jul 2011 09:16:10 GMT) Full text and rfc822 format available.

Message #75 received at 634149@bugs.debian.org (full text, mbox):

From: Jonathan Nieder <jrnieder@gmail.com>
To: Michael Tokarev <mjt@tls.msk.ru>
Cc: 634149@bugs.debian.org, Svante Signell <srs@kth.se>
Subject: Re: kvm extremely __slow__ under 2.6.39-2-amd64 compared to 2.6.32-5-amd64
Date: Tue, 19 Jul 2011 04:13:09 -0500
Jonathan Nieder wrote:
> Michael Tokarev wrote:

>> So, the problem is that with this kernel commit:
>>
>> commit 7972995b0c346de76fe260ce0fd6bcc8ffab724a
>> Author: Gleb Natapov <gleb@redhat.com>
>> Date:   Thu Mar 18 15:20:24 2010 +0200
>>
>>     KVM: x86 emulator: Move string pio emulation into emulator.c
>[...]
>> which went into mainline with 2.6.35, there has been
>> quite some changes in PIO emulation handling in kvm
>> which resulted in correct but slow (as opposed by
>> fast but incorrect) emulation.  This slowed down
>> guests that use PIO to access disks.
[...]
> That means "wishlist", and probably not worth tracking unless we want
> to document it somewhere or someone is interested in working on it,
> right?

On second thought, a regression's a regression --- going from usable
but subtly buggy to unusable is not progress. :)  Maybe it would be
possible to introduce some kind of parameter to get back the speed
at the cost of correctness again?

Just musing.  Sorry for the noise; I'll leave this to the experts.




Changed Bug title to 'KVM string PIO emulation slow since Linux 2.6.39' from 'kvm extremely __slow__ under 2.6.39-2-amd64 compared to 2.6.32-5-amd64' Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Sat, 02 Jun 2012 18:12:06 GMT) Full text and rfc822 format available.

Added indication that 634149 affects hurd Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Sat, 02 Jun 2012 18:12:07 GMT) Full text and rfc822 format available.

Added tag(s) wontfix. Request was from Ben Hutchings <ben@decadent.org.uk> to control@bugs.debian.org. (Sat, 02 Jun 2012 18:18:04 GMT) Full text and rfc822 format available.

Message #82 received at 633589-done@bugs.debian.org (full text, mbox):

From: Moritz Mühlenhoff <jmm@inutil.org>
To: 483781-done@bugs.debian.org, 521515-done@bugs.debian.org, 558265-done@bugs.debian.org, 572639-done@bugs.debian.org, 586386-done@bugs.debian.org, 609961-done@bugs.debian.org, 630788-done@bugs.debian.org, 633589-done@bugs.debian.org, 639919-done@bugs.debian.org, 642005-done@bugs.debian.org, 705711-done@bugs.debian.org
Subject: Closing
Date: Thu, 27 Jun 2013 19:45:22 +0200
Hi,
your bug has been filed against the "linux-2.6" source package and has
been tagged "wontfix" for some time, we're closing it now.

Cheers,
        Moritz



Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Sat, 27 Jul 2013 07:38:06 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 17 01:51:41 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.