Debian Bug report logs - #594069
qemu-kvm: really poor write performance with qcow2 hdd image

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Alexander Loob <psionyx@gmx.de>

To: submit@bugs.debian.org

Subject: qemu-kvm: really poor write performance with qcow2 hdd image

Date: Mon, 23 Aug 2010 15:22:13 +0200

[Message part 1 (text/plain, inline)]

Package: qemu-kvm
Version: 0.12.5+dfsg-1
Severity: normal
Distribution: testing
Architecture: amd64


Dear Maintainers,

after the update to the qemu-kvm package to version 0.12.5+dfsg-1 the
write performance of my virtual machines is absolutely solw! I've tested
this with Windows 7, Windows XP Prof. and debian testing. E.g. nearly 10
mins for 600MB (Win7 hybernate).

Here is an example of my kvm start parameters:

kvm -drive
file=/home/virtual/KVM/win7.ovl,index=0,media=disk,cache=writeback -k de
-m 1536m -monitor stdio -name "Windows 7" -net
nic,model=virtio,macaddr=52:54:00:12:34:57 -net user -rtc base=localtime
-smp 2 -usb -usbdevice tablet -vga std -vnc :3

Changing the cache value to cache=none or writethrough has no effect to
the write performance.

Downgrading back to qemu-kvm-0.12.4+dfsg-1 solved the problem for me
(win7 hybernate writes 600MB in nearly 30 secs.)

Best regards
Alexander Loob

[signature.asc (application/pgp-signature, attachment)]

Message #10 received at 594069@bugs.debian.org (full text, mbox, reply):

From: Michael Tokarev <mjt@tls.msk.ru>

To: Alexander Loob <psionyx@gmx.de>, 594069@bugs.debian.org

Subject: Re: Bug#594069: qemu-kvm: really poor write performance with qcow2 hdd image

Date: Tue, 24 Aug 2010 12:11:15 +0400

23.08.2010 17:22, Alexander Loob wrote:
> Package: qemu-kvm
> Version: 0.12.5+dfsg-1
> Severity: normal
> Distribution: testing
> Architecture: amd64
> 
> Dear Maintainers,
> 
> after the update to the qemu-kvm package to version 0.12.5+dfsg-1 the
> write performance of my virtual machines is absolutely solw! I've tested
> this with Windows 7, Windows XP Prof. and debian testing. E.g. nearly 10
> mins for 600MB (Win7 hybernate).

Can you tell me what exactly are you doing when you observe the
slowdown?  I just tested both 0.12.4 and 0.12.5 on a few windows
images I have locally (winXP and win7), -- I see no speed difference
in regular operations between the two qemu-kvm versions.

> Here is an example of my kvm start parameters:
> 
> kvm \
  -drive file=/home/virtual/KVM/win7.ovl,index=0,media=disk,cache=writeback \
  -k de \
  -m 1536m \
  -monitor stdio \
  -name "Windows 7" \
  -net nic,model=virtio,macaddr=52:54:00:12:34:57 \
  -net user \
  -rtc base=localtime \
  -smp 2 \
  -usb \
  -usbdevice tablet \
  -vga std \
  -vnc :3

That makes sense and looks good, except of the -net user, which
is terrible slow by its own, but it weren't changed in a very
long time.

What's win7.ovl?  Is it a qcow2 file with a base image elsewhere?

> Changing the cache value to cache=none or writethrough has no effect to
> the write performance.
> 
> Downgrading back to qemu-kvm-0.12.4+dfsg-1 solved the problem for me
> (win7 hybernate writes 600MB in nearly 30 secs.)

Why do you think it writes 600Mb?

Thanks!

/mjt

Message #15 received at 594069@bugs.debian.org (full text, mbox, reply):

From: Michael Tokarev <mjt@tls.msk.ru>

To: Alexander Loob <psionyx@gmx.de>, 594069@bugs.debian.org

Subject: Re: Bug#594069: qemu-kvm: really poor write performance with qcow2 hdd image

Date: Tue, 24 Aug 2010 12:43:41 +0400

24.08.2010 12:11, Michael Tokarev wrote:
[]
>> Downgrading back to qemu-kvm-0.12.4+dfsg-1 solved the problem for me
>> (win7 hybernate writes 600MB in nearly 30 secs.)
> 
> Why do you think it writes 600Mb?

I just tried hybernation in win7 on a raw image (on ext4 fs)
and 1536Mb memory in guest.  It takes about 20 seconds here
with either 0.12.5+dfsg-1 or 0.12.4+dfsg-1 - i.e. qemu-kvm
version makes no difference whatsoever.

However I noticed that sometimes, when version strings of
some emulated devices changes (due to qemu-kvm version change),
windows sometimes (very rare) does a bad job at "updating"
drivers, and slows down to a crawl, like, any mouse movement
makes high disk trashing.  I've seen this at least 2 times
already (since kvm-68), and there are other people who
observed the same.  So far, we only tried reinstall.

I'm not saying it's your case.  Just providing some more
data, maybe irrelevant.

/mjt

Message #20 received at 594069@bugs.debian.org (full text, mbox, reply):

From: Michael Tokarev <mjt@tls.msk.ru>

To: Alexander Loob <psionyx@gmx.de>, 594069@bugs.debian.org

Subject: Re: Bug#594069: qemu-kvm: really poor write performance with qcow2 hdd image

Date: Tue, 24 Aug 2010 14:14:48 +0400

tags 594069 + confirmed upstream
severity 594069 important
thanks

Ok.  Indeed, qcow2 become MUCH slower in 0.12.5 compared with
0.12.4.  This is due to a series of patches that went into 0.12.5
upstream to make qcow2 (and other similar formats) metadata
updates syncronous.  Like this:

commit 37060c28e522843fbf6f7e59af745dfcb05b132c
Author: Kevin Wolf <kwolf@redhat.com>
Date:   Wed Jul 14 12:11:51 2010 +0200

    qcow2: Use bdrv_(p)write_sync for metadata writes

    Use bdrv_(p)write_sync to ensure metadata integrity in case of a crash.

    Signed-off-by: Kevin Wolf <kwolf@redhat.com>
    (cherry picked from commit 8b3b720620a1137a1b794fc3ed64734236f94e06)

...
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -62,8 +62,8 @@ int qcow2_grow_l1_table(BlockDriverState *bs, int min_size)
-    ret = bdrv_pwrite(s->hd, new_l1_table_offset, new_l1_table, new_l1_size2);
+    ret = bdrv_pwrite_sync(s->hd, new_l1_table_offset, new_l1_table, new_l1_size2);


In other words, before that change, qcow2 format were unsafe
against host crashes, it were possible to get data corruptions.
Now that is fixed, but the cost is insanely slow operations.

I'll revert these patches for squeeze, restoring the previous
(unsafe) behavour: it is better to have something than not
have anything, because the resulting qcow2 is now completely
unusable.

Raising severity to important because this is really important
problem.

Thanks!

/mjt

Message #31 received at 594069@bugs.debian.org (full text, mbox, reply):

From: Michael Tokarev <mjt@tls.msk.ru>

To: 594069@bugs.debian.org

Cc: Alexander Loob <psionyx@gmx.de>

Subject: Re: Bug#594069: Summary

Date: Wed, 25 Aug 2010 12:11:19 +0400

So, after some testing, I want to provide a summary
about this.

There are 3 cache modes in qemu for block devices:
none, writeback, and writethrough (default).

The series of patches we're talking about changed
all metadata updates to be syncronous.

The change in question does _not_ affect writethrough
mode, since there, all writes always were syncronous
anyway.  This is despite the information found in
first message that started this bugreport.  Alexander
Loob said:

  Changing the cache value to cache=none or writethrough
  has no effect to the write performance.

This is incorrect.  Yet again, the change does not
affect the default writethrough cache mode, since
it were syncronous before, and the change is a no-op
for that mode.  Note that this is the default cache
mode, so actually the change does not affect most
users.

But other cache modes, none and writeback, are affected,
both highly negative, the write performance reduced for
about 10 times or more on some workloads.

This, again, needs some clarifications.  The change only
affects metadata updates, i.e., roughly speaking, new
allocations in qcow2 file.  So if the guest is writing
to already allocated place, there's no penalty from the
changes in question, since in that case there's no
metadata updates happening.

This change affects qcow2 snapshots very significantly.
This is because after taking a snapshot, _all_ blocks
are marked as copy-on-write, so _all_ writes will require
both allocating a new block and decrementing reference
count in the old block.  So there are 2x more metadata
updates going on for every write after taking snapshot,
hence the usage case in question (using qcow2 snapshots)
is affected much more seriously than others.

Another case when cache=writeback is used often is an
OS install, to speed up installation process when it is
obvious that data integrity isn't important (in case of
crash it's just as easy to re-run installation procedure
from the beginning, no big deal).  This usage case is
obviously affected too, increasing install time for up
to 5 times.

And yet another place where this problem hit us is
qemu-img (kvm-img) utility.  When it creates a qcow2
image, it uses writeback cache.  On my home machine,
`kvm-img convert -O qcow2 f.raw f.qcow2' for a 8Gb
win7 image now requires 48 minutes to complete,
instead of usual ~4m.

So, the change does not affect default (and probably
most commonly used) operations of kvm itself.  It makes
unsafe operations (cache=writeback) partially safe at
a cost of large decrease in speed, and it makes some
utilities almost unusable.

I'm reverting it all back for the next debian release.
It may even be reverted in upstream for upcoming 0.13,
but I'm not sure.  This change needs some work before
it will be acceptable.

/mjt

Message #36 received at 594069@bugs.debian.org (full text, mbox, reply):

From: Alexander Loob <psionyx@gmx.de>

To: Michael Tokarev <mjt@tls.msk.ru>

Cc: 594069@bugs.debian.org

Subject: Re: Bug#594069: Summary

Date: Wed, 25 Aug 2010 12:07:33 +0200

[Message part 1 (text/plain, inline)]

So, after this summary here are my measurement results:

Host environment:

Lenovo X 61 Tablet with Core 2 Duo @ 1.6 GHz, $ GB RAM and
a Hitachi HTE 320GB HDD @ 7200rpm
debian testing amd64, all partitions formatted with ext4.

Guest environment:
debian testing amd64, all partitions formatted with ext4.

kvm command:
kvm -drive
file=/home/virtual/KVM/debian.ovl,index=0,media=disk,cache=writeback -k de
-m 1G -monitor stdio -net nic,model=virtio,macaddr=52:54:00:12:34:57 -net
user -rtc base=localtime -smp 2 -usb -usbdevice tablet -vga std -vnc :4

testcommand:
time dd if=/dev/zero of=/root/zero.img bs=1024k count=1024

qemu-kvm_0.12.4+dfsg1_amd64.deb
cache=writeback
 real 0m24.837s
 user 0m 0.008s
 sys  0m 9.997s

cache=writethrough
 real 3m11.935s
 user 0m 0.024s
 sys  0m10.153s

cache=none
 real 0m40.837s
 user 0m 0.012s
 sys  0m10.713s

qemu-kvm_0.12.5+dfsg1_amd64.deb
cache=writeback
 real 2m26.086s
 user 0m 0.020s
 sys  0m10.537s

cache=writethrough
 real 3m18.835s
 user 0m 0.008s
 sys  0m 9.985s

cache=none
 real 2m22.439s
 user 0m 0.024s
 sys  0m10.317s

The measurements of the first bugreport were subjective and for my
machine you can see that with 0.12.5 changing the cache to
writethrough|writeback|none has "nearly" no effect to the write
performance.

On Wed, 25 Aug 2010 12:11:19 +0400
Michael Tokarev <mjt@tls.msk.ru> wrote:

> The change in question does _not_ affect writethrough
> mode, since there, all writes always were syncronous
> anyway.  This is despite the information found in
> first message that started this bugreport.  Alexander
> Loob said:
> 
>   Changing the cache value to cache=none or writethrough
>   has no effect to the write performance.
> 
> This is incorrect.  Yet again, the change does not
> affect the default writethrough cache mode, since
> it were syncronous before, and the change is a no-op
> for that mode.  Note that this is the default cache
> mode, so actually the change does not affect most
> users.

[signature.asc (application/pgp-signature, attachment)]

Message #41 received at 594069-close@bugs.debian.org (full text, mbox, reply):

From: Jan Lübbe <jluebbe@debian.org>

To: 594069-close@bugs.debian.org

Subject: Bug#594069: fixed in qemu-kvm 0.12.5+dfsg-2

Date: Wed, 25 Aug 2010 11:47:34 +0000

Source: qemu-kvm
Source-Version: 0.12.5+dfsg-2

We believe that the bug you reported is fixed in the latest version of
qemu-kvm, which is due to be installed in the Debian FTP archive:

kvm_0.12.5+dfsg-2_amd64.deb
  to main/q/qemu-kvm/kvm_0.12.5+dfsg-2_amd64.deb
qemu-kvm-dbg_0.12.5+dfsg-2_amd64.deb
  to main/q/qemu-kvm/qemu-kvm-dbg_0.12.5+dfsg-2_amd64.deb
qemu-kvm_0.12.5+dfsg-2.diff.gz
  to main/q/qemu-kvm/qemu-kvm_0.12.5+dfsg-2.diff.gz
qemu-kvm_0.12.5+dfsg-2.dsc
  to main/q/qemu-kvm/qemu-kvm_0.12.5+dfsg-2.dsc
qemu-kvm_0.12.5+dfsg-2_amd64.deb
  to main/q/qemu-kvm/qemu-kvm_0.12.5+dfsg-2_amd64.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 594069@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Jan Lübbe <jluebbe@debian.org> (supplier of updated qemu-kvm package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Wed, 25 Aug 2010 13:15:11 +0200
Source: qemu-kvm
Binary: qemu-kvm qemu-kvm-dbg kvm
Architecture: source amd64
Version: 0.12.5+dfsg-2
Distribution: unstable
Urgency: low
Maintainer: Jan Lübbe <jluebbe@debian.org>
Changed-By: Jan Lübbe <jluebbe@debian.org>
Description: 
 kvm        - dummy transitional package from kvm to qemu-kvm
 qemu-kvm   - Full virtualization on x86 hardware
 qemu-kvm-dbg - Debugging info for qemu-kvm
Closes: 588739 593144 594069
Changes: 
 qemu-kvm (0.12.5+dfsg-2) unstable; urgency=low
 .
   [ Michael Tokarev ]
   * patch from upstream: add-support-for-GET-EVENT-STATUS-NOTIFICATION.diff
     (closes: #593144)
   * two patches from upstream qemu/stable-0.12:
     upstream-stable01-make-PIIX-and-ISA-IDE-init-functions-return-the-qdev.diff
     upstream-stable02-fix-CMOS-info-for-drives-defined-with--device.diff
     (closes: #588739)
   * revert-all-bdrv_write_sync-changes.diff - this patch reverts the changes
     which went into 0.12.5 upstream and make all metadata writes (for qcow&Co)
     syncronous.  Yes it's more correct, but the resulting write speed is
     insanely slow and makes all image formats except raw unusable.
     (closes: #594069)
Checksums-Sha1: 
 3d4f74f4fa55226746f725416b49332744ea1353 1649 qemu-kvm_0.12.5+dfsg-2.dsc
 7cc5ee0eb103ef0200846da18eaccaa461804e7c 302793 qemu-kvm_0.12.5+dfsg-2.diff.gz
 81034249e8ac34e0e16c06d134d079f1019419dd 1615748 qemu-kvm_0.12.5+dfsg-2_amd64.deb
 247bb842b1ccc4537b2755a200e1b4c3abb87107 2817174 qemu-kvm-dbg_0.12.5+dfsg-2_amd64.deb
 7f6af88b57230838d08f26d4fe73d853c4e5f2d5 11708 kvm_0.12.5+dfsg-2_amd64.deb
Checksums-Sha256: 
 0403b3c25eef204b89e15dcb0056de0de0138c51f86d52f9412b928b5b014b81 1649 qemu-kvm_0.12.5+dfsg-2.dsc
 00a671690703918c441651d275deb9389373b3e042ecd9bb22eea3cc692b64e7 302793 qemu-kvm_0.12.5+dfsg-2.diff.gz
 56da5e5eb713d8a6a8b0ae8f88a3416bbb9247c2171d2eb619c4f2d388de3ded 1615748 qemu-kvm_0.12.5+dfsg-2_amd64.deb
 45b60c1b623ff16da346a8b1624456e97b0fb12a9f949c54d649c5662fa6a0a9 2817174 qemu-kvm-dbg_0.12.5+dfsg-2_amd64.deb
 8b7e6df15bf2a2e03065d22c6791af7259de4a93302372e1487e8b2c45b9eeae 11708 kvm_0.12.5+dfsg-2_amd64.deb
Files: 
 75f57400f4b6a7358ef788c3621f3ceb 1649 misc optional qemu-kvm_0.12.5+dfsg-2.dsc
 115609e63df142dd5c1accfb00d2bd7c 302793 misc optional qemu-kvm_0.12.5+dfsg-2.diff.gz
 8972304aa8920df7dd5c4597c0271114 1615748 misc optional qemu-kvm_0.12.5+dfsg-2_amd64.deb
 c5c2af4ac4dd34e866f019e14f873a8c 2817174 debug extra qemu-kvm-dbg_0.12.5+dfsg-2_amd64.deb
 68982a4bd479eebad527c4fabd687389 11708 oldlibs extra kvm_0.12.5+dfsg-2_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkx0/JcACgkQioOL5NhIDy5jgwCgnhcgUm1MUVyTfMzOmPrJIa70
kPYAmgKp2TvIpgGenBsF3dP+LDqX0LSV
=E8Rk
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.