Debian Bug report logs - #578635
dpkg: severe performance degradation at least on ext4 filesystems

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Sven Joachim <svenjoac@gmx.de>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: dpkg: severe performance degradation at least on ext4 filesystems

Date: Wed, 21 Apr 2010 14:19:41 +0200

Package: dpkg
Version: 1.15.7
Severity: important

The fix for #577756 reintroduced a severe performance degradation first
reported in [1].  On my system, unpacking the emacs23-common package
with a warm cache takes ~ 40 seconds, about seven times as long as with
1.15.5.6 and 1.15.6.1.


-- System Information:
Debian Release: squeeze/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: i386 (x86_64)

Kernel: Linux 2.6.34-rc5-nouveau (SMP w/2 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dpkg depends on:
ii  coreutils                     8.4-2      GNU core utilities
ii  libc6                         2.10.2-6   Embedded GNU C Library: Shared lib
ii  lzma                          4.43-14    Compression method of 7z format in

dpkg recommends no packages.

Versions of packages dpkg suggests:
ii  apt                           0.7.25.3   Advanced front-end for dpkg

-- no debconf information


1. http://lists.debian.org/debian-dpkg/2010/03/msg00029.html

Message #10 received at 578635@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Sven Joachim <svenjoac@gmx.de>, 578635@bugs.debian.org

Subject: Re: Bug#578635: dpkg: severe performance degradation at least on ext4 filesystems

Date: Tue, 11 May 2010 10:59:51 +0200

Hi,

On Wed, 21 Apr 2010, Sven Joachim wrote:
> The fix for #577756 reintroduced a severe performance degradation first
> reported in [1].  On my system, unpacking the emacs23-common package
> with a warm cache takes ~ 40 seconds, about seven times as long as with
> 1.15.5.6 and 1.15.6.1.

Can you try with the dpkg version that is in the branch pu/async-sync
of git://git.hadrons.org/git/debian/dpkg.git ?

I also attach the relevant patch (you can apply it on the master branch,
or even on the sid version if you prefer). Be sure to run autoreconf -f -i
after having applied the patch.

It would be nice to have a quick answer so that we can decide if we can
push that change in 1.15.7.2.

Cheers,
-- 
Raphaël Hertzog

Like what I do? Sponsor me: http://ouaza.com/wp/2010/01/05/5-years-of-freexian/
My Debian goals: http://ouaza.com/wp/2010/01/09/debian-related-goals-for-2010/

Message #15 received at 578635@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Sven Joachim <svenjoac@gmx.de>, 578635@bugs.debian.org

Subject: Re: Bug#578635: dpkg: severe performance degradation at least on ext4 filesystems

Date: Tue, 11 May 2010 11:01:17 +0200

[Message part 1 (text/plain, inline)]

On Tue, 11 May 2010, Raphael Hertzog wrote:
> I also attach the relevant patch (you can apply it on the master branch,
> or even on the sid version if you prefer). Be sure to run autoreconf -f -i
> after having applied the patch.

This time the patch is here. It's also here:
http://git.hadrons.org/?p=debian/dpkg.git;a=commitdiff;h=b405384cfeff583197198213f0e86f7accccf4ce

Cheers,
-- 
Raphaël Hertzog

Like what I do? Sponsor me: http://ouaza.com/wp/2010/01/05/5-years-of-freexian/
My Debian goals: http://ouaza.com/wp/2010/01/09/debian-related-goals-for-2010/

[patch (text/plain, attachment)]

Message #20 received at 578635@bugs.debian.org (full text, mbox, reply):

From: Sven Joachim <svenjoac@gmx.de>

To: Raphael Hertzog <hertzog@debian.org>

Cc: 578635@bugs.debian.org

Subject: Re: Bug#578635: dpkg: severe performance degradation at least on ext4 filesystems

Date: Tue, 11 May 2010 11:54:49 +0200

On 2010-05-11 10:59 +0200, Raphael Hertzog wrote:

> On Wed, 21 Apr 2010, Sven Joachim wrote:
>> The fix for #577756 reintroduced a severe performance degradation first
>> reported in [1].  On my system, unpacking the emacs23-common package
>> with a warm cache takes ~ 40 seconds, about seven times as long as with
>> 1.15.5.6 and 1.15.6.1.
>
> Can you try with the dpkg version that is in the branch pu/async-sync
> of git://git.hadrons.org/git/debian/dpkg.git ?
>
> I also attach the relevant patch (you can apply it on the master branch,
> or even on the sid version if you prefer). Be sure to run autoreconf -f -i
> after having applied the patch.

Applied this to the sid branch.

> It would be nice to have a quick answer so that we can decide if we can
> push that change in 1.15.7.2.

Definitely a vast improvement over 1.15.7.1.  A short test reveals that
with a hot cache there is a slowdown of ~30% compared to 1.15.5.6 on a
system without much other disk activity.  When there are lots of other
activities (initiated by unpacking a Linux kernel tarball in parallel),
the many sync() calls cause a slowdown of ~100%.  Still much better than
the ~700% slowdown in 1.15.7.1.

Cheers,
       Sven

Message #25 received at 578635@bugs.debian.org (full text, mbox, reply):

From: Jonathan Nieder <jrnieder@gmail.com>

To: Raphael Hertzog <hertzog@debian.org>, 578635@bugs.debian.org

Cc: Sven Joachim <svenjoac@gmx.de>

Subject: Re: Bug#578635: dpkg: severe performance degradation at least on ext4 filesystems

Date: Tue, 11 May 2010 08:36:02 -0500

Hi,

Raphael Hertzog wrote:

> Can you try with the dpkg version that is in the branch pu/async-sync
> of git://git.hadrons.org/git/debian/dpkg.git ?

Some rough numbers.

master+sid+asyncsync:

  cold cache 16.86user 4.21system 0:50.52elapsed 41%CPU (119major)pagefaults
  warm cache 16.82user 4.05system 0:43.72elapsed 47%CPU (77major)pagefaults

master+sid:

  cold cache 16.80user 4.36system 1:25.42elapsed 25%CPU (104major)pagefaults
  warm cache 16.75user 4.24system 1:19.49elapsed 26%CPU (64major)pagefaults

master+sid-fsync [1]:

  cold cache 17.02user 4.21system 0:42.79elapsed 50%CPU (85major)pagefaults
  warm cache 16.76user 3.95system 0:32.36elapsed 64%CPU (33major)pagefaults

lenny (1.14.29):

  cold cache 16.82user 4.32system 0:52.91elapsed 40%CPU (90major)pagefaults
  warm cache 16.72user 4.18system 0:45.77elapsed 46%CPU (58major)pagefaults

kernel: Linux 2.6.33-2-686 (Debian version 2.6.33-1~experimental.5)
filesystem: ext4 with data=ordered
disk: an old 4200 rpm 50 GiB laptop drive.
nothing interesting going on in the background.
all times above are best of 3.

I also tried inducing swapping in the background, but (because I am
too impatient to run enough trials) that didn’t yield stable numbers.
Roughly speaking, master+sid+asyncsync and master+sid-fsync were about
the same in that case: one dpkg run takes three minutes or so with
some huge margin of error.

So: a little faster than lenny, 25% slower than it could be by
deciding not to sync, 1½ times as fast as current sid.  Not bad.

Hope that helps,
Jonathan

[1] merge of current master with current sid with commits 8467206,
3927a9, 18b12083, 9cd41fd, 20fdb39, 8ccebf62, and 62668eb4 reverted.

Message #30 received at 578635@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: 578635@bugs.debian.org

Cc: cwillu@cwillu.com, trentbuck@gmail.com, Joey Hess <joey@kitenet.net>, btrfs-devel@oss.oracle.com

Subject: Re: Bug#578635: dpkg: severe performance degradation at least on ext4 filesystems

Date: Thu, 13 May 2010 06:52:15 +0200

Hi!

[ Context for people CCed, this is a bug report against dpkg due to it
  recently starting to use deferred fsync()+rename() and introducing a
  major performance degradation on ext4. I've taken the liberty to add
  people previously involved in the btrfs dpkg bug report. ]

On Tue, 2010-05-11 at 10:59:51 +0200, Raphael Hertzog wrote:
> On Wed, 21 Apr 2010, Sven Joachim wrote:
> > The fix for #577756 reintroduced a severe performance degradation first
> > reported in [1].  On my system, unpacking the emacs23-common package
> > with a warm cache takes ~ 40 seconds, about seven times as long as with
> > 1.15.5.6 and 1.15.6.1.

> Can you try with the dpkg version that is in the branch pu/async-sync
> of git://git.hadrons.org/git/debian/dpkg.git ?

The relevant patch is
<http://git.hadrons.org/?p=debian/dpkg/dpkg.git;a=commitdiff;h=b405384c>,
which on Linux switches from using an fsync() per file to a global
sync() per package. This seems to help quite a bit with ext4, as
reported by few people who have done some tests.

> It would be nice to have a quick answer so that we can decide if we can
> push that change in 1.15.7.2.

I just happened to read <http://lwn.net/Articles/358940/>, and was
wondering if this would affect badly on btrfs (by being slower with
sync() than fsync()), so it would be nice to see some tests w/ the
patch using the pu/async-sync branch, and w/o using dpkg 1.15.7,
1.15.7.1 or the sid branch. And probably by doing some parallel I/O
to see the effect of sync().

To build on a Debian based system, you just need to “autoreconf -f -i”
and then “debuild -us -uc -b”. If pre-built packages would make this
easier I can prepare some for requested architectures. If you are
using Ubuntu, Colin Watson has prepared packages on his PPA
<https://launchpad.net/~cjwatson/+archive/ppa> with the patch.

thanks,
guillem

Message #35 received at 578635-submitter@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: 578635-submitter@bugs.debian.org

Subject: Bug#578635 marked as pending

Date: Mon, 17 May 2010 19:31:41 +0000

tag 578635 pending
thanks

Hello,

Bug #578635 reported by you has been fixed in the Git repository. You can
see the changelog below, and you can check the diff of the fix at:

    http://git.debian.org/?p=dpkg/dpkg.git;a=commitdiff;h=0ff43ca

---
commit 0ff43ca667f05cf299364e5b9cff8abe4e05a903
Author: Guillem Jover <guillem@debian.org>
Date:   Tue May 11 09:38:04 2010 +0200

    dpkg: On Linux use sync() instead of an fsync() per file
    
    Due to the performance degradation on ext4 file systems, as a
    workaround on Linux, we use sync() which is synchronous, before
    rename() to make sure it's truly atomic.
    
    Closes: #578635

diff --git a/debian/changelog b/debian/changelog
index 710b0dc..80e7300 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -35,6 +35,8 @@ dpkg (1.15.7.2) UNRELEASED; urgency=low
     Kyle Moffett <Kyle.D.Moffett@boeing.com>. Closes: #568123, #575158
   * Fix dpkg --root by properly stripping again the root directory from the
     path of the maintainer script to execute. Closes: #580984
+  * On Linux use sync() instead of an fsync() per file on deferred extraction,
+    to workaround performance degradation on ext4. Closes: #578635
 
   [ Gerfried Fuchs ]
   * Fix syntax error in dpkg-name. Closes: #581315

Message #40 received at 578635-close@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: 578635-close@bugs.debian.org

Subject: Bug#578635: fixed in dpkg 1.15.7.2

Date: Wed, 19 May 2010 06:47:20 +0000

Source: dpkg
Source-Version: 1.15.7.2

We believe that the bug you reported is fixed in the latest version of
dpkg, which is due to be installed in the Debian FTP archive:

dpkg-dev_1.15.7.2_all.deb
  to main/d/dpkg/dpkg-dev_1.15.7.2_all.deb
dpkg_1.15.7.2.dsc
  to main/d/dpkg/dpkg_1.15.7.2.dsc
dpkg_1.15.7.2.tar.bz2
  to main/d/dpkg/dpkg_1.15.7.2.tar.bz2
dpkg_1.15.7.2_amd64.deb
  to main/d/dpkg/dpkg_1.15.7.2_amd64.deb
dselect_1.15.7.2_amd64.deb
  to main/d/dpkg/dselect_1.15.7.2_amd64.deb
libdpkg-dev_1.15.7.2_amd64.deb
  to main/d/dpkg/libdpkg-dev_1.15.7.2_amd64.deb
libdpkg-perl_1.15.7.2_all.deb
  to main/d/dpkg/libdpkg-perl_1.15.7.2_all.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 578635@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Guillem Jover <guillem@debian.org> (supplier of updated dpkg package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Wed, 19 May 2010 07:57:14 +0200
Source: dpkg
Binary: libdpkg-dev dpkg dpkg-dev libdpkg-perl dselect
Architecture: source amd64 all
Version: 1.15.7.2
Distribution: unstable
Urgency: low
Maintainer: Dpkg Developers <debian-dpkg@lists.debian.org>
Changed-By: Guillem Jover <guillem@debian.org>
Description: 
 dpkg       - Debian package management system
 dpkg-dev   - Debian package development tools
 dselect    - Debian package management front-end
 libdpkg-dev - Debian package management static library
 libdpkg-perl - Dpkg perl modules
Closes: 568123 575158 575410 578002 578635 579722 579724 580774 580984 581315
Changes: 
 dpkg (1.15.7.2) unstable; urgency=low
 .
   [ Raphaël Hertzog ]
   * Update dpkg-buildflags to respect $XDG_CONFIG_HOME and to use
     $XDG_CONFIG_HOME/dpkg/buildflags.conf by default.
   * Update deb-substvars(5) to codify how variables containing multiple
     lines must be managed.
   * Fix boolean evaluation of Dpkg::Version so that version 0 evaluates to
     false and dpkg-shlibdeps can strip the minimal version specification.
     Closes: #579724
     Document this behaviour in the API and add non-regression test to ensure
     it's kept.
   * Let dpkg-buildflags error out when a required parameter is missing.
     Closes: #579722
   * Add Bug-Ubuntu field in DEP-3 template provided in the automatic header
     of patches in 3.0 (quilt) source packages. Thanks to Benjamin Drung
     <bdrung@ubuntu.com> for the patch. Closes: #578002
   * Update deb-override(5) by removing references to usage of sections
     to place the packages on the mirrors and by indicating that the Debian
     policy offers a list of allowed values for section and priority.
     Closes: #575410
   * Update reference to triggers.txt.gz in dpkg-trigger(1) and deb-triggers(5)
     to match the new location. Closes: #580774
   * Drop mention of PKG_CONFIG_LIBDIR in dpkg-buildpackage(1), the feature has
     been removed in 1.15.6.
   * Rename /usr/lib/dpkg/maintscript-helper into
     /usr/bin/dpkg-maintscript-helper, it is a public interface even if working
     around known limitations.
   * Add "supports" command to dpkg-maintscript-helper to ensure the wanted
     command is supported before calling it.
 .
   [ Guillem Jover ]
   * Add powerpcspe support to ostable and triplettable.
     Thanks to Sebastian Andrzej Siewior <sebastian@breakpoint.cc> and
     Kyle Moffett <Kyle.D.Moffett@boeing.com>. Closes: #568123, #575158
   * Fix dpkg --root by properly stripping again the root directory from the
     path of the maintainer script to execute. Closes: #580984
   * On Linux use sync() instead of an fsync() per file on deferred extraction,
     to workaround performance degradation on ext4. Closes: #578635
 .
   [ Gerfried Fuchs ]
   * Fix syntax error in dpkg-name. Closes: #581315
Checksums-Sha1: 
 420ab2514738462b7b10213a3de84dff13df4dd6 1221 dpkg_1.15.7.2.dsc
 50e0f02d136df495f28ab522c7afc79f1dc8126e 4921845 dpkg_1.15.7.2.tar.bz2
 f6b0eb3be544d0fabb54ce41843048c0645a77ae 377240 libdpkg-dev_1.15.7.2_amd64.deb
 c9df22429998e8b19e7d99674fce687df519d695 1918510 dpkg_1.15.7.2_amd64.deb
 b694bd54de2b8166c72e5820c744f7ca3111fd20 792554 dselect_1.15.7.2_amd64.deb
 515a0a45eb4bf1d2b4ecbc1fbb2fd5fd74dcc07c 407772 dpkg-dev_1.15.7.2_all.deb
 fb1db2c154d53dc896b665bf36200a634097bb47 582860 libdpkg-perl_1.15.7.2_all.deb
Checksums-Sha256: 
 6db728bc70b1e99732f8f7801ce5760b5fccb1143d90b8d863b1e99b87474aa7 1221 dpkg_1.15.7.2.dsc
 d3235cbcd5580c4b8d83b6dcf3ab71a076178bcca327c1339a1f95a8324103a5 4921845 dpkg_1.15.7.2.tar.bz2
 b635959be15b837a4e3ffddb61ef606e8ae523ea8f77a42dd3bcf856d892975d 377240 libdpkg-dev_1.15.7.2_amd64.deb
 eb4499b7d46a275f23307f2ac2100c8b635884be2ca58d7d5fe2afdaebb72e0b 1918510 dpkg_1.15.7.2_amd64.deb
 b38f2454ae934d7c35e9404686c02290b54b941556643fde465d41d22116c0e1 792554 dselect_1.15.7.2_amd64.deb
 889c3311165f370e1251b00fec283b94f73c09add812bd2e20756bf94e16a0f7 407772 dpkg-dev_1.15.7.2_all.deb
 4d08e26e78a64771c334623ced12a62e58ab36c9f811b68dfacfe64acbaaf7b4 582860 libdpkg-perl_1.15.7.2_all.deb
Files: 
 0153b46014452aad73a53e76698a19b1 1221 admin required dpkg_1.15.7.2.dsc
 bcd5c89035049352df03ebc75464e807 4921845 admin required dpkg_1.15.7.2.tar.bz2
 b3ecd33aa1fb9b2650fa5bc5ff16c5a7 377240 libdevel optional libdpkg-dev_1.15.7.2_amd64.deb
 44002bb70fda5c3275b6801f62a45664 1918510 admin required dpkg_1.15.7.2_amd64.deb
 0fb773d6db419958469662c075bd83fd 792554 admin optional dselect_1.15.7.2_amd64.deb
 efb65b21888ab7f4ca4bc4b1448c6d8e 407772 utils optional dpkg-dev_1.15.7.2_all.deb
 980504122618351c753737f4f1925bd2 582860 perl optional libdpkg-perl_1.15.7.2_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkvzgfEACgkQuW9ciZ2SjJuSdgCg91mEwlOV6P6mWK8fjX/l4dxG
+fwAoPOGKwSe3i7GxFSFN5CvTusw1oy+
=vTo4
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.