Debian Bug report logs - #605384
d-i should use dpkg --force-unsafe-io to optimize installation time

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: serious performance regression with ext4

Date: Fri, 26 Nov 2010 10:32:57 +0100

Package: dpkg
Version: 1.15.8.6
Severity: important

Hi!

I'm using ext4 with the default mount option and the fs was created with
the default settings from /etc/mke2fs.conf.
Since the latest upgrade, performance suffered badly.

E.g. installing vim-runtime took ~8 secs with 1.15.8.5, now it takes
~44secs.

It was suggested that I use the nodelalloc mount option for ext4.
But that not only takes away some of the benefits of ext4, it also
requires explicit configuration.

dpkg should work properly(with good performance) out-of-the-box on ext4.

Cheers,
Michael

-- System Information:
Debian Release: squeeze/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (200, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.32-5-686 (SMP w/1 CPU core)
Locale: LANG=de_DE.utf8, LC_CTYPE=de_DE.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dpkg depends on:
ii  coreutils               8.5-1            GNU core utilities
ii  libbz2-1.0              1.0.5-6          high-quality block-sorting file co
ii  libc6                   2.11.2-7         Embedded GNU C Library: Shared lib
ii  libselinux1             2.0.96-1         SELinux runtime shared libraries
ii  xz-utils                5.0.0-2          XZ-format compression utilities
ii  zlib1g                  1:1.2.3.4.dfsg-3 compression library - runtime

dpkg recommends no packages.

Versions of packages dpkg suggests:
ii  apt                           0.8.9      Advanced front-end for dpkg

-- no debconf information

Message #10 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Michael Biebl <biebl@debian.org>, 605009@bugs.debian.org

Cc: tytso@debian.org, debian-devel@lists.debian.org, debian-kernel@lists.debian.org, debian-boot@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Fri, 26 Nov 2010 15:53:27 +0100

Hi,

adding debian-devel, debian-boot, debian-kernel and Theodore Y. Ts'o in CC
because I'm fed up with this problem. Sorry for the massive crosspost, you
might want to follow up only on -devel and on the bug report.

Some clones/reassign should probably result from this discussion anyway.

On Fri, 26 Nov 2010, Michael Biebl wrote:
> I'm using ext4 with the default mount option and the fs was created with
> the default settings from /etc/mke2fs.conf.
> Since the latest upgrade, performance suffered badly.
> 
> E.g. installing vim-runtime took ~8 secs with 1.15.8.5, now it takes
> ~44secs.
>
> It was suggested that I use the nodelalloc mount option for ext4.
> But that not only takes away some of the benefits of ext4, it also
> requires explicit configuration.
> 
> dpkg should work properly(with good performance) out-of-the-box on ext4.

There's currently no way we're going to fix this.

It all started with report of corrupted (zero-length) files on ext4/ubifs
(see http://bugs.debian.org/430958). We did the right thing to fix this
which is to call fsync() on the fly on each file that dpkg unpacks. That
was introduced in dpkg 1.15.6. (Ubuntu had more than 80 bug reports
related to this and Debian very few because we don't use ext4 by default)

That was disastrous in terms of performance, so we then grouped the
fsync()+rename() calls at the end of the unpack phase before updating the
dpkg database. That can be witnessed in dpkg 1.15.7.

That was ok everywhere except on ext4. For some unknown reasons, with the
nodelalloc option the performance are again reasonable (but we discovered
that only recently). Ubuntu is using ext4 by default and this performance
was not acceptable and we tried to work-around the problem by replacing
all the fsync() by a single sync() on Linux only (because Linux's sync()
is synchronous while it's not necessarily the case on other systems). This
was enabled in dpkg 1.15.7.2 in response to http://bugs.debian.org/578635.

Now using sync() appears to be a bad choice since it resulted in RC bugs
like http://bugs.debian.org/595927 and http://bugs.debian.org/600075 and
was an annoyance for people using dpkg on tmpfs to get super-high speed
(http://bugs.debian.org/588339).

So we reverted that hackish work-around of using sync() and we're back
to the situation where dpkg is very slow on ext4. But we have added a new
option --force-unsafe-io that can be used when data safety is not
important (in a temporary chroot, during initial installation, etc.) as
requested in http://bugs.debian.org/584254.

Where does that leaves us? Here are various options to consider (they are
not necessarily mutually exclusive):

----
1/ This problem and the known work-arounds should be documented in the
release notes.

2/ d-i should be modified to use --force-unsafe-io if it detects a dpkg
version >= 1.15.8.6.

3/ d-i might want to setup the nodelalloc option by default when a
partition is formatted with ext4 and mounted as / or /usr.

4/ Theodore might want to find out why ext4 is behaving so badly under
this usage pattern while ext3 doesn't... i.e. fix
https://bugzilla.kernel.org/show_bug.cgi?id=15910
Or at least suggest another usage pattern which result in the same
guaranty for dpkg but without the poor performance (and sync() is not the
right answer as experience showed us).

Note that doing N x fsync() followed by N x rename() is not giving us any
better result that doing N x (fsync()+rename()).

Note that calling posix_fallocate() before writing the files that are
fsynced() did not help either (Mike Hommey thought it could be a way to
emulate nodelalloc at the dpkg level only but apparently not).

What has not been tried: reordering the fsync() based on FIEMAP
information. Or using fdatasync() instead of fsync() and calling fsync()
on directories.

5/ maybe the debian-kernel team wants to discuss the issue on LKML. Both
for the bad ext4 performances (see above) and the (incorrect?) behaviour
of sync() which is never finishing under important I/O loads (cf
https://bugzilla.kernel.org/show_bug.cgi?id=18632).
----

But right now from the point of view of dpkg maintainers, this bug is a
"wontfix" at our level.

Just to sum up what dpkg --unpack does in 1.15.8.6:
1/ set the package status as half-installed/reinst-required
2/ extract all the new files as *.dpkg-new
3/ for all the unpacked files: fsync(foo.dpkg-new) followed by
   rename(foo.dpkg-new, foo)
4/ set the package status as unpacked

Note that the directories are not fsynced() because we mainly don't care
if the rename is recorded or not, as long as the installed file always has
the content of either the old or the new file. This could be fixed but is
unlikely to help in getting better performances I guess.

The only thing we want to achieve is that:
- when the package is set to unpacked status, all changes have been
  committed
- when the process is abruptly interrupted, we don't leave corrupted files
  around (like unwanted empty files)

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Message #15 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Mathias Behrle <mathiasb@mbsolutions.selfip.biz>

To: debian-devel@lists.debian.org

Cc: 605009@bugs.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Fri, 26 Nov 2010 16:22:20 +0100

[Message part 1 (text/plain, inline)]

* Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri, 26
  Nov 2010 15:53:27 +0100):

> That was ok everywhere except on ext4.

JFTR: I am experiencing those problems as well on XFS.

Cheers,

-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0x8405BBF6

[signature.asc (application/pgp-signature, attachment)]

Message #20 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Aron Xu <happyaron.xu@gmail.com>

To: 605009@bugs.debian.org, debian-devel@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Fri, 26 Nov 2010 23:26:18 +0800

Low performance with Btrfs as well, :(

(Even Btrfs is not supported in squeeze, I think this could help on
digging whether it is a more generic problem than EXT4 only.)

-- 
Regards,
Aron Xu

Message #25 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Mathias Behrle <mathiasb@mbsolutions.selfip.biz>, 605009@bugs.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Fri, 26 Nov 2010 16:33:06 +0100

Hi,

On Fri, 26 Nov 2010, Mathias Behrle wrote:
> * Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri, 26
>   Nov 2010 15:53:27 +0100):
> 
> > That was ok everywhere except on ext4.
> 
> JFTR: I am experiencing those problems as well on XFS.

Can you give us figures to quantify the slowdown that you experience?
Please compare dpkg 1.15.8.5 and 1.15.8.6.

(I suppose that's the problem you're referring to)

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Message #30 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Aron Xu <happyaron.xu@gmail.com>

To: Raphael Hertzog <hertzog@debian.org>

Cc: 605009@bugs.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Sat, 27 Nov 2010 00:27:35 +0800

I'm reinstalling vim-runtime using the following command to do this benchmark:
$ time sudo dpkg -i vim-runtime_7.2.330-1ubuntu3_all.deb

Results:
1.15.8.5: 3.02s user 1.96s system 16% cpu 30.364 total
1.15.8.6: 3.48s user 2.86s system 9% cpu 1:04.43 total

My system infomation:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 10.04.1 LTS
Release:        10.04
Codename:       lucid

$ uname -r
Linux aron-desktop 2.6.37-5-generic #14~lucid1-Ubuntu SMP Sat Nov 20
06:26:57 UTC 2010 i686 GNU/Linux

/ is btrfs mounted with (rw,compress)

The hard drive is an old PATA one so the performance is not comparable
to the mainstream SATA ones.


-- 
Regards,
Aron Xu

Message #35 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Mathias Behrle <mathiasb@mbsolutions.selfip.biz>

To: debian-devel@lists.debian.org

Cc: 605009@bugs.debian.org

Subject: Fw: Bug#605009: serious performance regression with ext4

Date: Fri, 26 Nov 2010 22:15:06 +0100

[Message part 1 (text/plain, inline)]

* Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri, 26
  Nov 2010 16:33:06 +0100):

> On Fri, 26 Nov 2010, Mathias Behrle wrote:
> > * Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri,
> > 26 Nov 2010 15:53:27 +0100):
> > 
> > > That was ok everywhere except on ext4.
> > 
> > JFTR: I am experiencing those problems as well on XFS.
> 
> Can you give us figures to quantify the slowdown that you experience?
> Please compare dpkg 1.15.8.5 and 1.15.8.6.
> 
> (I suppose that's the problem you're referring to)

I have to correct my previous mail.

I am experiencing since some time a substantial slowdown on my XFS file
systems (could also be related to upgrading to kernel 2.6.32). This slowdown is
also quite perceivable on system upgrades with apt-get/aptitude. Now for the
first time Bug#605009 came to my knowledge and I guessed it could also be a
cause for the meanwhile rather painful upgrades on my desktop machine.

I just upgraded my testing machine to dpkg 1.15.8.6. and tested with
'aptitude -o DPkg::options::="--force-unsafe-io"' and see that my guess
was at least partly wrong. Unpacking of the debs still causes very heavy load
on the harddisk while final install seems to be faster.

So the problems I am facing with XFS are probably at least not closely related
to #605009.

Sorry for the noise,

-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0x8405BBF6

[signature.asc (application/pgp-signature, attachment)]

Message #40 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Ted Ts'o <tytso@mit.edu>

To: Michael Biebl <biebl@debian.org>, 605009@bugs.debian.org, tytso@debian.org, debian-devel@lists.debian.org, debian-kernel@lists.debian.org, debian-boot@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Fri, 26 Nov 2010 16:52:54 -0500

On Fri, Nov 26, 2010 at 03:53:27PM +0100, Raphael Hertzog wrote:
> Just to sum up what dpkg --unpack does in 1.15.8.6:
> 1/ set the package status as half-installed/reinst-required
> 2/ extract all the new files as *.dpkg-new
> 3/ for all the unpacked files: fsync(foo.dpkg-new) followed by
>    rename(foo.dpkg-new, foo)

What are you doing?

1) Suppose package contains files "a", "b", and "c".  Which are you
doing?

a)  extract a.dpkg-new ; fsync(a.dpkg-new); rename(a.dpkg-new, a);
    extract b.dpkg-new ; fsync(b.dpkg-new); rename(b.dpkg-new, b);
    extract c.dpkg-new ; fsync(c.dpkg-new); rename(c.dpkg-new, c);

or

b)  extract a.dpkg-new ; fsync(a.dpkg-new);
    extract b.dpkg-new ; fsync(b.dpkg-new);
    extract c.dpkg-new ; fsync(c.dpkg-new);
    rename(a.dpkg-new, a);
    rename(b.dpkg-new, b);
    rename(c.dpkg-new, c);

or

c)  extract(a.dpkg-new);
    extract(b.dpkg-new);
    extract(c.dpkg-new);
    fsync(a.dpkg-new);
    fsync(b.dpkg-new);
    fsync(c.dpkg-new);
    rename(a.dpkg-new, a);
    rename(b.dpkg-new, b);
    rename(c.dpkg-new, c);


(c) will perform the best for most file systems, including ext4.  As a
further optimization, if "b" and "c" does not exist, of course it
would be better to extract into "b" and "c" directly, and skip the
rename, i.e.:

d)  extract(a.dpkg-new);
    extract(b);			# assuming the file "b" does not yet exist
    extract(c);			# assuming the file "c" does not yet exist
    fsync(a.dpkg-new);
    fsync(b);
    fsync(c);
    rename(a.dpkg-new, a);

... and then set the package status as unpacked.

I am guessing you are doing (a) today --- am I right?  (c) or (d)
would be best.

						- Ted

Message #45 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Jonathan Nieder <jrnieder@gmail.com>

To: Ted Ts'o <tytso@mit.edu>, 605009@bugs.debian.org

Cc: Michael Biebl <biebl@debian.org>, tytso@debian.org, debian-devel@lists.debian.org, debian-kernel@lists.debian.org, debian-boot@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Sat, 27 Nov 2010 01:58:31 -0600

Hi Ted,

Ted Ts'o wrote:

> 1) Suppose package contains files "a", "b", and "c".  Which are you
> doing?
> 
> a)  extract a.dpkg-new ; fsync(a.dpkg-new); rename(a.dpkg-new, a);
>     extract b.dpkg-new ; fsync(b.dpkg-new); rename(b.dpkg-new, b);
>     extract c.dpkg-new ; fsync(c.dpkg-new); rename(c.dpkg-new, c);
> 
> or
> 
> b)  extract a.dpkg-new ; fsync(a.dpkg-new);
>     extract b.dpkg-new ; fsync(b.dpkg-new);
>     extract c.dpkg-new ; fsync(c.dpkg-new);
>     rename(a.dpkg-new, a);
>     rename(b.dpkg-new, b);
>     rename(c.dpkg-new, c);
> 
> or
> 
> c)  extract(a.dpkg-new);
>     extract(b.dpkg-new);
>     extract(c.dpkg-new);
>     fsync(a.dpkg-new);
>     fsync(b.dpkg-new);
>     fsync(c.dpkg-new);
>     rename(a.dpkg-new, a);
>     rename(b.dpkg-new, b);
>     rename(c.dpkg-new, c);
> 
> 
> (c) will perform the best for most file systems, including ext4.
[...]
> I am guessing you are doing (a) today --- am I right?  (c) or (d)
> would be best.

We are doing (c) today.

Regards,
Jonathan

Message #50 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Mathias Behrle <mathiasb@mbsolutions.selfip.biz>

To: Raphael Hertzog <hertzog@debian.org>

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Fri, 26 Nov 2010 22:13:10 +0100

[Message part 1 (text/plain, inline)]

* Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri, 26
  Nov 2010 16:33:06 +0100):

> On Fri, 26 Nov 2010, Mathias Behrle wrote:
> > * Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri,
> > 26 Nov 2010 15:53:27 +0100):
> > 
> > > That was ok everywhere except on ext4.
> > 
> > JFTR: I am experiencing those problems as well on XFS.
> 
> Can you give us figures to quantify the slowdown that you experience?
> Please compare dpkg 1.15.8.5 and 1.15.8.6.
> 
> (I suppose that's the problem you're referring to)

I have to correct my previous mail.

I am experiencing since some time a substantial slowdown on my XFS file
systems (could also be related to upgrading to kernel 2.6.32). This slowdown is
also quite perceivable on system upgrades with apt-get/aptitude. Now for the
first time Bug#605009 came to my knowledge and I guessed it could also be a
cause for the meanwhile rather painful upgrades on my desktop machine.

I just upgraded my testing machine to dpkg 1.15.8.6. and tested with
'aptitude -o DPkg::options::="--force-unsafe-io"' and see that my guess
was at least partly wrong. Unpacking of the debs still causes very heavy load
on the harddisk while final install seems to be faster.

So the problems I am facing with XFS are probably at least not closely related
to #605009.

Sorry for the noise,

-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0x8405BBF6

[signature.asc (application/pgp-signature, attachment)]

Message #55 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Jonathan Nieder <jrnieder@gmail.com>, 605009@bugs.debian.org

Cc: Ted Ts'o <tytso@mit.edu>, Michael Biebl <biebl@debian.org>, tytso@debian.org, debian-devel@lists.debian.org, debian-kernel@lists.debian.org, debian-boot@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Sat, 27 Nov 2010 09:53:46 +0100

On Sat, 27 Nov 2010, Jonathan Nieder wrote:
> > c)  extract(a.dpkg-new);
> >     extract(b.dpkg-new);
> >     extract(c.dpkg-new);
> >     fsync(a.dpkg-new);
> >     fsync(b.dpkg-new);
> >     fsync(c.dpkg-new);
> >     rename(a.dpkg-new, a);
> >     rename(b.dpkg-new, b);
> >     rename(c.dpkg-new, c);
> > 
> > 
> > (c) will perform the best for most file systems, including ext4.
> [...]
> > I am guessing you are doing (a) today --- am I right?  (c) or (d)
> > would be best.
> 
> We are doing (c) today.

Actually we are doing this:
    extract(a.dpkg-new);
    extract(b.dpkg-new);
    extract(c.dpkg-new);
    fsync(a.dpkg-new);
    rename(a.dpkg-new, a);
    fsync(b.dpkg-new);
    rename(b.dpkg-new, b);
    fsync(c.dpkg-new);
    rename(c.dpkg-new, c);

But as I said, I tried (c) and it's not performing noticably better than
the above.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Message #60 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Ted Ts'o <tytso@mit.edu>

Cc: Michael Biebl <biebl@debian.org>, 605009@bugs.debian.org, tytso@debian.org, debian-devel@lists.debian.org, debian-kernel@lists.debian.org, debian-boot@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Sat, 27 Nov 2010 10:40:18 +0100

Hi!

On Fri, 2010-11-26 at 16:52:54 -0500, Ted Ts'o wrote:
> On Fri, Nov 26, 2010 at 03:53:27PM +0100, Raphael Hertzog wrote:
> > Just to sum up what dpkg --unpack does in 1.15.8.6:
> > 1/ set the package status as half-installed/reinst-required
> > 2/ extract all the new files as *.dpkg-new
> > 3/ for all the unpacked files: fsync(foo.dpkg-new) followed by
> >    rename(foo.dpkg-new, foo)

> What are you doing?

We already had this conversation some time ago in
<https://bugzilla.kernel.org/show_bug.cgi?id=15910>.

> 1) Suppose package contains files "a", "b", and "c".  Which are you
> doing?

[...]

Anyway, dpkg is currently doing the variation on c that Raphaël
posted, including making backups so that it can rollback the entire
package if something goes wrong.

> (c) will perform the best for most file systems, including ext4.

Well it does not, and that's also what was mentioned in the bug
report. Something we've found out recently (as Raphaël mentioned too)
is that with nodelalloc the performance issues *and* the zero-length
issues disappear, which seems like a clear win to me, and so IMO
changing the default file system mount option to nodelalloc seems to
be the way to go.

> As a further optimization, if "b" and "c" does not exist, of course
> it would be better to extract into "b" and "c" directly, and skip the
> rename, i.e.:

> d)  extract(a.dpkg-new);
>     extract(b);			# assuming the file "b" does not yet exist
>     extract(c);			# assuming the file "c" does not yet exist
>     fsync(a.dpkg-new);
>     fsync(b);
>     fsync(c);
>     rename(a.dpkg-new, a);
> 
> ... and then set the package status as unpacked.

That would make possible for partial files to appear on their final path
and thus available for external use while they are being extracted. I
don't think that's a good idea.

regards,
guillem

Message #65 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Mathias Behrle <mathiasb@mbsolutions.selfip.biz>

To: debian-devel@lists.debian.org

Cc: 605009@bugs.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Sat, 27 Nov 2010 11:45:04 +0100

[Message part 1 (text/plain, inline)]

* Betr.: " Fw: Bug#605009: serious performance regression with ext4" (Fri, 26
  Nov 2010 22:15:06 +0100):

> * Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri, 26
>   Nov 2010 16:33:06 +0100):
> 
> > On Fri, 26 Nov 2010, Mathias Behrle wrote:
> > > * Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri,
> > > 26 Nov 2010 15:53:27 +0100):
> > > 
> > > > That was ok everywhere except on ext4.
> > > 
> > > JFTR: I am experiencing those problems as well on XFS.
> > 
> > Can you give us figures to quantify the slowdown that you experience?
> > Please compare dpkg 1.15.8.5 and 1.15.8.6.

Finally some real numbers:

command:
time dpkg -i openjdk-6-doc_6b18-1.8.2-4_all.deb


1.15.8.6:
=========
dpkg 1.15.8.6 with force-unsafe-io (1. install):
real    8m12.093s
user    0m1.737s
sys     0m2.290s

dpkg 1.15.8.6 with force-unsafe-io (2. run = replace):
real    13m57.933s
user    0m1.657s
sys     0m3.596s

dpkg 1.15.8.6 without force-unsafe-io (3. run = replace):
real    19m50.008s
user    0m1.940s
sys     0m4.240s


1.15.8.5:
=========
dpkg 1.15.8.5 (4. run = replace):
real    15m13.063s
user    0m1.723s
sys     0m6.453s


Seems, that 1.15.8.5 performs much better than 1.15.8.6 (without
force-unsafe-io).

Filesystem:
/ xfs     defaults,noatime,nodiratime,logbufs=8
on an lvm volume


Cheers,
-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0x8405BBF6

[signature.asc (application/pgp-signature, attachment)]

Message #70 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Olaf van der Spek <olafvdspek@gmail.com>

To: "Ted Ts'o" <tytso@mit.edu>

Cc: Michael Biebl <biebl@debian.org>, 605009@bugs.debian.org, tytso@debian.org, debian-devel@lists.debian.org, debian-kernel@lists.debian.org, debian-boot@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Sat, 27 Nov 2010 15:54:12 +0100

On Fri, Nov 26, 2010 at 10:52 PM, Ted Ts'o <tytso@mit.edu> wrote:
> I am guessing you are doing (a) today --- am I right?  (c) or (d)
> would be best.

Are there any plans to provide an API for atomic (non-durable) file
updates, not involving fsync?
Would be simpler (for apps), faster and more general (because it makes
less assumptions).

Olaf

Message #75 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Ted Ts'o <tytso@mit.edu>

To: Jonathan Nieder <jrnieder@gmail.com>, 605009@bugs.debian.org, Michael Biebl <biebl@debian.org>, tytso@debian.org, debian-devel@lists.debian.org, debian-kernel@lists.debian.org, debian-boot@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Sun, 28 Nov 2010 23:11:52 -0500

[Message part 1 (text/plain, inline)]

I did some experimenting, and I figured out what was going on.  You're
right, (c) doesn't quite work, because delayed allocation meant that
the writeout didn't take place until the fsync() for each file
happened.  I didn't see this at first; my apologies.

However, this *does* work:

    extract(a);
    sync_file_range(fd.a, 0, 0, SYNC_FILE_RANGE_WRITE); 
    extract(b.dpkg-new);
    sync_file_range(fd.b, 0, 0, SYNC_FILE_RANGE_WRITE); 
    extract(c.dpkg-new);
    sync_file_range(fd.c, 0, 0, SYNC_FILE_RANGE_WRITE); 

    sync_file_range(fd.a, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 
    sync_file_range(fd.b, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 
    sync_file_range(fd.c, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 

    fdatasync(a);
    fdatasync(b.dpkg-new);
    fdatasync(c.dpkg-new);

    rename(b.dpkg-new, b);
    rename(c.dpkg-new, c);

This assumes that files b and c existed beforehand, but a is a new file.

What's going on here?  sync_file_range() is a Linux specific system
call that has been around for a while.  It allows program to control
when writeback happens in a very low-level fashion.  The first set of
sync_file_range() system calls causes the system to start writing back
each file once it has finished being extracted.  It doesn't actually
wait for the write to finish; it just starts the writeback.

The second series of sync_file_range() calls, with the operation
SYNC_FILE_RANGE_WAIT_BEFORE, will block until the previously initiated
writeback has completed.  This basically ensures that the delayed
allocation has been resolved; that is, the data blocks have been
allocated and written, and the inode updated (in memory), but not
necessarily pushed out to disk.

The fdatasync() call will actually force the inode to disk.  In the
case of the ext4 file system, the first fdatasync() will actually push
all of the inodes to disk, and all of the subsequent fdatasync() calls
are in fact no-ops (assuming that files 'a', 'b', and 'c' are all on
the same file system).  But what it means is that it minimizes the
number of (heavyweight) jbd2 commits to a minimum.

It uses a linux-specific system call --- sync_file_range --- but the
result should be faster performance across the board for all file
systems.  So I don't consider this an ext4-specific hack, although it
probably does makes things faster for ext4 more than any other file
system.

I've attached the program I used to test and prove this mechanism, as
well as the kernel tracepoint script I used to debug why (c) wasn't
working, which might be of interest to folks on debian-kernel.
Basically it's a demonstration of how cool ftrace is.  :-)

But using this program on a file system composed of a 5400rpm laptop
drive running LVM and LUKS, I get:

mass-sync-tester -d:	dpkg current: time:  0.83/ 0.01/ 0.00

versus

mass-sync-tester -n:	dpkg fixed: time:  0.07/ 0.00/ 0.01

	       	 	       	      	   - Ted

[mass-sync-tester.c (text/x-csrc, attachment)]

[do-sync-test (text/plain, attachment)]

Message #80 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Ted Ts'o <tytso@mit.edu>

Cc: Jonathan Nieder <jrnieder@gmail.com>, 605009@bugs.debian.org, Michael Biebl <biebl@debian.org>, tytso@debian.org, debian-devel@lists.debian.org, debian-kernel@lists.debian.org, debian-boot@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Mon, 29 Nov 2010 07:18:17 +0100

[Message part 1 (text/plain, inline)]

Hi Ted!

On Sun, 2010-11-28 at 23:11:52 -0500, Ted Ts'o wrote:
> I did some experimenting, and I figured out what was going on.  You're
> right, (c) doesn't quite work, because delayed allocation meant that
> the writeout didn't take place until the fsync() for each file
> happened.  I didn't see this at first; my apologies.

Thanks for the analysis.

> However, this *does* work:
> 
>     extract(a);
>     sync_file_range(fd.a, 0, 0, SYNC_FILE_RANGE_WRITE); 
>     extract(b.dpkg-new);
>     sync_file_range(fd.b, 0, 0, SYNC_FILE_RANGE_WRITE); 
>     extract(c.dpkg-new);
>     sync_file_range(fd.c, 0, 0, SYNC_FILE_RANGE_WRITE); 

The man page and the kernel sources seem to indicate this might block
depending on the size of the request queue?

>     sync_file_range(fd.a, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 
>     sync_file_range(fd.b, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 
>     sync_file_range(fd.c, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 

>     fdatasync(a);
>     fdatasync(b.dpkg-new);
>     fdatasync(c.dpkg-new);
> 
>     rename(b.dpkg-new, b);
>     rename(c.dpkg-new, c);

> What's going on here?  sync_file_range() is a Linux specific system
> call that has been around for a while.  It allows program to control
> when writeback happens in a very low-level fashion.  The first set of
> sync_file_range() system calls causes the system to start writing back
> each file once it has finished being extracted.  It doesn't actually
> wait for the write to finish; it just starts the writeback.

Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
instead, skimming over the kernel source seems to indicate it might
end up doing more or less the same thing but in a portable way?

Could someone with ext4/btrfs/xfs/etc test w/ and w/o the attached patch
against dpkg?

thanks,
guillem

[dpkg_fadvise.patch (text/x-diff, attachment)]

Message #85 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Jonathan Nieder <jrnieder@gmail.com>

To: 605009@bugs.debian.org

Cc: Ted Ts'o <tytso@mit.edu>, debian-kernel@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Mon, 29 Nov 2010 00:24:58 -0600

(pruned cc list)

Guillem Jover wrote:

> Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
> instead, skimming over the kernel source seems to indicate it might
> end up doing more or less the same thing but in a portable way?

Probably a silly question, but what does "The specified data will not
be accessed in the near future" have to do with preventing delayed
allocation?

Put another way: if this works now, is it likely to continue to work?

Message #90 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Jonathan Nieder <jrnieder@gmail.com>

To: 605009@bugs.debian.org

Cc: Ted Ts'o <tytso@mit.edu>, debian-kernel@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Mon, 29 Nov 2010 01:01:41 -0600

Hi Guillem,

Guillem Jover wrote:

> Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
> instead, skimming over the kernel source seems to indicate it might
> end up doing more or less the same thing but in a portable way?
> 
> Could someone with ext4/btrfs/xfs/etc test w/ and w/o the attached patch
> against dpkg?

ext4: yes, this brings the running time down from 34sec to 10.5sec,
just like sync_file_range with SYNC_FILE_RANGE_WRITE does.

Regards,
Jonathan

Message #95 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Jonathan Nieder <jrnieder@gmail.com>, 605009@bugs.debian.org

Cc: Ted Ts'o <tytso@mit.edu>, debian-kernel@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Mon, 29 Nov 2010 14:16:02 +0100

On Mon, 29 Nov 2010, Jonathan Nieder wrote:
> Guillem Jover wrote:
> > Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
> > instead, skimming over the kernel source seems to indicate it might
> > end up doing more or less the same thing but in a portable way?
> 
> Probably a silly question, but what does "The specified data will not
> be accessed in the near future" have to do with preventing delayed
> allocation?

It means we don't need to keep it in RAM since we're not going to
read/modifiy it again in the near future. Thus the writeback can be
started right now since delaying it will not save us anything.

At least that's the way I understand the situation.

> Put another way: if this works now, is it likely to continue to work?

Well, it will always work (the code is unlikely to introduce failures),
but the resulting behaviour is entirely up to the kernel to decide. So
there's no guaranty that the optimization will last.

On the other hand, the whole point of posix_fadvise() is to give hints to
the kernel so that he can decide on the best course of action. So I hope
the interpretation above is one the main motivation behind that hint.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Message #100 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Ian Jackson <ijackson@chiark.greenend.org.uk>

To: Olaf van der Spek <olafvdspek@gmail.com>

Cc: "Ted Ts'o" <tytso@mit.edu>, Michael Biebl <biebl@debian.org>, 605009@bugs.debian.org, tytso@debian.org, debian-devel@lists.debian.org, debian-kernel@lists.debian.org, debian-boot@lists.debian.org

Subject: Re: Bug#605009: serious performance regression with ext4

Date: Mon, 29 Nov 2010 13:22:25 +0000

Olaf van der Spek writes ("Re: Bug#605009: serious performance regression with ext4"):
> On Fri, Nov 26, 2010 at 10:52 PM, Ted Ts'o <tytso@mit.edu> wrote:
> > I am guessing you are doing (a) today --- am I right? =C2=A0(c) or (d)
> > would be best.
> 
> Are there any plans to provide an API for atomic (non-durable) file
> updates, not involving fsync?

Yes.  Such an API has already been defined by POSIX, SuSv3, et al.
It's called "rename".

dpkg used it correctly before filesystem bugs (and associated
intransigence by fs maintainers[1]) forced the current dpkg
maintainers to add a whole pile of pointless fsyncs.

dpkg does _not_ need durable updates; it just needs atomicity and
correctness.  If after a crash the system is rewound to some earlier
state that's absolutely fine.

I'm told that the Linux fs maintainers have now accepted that 
   open("file.new",O_CREAT);
   write();
   close();
   rename("file.new","file");
should not result, even after a crash, in "file" containing garbage.

If this is the case then all the fsyncs can be taken out again.

Ian.

[1] The view that fsync is the new IAC DONT RANDOMLY-LOSE.

Message #105 received at 605009@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: debian-boot@lists.debian.org, 605009@bugs.debian.org

Subject: d-i should use dpkg --force-unsafe-io

Date: Mon, 29 Nov 2010 14:31:09 +0100

clone 605009 -1
reassign -1 base-installer
retitle -1 d-i should use dpkg --force-unsafe-io to optimize installation time
severity -1 wishlist
thanks

I put severity wishlist but it would still be nice if this could be
implemented for squeeze. It should not be a major change.

On Fri, 26 Nov 2010, Raphael Hertzog wrote:
> 2/ d-i should be modified to use --force-unsafe-io if it detects a dpkg
> version >= 1.15.8.6.

dpkg has become slower due to the usage of fsync() to ensure data
consistency in case of crashes/power failures.

In the context of d-i, a crash means restarting the installation 
so it's not a big deal to not use fsync() and on the opposite, it's rather
important that it goes fast...

So it would be nice if base-installer could auto-detect whether dpkg >=
1.15.8.6 is used and if yes, then it could add a temporary configuration
file in /etc/dpkg.cfg.d/ with "force-unsafe-io" inside.

That file should obviously be dropped at the end of the installation.

Thanks.
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Message #122 received at 605384-close@bugs.debian.org (full text, mbox, reply):

From: Joey Hess <joeyh@debian.org>

To: 605384-close@bugs.debian.org

Subject: Bug#605384: fixed in base-installer 1.121

Date: Wed, 24 Aug 2011 23:32:11 +0000

Source: base-installer
Source-Version: 1.121

We believe that the bug you reported is fixed in the latest version of
base-installer, which is due to be installed in the Debian FTP archive:

base-installer_1.121.dsc
  to main/b/base-installer/base-installer_1.121.dsc
base-installer_1.121.tar.gz
  to main/b/base-installer/base-installer_1.121.tar.gz
base-installer_1.121_all.udeb
  to main/b/base-installer/base-installer_1.121_all.udeb
bootstrap-base_1.121_i386.udeb
  to main/b/base-installer/bootstrap-base_1.121_i386.udeb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 605384@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Joey Hess <joeyh@debian.org> (supplier of updated base-installer package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Wed, 24 Aug 2011 19:11:09 -0400
Source: base-installer
Binary: base-installer bootstrap-base
Architecture: source all i386
Version: 1.121
Distribution: unstable
Urgency: low
Maintainer: Debian Install System Team <debian-boot@lists.debian.org>
Changed-By: Joey Hess <joeyh@debian.org>
Description: 
 base-installer - base system installation framework (udeb)
 bootstrap-base - Install the base system (udeb)
Closes: 605384 637432
Changes: 
 base-installer (1.121) unstable; urgency=low
 .
   [ Colin Watson ]
   * Merge from Ubuntu:
     - Run dpkg with --force-unsafe-io during installation; syncing is
       unnecessary in this context and can slow things down quite a bit
       (closes: #605384).
   * Consider 3.0 and later kernels as being like 2.6 for the purposes of
     debconf template selection.
 .
   [ Samuel Thibault ]
   * Propose kernels only of the same kind (closes: Bug#637432)
 .
   [ Joey Hess ]
   * Treat 3.x the same as 2.6 in arch_get_kernel.
 .
   [ Updated translations ]
   * German (de.po) by Holger Wansing
   * Basque (eu.po)
   * Italian (it.po) by Milo Casagrande
   * Macedonian (mk.po) by Arangel Angov
   * Simplified Chinese (zh_CN.po) by YunQiang Su
Checksums-Sha1: 
 94f35e0e37a929e4d6c77368a5f577b8757dbb6a 1720 base-installer_1.121.dsc
 56f7cb3c5be92caf11637bf890b9903f90f76db5 296253 base-installer_1.121.tar.gz
 dacfbfe11733dfe64ffc4b05342d45ca97fd2b73 45876 base-installer_1.121_all.udeb
 06761e76830e171fcc531277d92c9bc2eaa49837 183874 bootstrap-base_1.121_i386.udeb
Checksums-Sha256: 
 587c29afdb29431e51db073d763d50de260ed349f88099b786b4ac5ac04428c9 1720 base-installer_1.121.dsc
 58ba793ca585917fcc5ea354aefc85f39a69e3f331c3da9dc52b1e48bdd888c7 296253 base-installer_1.121.tar.gz
 70814367d4aeb923db60a48202237ca6aba0c16660631f1c82fc4942aed3e864 45876 base-installer_1.121_all.udeb
 ddaac68572faaabcd4c7e030b2a2e3d0f7a70f228bc044063c18a85cbf342e04 183874 bootstrap-base_1.121_i386.udeb
Files: 
 43a49cfb23fceaecc6238a828f987144 1720 debian-installer required base-installer_1.121.dsc
 71304a3c02add750ec7fdec0806aaab4 296253 debian-installer required base-installer_1.121.tar.gz
 21b37b216bdfc33a67c6dca351fe509b 45876 debian-installer required base-installer_1.121_all.udeb
 18c7e45d2f641886d687472efa414236 183874 debian-installer required bootstrap-base_1.121_i386.udeb
Package-Type: udeb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIVAwUBTlWFsMkQ2SIlEuPHAQgwaRAAl0wKTjazARljZNLPvDruVNDPv9Dk9NAF
oIe61Ikmv60JAgneTkXYopN7aX72wxs1ztBc8A4hNXKs1mXbACCWHpQQV2GwF4TJ
xPljj0w7QMfGOJ9C6etHrUe5gh8wVWv2MUEPUbPDNsa7Xey/ZEyPC0flayhunL5h
gRORdrIYjAY63JRwZn7Ke4jzvTlb6wkUD96lvIECGB9Rhp2vEM8XTQhwFrZHd6w9
FCI6dy6Me8Y7SeF7xSd8w7wtu8nEwbcNJgHGtSKKXuve6fwIfJ+8bEAA2oGpPCya
wXg+2y3p+kDkvzZ8elLuFcPVSVlq2lqB4kq70ekS6JgjEzEtvntX4sPHyH3pY8cw
x1Bus7Izr6HjvrzoKss9RA7G+asGvYEugkchEwvEmjwBNf5Mqto3myDtUfM/u+O9
1U8JJtAT7ylrh7j5jGD+Jh6DCQ7ZWpOTfKEU5Ul9PjPKxtJlsaKbrpSFM8lH9ENu
BbFxsoRL4AjrKho66qVKm5pkM5wgDvMYqnhWrKNPLg4E58xTqTrIrUEX5poonPFW
xx+V4IHbdvD86rxO2MUfiDOs46aEhL2t8gxXKBsfd2t+oVKbFR2tftxlYW+RGSWj
SxIxwENVygoZesZuA4Y/X6Pc0c+Difz3GyNZRHNH6A5BemBQVj0+SC2+8GhtUUAp
zY2D7jtJQa4=
=jgw2
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.