Debian Bug report logs - #635993
dpkg is very slow with btrfs filesystem

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Nicolas STRANSKY <Nico@stransky.cx>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: dpkg is very slow with btrfs filesystem

Date: Fri, 29 Jul 2011 20:51:04 -0400

Package: dpkg
Version: 1.16.0.3
Severity: normal


Hello,

I have a btrfs filesystem on /, and I experience an extreme slowness
when updating or installing packages, presumably due to dpkg's way to
write or unpack files. As a consequence, any kind of dist-upgrade takes
a very long time and the system's load goes pretty high during that
process.

Simplistic tests involving "tar" show that the disk is able to produce 
throughputs that are about 30x higher than the ones I observe when dpkg
is unpacking files (60MB/s vs. 2MB/s). I have tried a couple
workarounds but they didn't change anything: using dpkg with
--force-unsafe-io or mounting / with nodatacow. I know that there has
been some debate about dpkg / btrfs in the past, but what I experience
would tend to show that things are not solved. On the other hand, I'd be
happy to help testing a possible workaround.

Thanks,

Nicolas

-- System Information:
Debian Release: wheezy/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (101, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 3.0.0 (SMP w/2 CPU cores; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages dpkg depends on:
ii  coreutils               8.5-1            GNU core utilities
ii  libbz2-1.0              1.0.5-6          high-quality block-sorting file co
ii  libc6                   2.13-11          Embedded GNU C Library: Shared lib
ii  libselinux1             2.0.98-1.1       SELinux runtime shared libraries
ii  xz-utils                5.0.0-2          XZ-format compression utilities
ii  zlib1g                  1:1.2.3.4.dfsg-3 compression library - runtime

dpkg recommends no packages.

Versions of packages dpkg suggests:
ii  apt                           0.8.15.5   Advanced front-end for dpkg

-- Configuration Files:
/etc/dpkg/dpkg.cfg changed:
no-debsig
log /var/log/dpkg.log
force-unsafe-io


-- no debconf information

Message #10 received at 635993@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Nicolas STRANSKY <Nico@stransky.cx>, 635993@bugs.debian.org

Subject: Re: Bug#635993: dpkg is very slow with btrfs filesystem

Date: Sat, 30 Jul 2011 10:02:54 +0200

On Fri, 29 Jul 2011, Nicolas STRANSKY wrote:
> Simplistic tests involving "tar" show that the disk is able to produce 
> throughputs that are about 30x higher than the ones I observe when dpkg
> is unpacking files (60MB/s vs. 2MB/s). I have tried a couple
> workarounds but they didn't change anything: using dpkg with
> --force-unsafe-io or mounting / with nodatacow.

Did you try running your apt-get under eatmydata? If it solves your issue
it means the slowness is the result of the fsync() that dpkg is doing.
And I heard it multiple times that btrfs was not very efficient when you
are doing lots of fsync() but I am afraid there's not much we can do to
improve this if we want to keep the reliabily of dpkg in general.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Message #15 received at 635993@bugs.debian.org (full text, mbox, reply):

From: Nicolas Stransky <Nico@stransky.cx>

To: Raphael Hertzog <hertzog@debian.org>

Cc: 635993@bugs.debian.org

Subject: Re: Bug#635993: dpkg is very slow with btrfs filesystem

Date: Sat, 30 Jul 2011 08:15:38 -0400

On 07/30/2011 04:02 AM, Raphael Hertzog wrote:
> On Fri, 29 Jul 2011, Nicolas STRANSKY wrote:
>> Simplistic tests involving "tar" show that the disk is able to produce
>> throughputs that are about 30x higher than the ones I observe when dpkg
>> is unpacking files (60MB/s vs. 2MB/s). I have tried a couple
>> workarounds but they didn't change anything: using dpkg with
>> --force-unsafe-io or mounting / with nodatacow.
>
> Did you try running your apt-get under eatmydata? If it solves your issue
> it means the slowness is the result of the fsync() that dpkg is doing.
> And I heard it multiple times that btrfs was not very efficient when you
> are doing lots of fsync() but I am afraid there's not much we can do to
> improve this if we want to keep the reliabily of dpkg in general.

I didn't know before about eatmydata but it indeed completely solves my 
issue! dist-upgrades are like a hundred times faster. So I guess the 
problem is really that the multiple fsync() in dpkg won't work 
efficiently with btrfs, which is a shame! It makes it almost impossible 
to use btrfs on / with Debian.

Thanks,
Nicolas

-- 
Nico
GPG FBFA4781

Message #20 received at 635993@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Nicolas Stransky <Nico@stransky.cx>, 635993@bugs.debian.org

Subject: Re: Bug#635993: dpkg is very slow with btrfs filesystem

Date: Wed, 3 Aug 2011 14:30:48 +0200

Hi!

On Sat, 2011-07-30 at 08:15:38 -0400, Nicolas Stransky wrote:
> On 07/30/2011 04:02 AM, Raphael Hertzog wrote:
> >On Fri, 29 Jul 2011, Nicolas STRANSKY wrote:
> >>Simplistic tests involving "tar" show that the disk is able to produce
> >>throughputs that are about 30x higher than the ones I observe when dpkg
> >>is unpacking files (60MB/s vs. 2MB/s). I have tried a couple
> >>workarounds but they didn't change anything: using dpkg with
> >>--force-unsafe-io or mounting / with nodatacow.
> >
> >Did you try running your apt-get under eatmydata? If it solves your issue
> >it means the slowness is the result of the fsync() that dpkg is doing.
> >And I heard it multiple times that btrfs was not very efficient when you
> >are doing lots of fsync() but I am afraid there's not much we can do to
> >improve this if we want to keep the reliabily of dpkg in general.
> 
> I didn't know before about eatmydata but it indeed completely solves
> my issue! dist-upgrades are like a hundred times faster. So I guess
> the problem is really that the multiple fsync() in dpkg won't work
> efficiently with btrfs, which is a shame! It makes it almost
> impossible to use btrfs on / with Debian.

Well I've said this before in the previous discussions about the
fsync() issues; the dpkg database is sacred and it must never get
damaged, providing an interface in dpkg to even allow this possibility
is not acceptable, but then if the users use something like eatmydata
then they are on their own.

In the btrfs specific case, I think we should either close or wontfix
this bug report (we already have enother one stating we are not going
to remove the db fsync()s), and you should probably take it with the
btrfs developers, as it seems extremely bad the file system degrades
so much when confronted with several fsync()s. They should probably
look at implementing the sync_file_range() interfaces for it if they
haven't yet, or checking why they are not efficient.

regards,
guillem

Message #27 received at 635993-done@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Nicolas Stransky <Nico@stransky.cx>, 635993-done@bugs.debian.org

Subject: Re: Bug#635993: dpkg is very slow with btrfs filesystem

Date: Sat, 3 May 2014 22:49:06 +0200

Hi!

On Wed, 2011-08-03 at 14:30:48 +0200, Guillem Jover wrote:
> On Sat, 2011-07-30 at 08:15:38 -0400, Nicolas Stransky wrote:
> > On 07/30/2011 04:02 AM, Raphael Hertzog wrote:
> > >On Fri, 29 Jul 2011, Nicolas STRANSKY wrote:
> > >>Simplistic tests involving "tar" show that the disk is able to produce
> > >>throughputs that are about 30x higher than the ones I observe when dpkg
> > >>is unpacking files (60MB/s vs. 2MB/s). I have tried a couple
> > >>workarounds but they didn't change anything: using dpkg with
> > >>--force-unsafe-io or mounting / with nodatacow.
> > >
> > >Did you try running your apt-get under eatmydata? If it solves your issue
> > >it means the slowness is the result of the fsync() that dpkg is doing.
> > >And I heard it multiple times that btrfs was not very efficient when you
> > >are doing lots of fsync() but I am afraid there's not much we can do to
> > >improve this if we want to keep the reliabily of dpkg in general.
> > 
> > I didn't know before about eatmydata but it indeed completely solves
> > my issue! dist-upgrades are like a hundred times faster. So I guess
> > the problem is really that the multiple fsync() in dpkg won't work
> > efficiently with btrfs, which is a shame! It makes it almost
> > impossible to use btrfs on / with Debian.
> 
> Well I've said this before in the previous discussions about the
> fsync() issues; the dpkg database is sacred and it must never get
> damaged, providing an interface in dpkg to even allow this possibility
> is not acceptable, but then if the users use something like eatmydata
> then they are on their own.
> 
> In the btrfs specific case, I think we should either close or wontfix
> this bug report (we already have enother one stating we are not going
> to remove the db fsync()s), and you should probably take it with the
> btrfs developers, as it seems extremely bad the file system degrades
> so much when confronted with several fsync()s. They should probably
> look at implementing the sync_file_range() interfaces for it if they
> haven't yet, or checking why they are not efficient.

I've added an entry to the dpkg FAQ [Q], so I'm closing this now.

 [Q] <https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slow_when_using_new_filesystems_such_as_btrfs_or_ext4.3F>

Regards,
Guillem

Send a report that this bug log contains spam.