Debian Bug report logs - #650077
dpkg: The Installed-Size estimate can be wrong by a factor of 8 or a difference of 100MB

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Helmut Grohne <helmut@subdivi.de>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: dpkg: The Installed-Size estimate can be wrong by a factor of 8 or a difference of 100MB

Date: Sat, 26 Nov 2011 12:06:42 +0100

Package: dpkg
Version: 1.16.1.2
Severity: wishlist

Symptom
~~~~~~~
I just installed libjs-mathjax. According to its Installed-Size this
would just consume 16512KB. Now according to policy this is just an
estimate of course. But how accurate is it actually? So I installed said
package on ext3. Turns out /usr/share/javascript/mathjax takes up
127296KB and /usr/share/doc/mathjax takes another 1200KB. So our
estimate is wrong by a factor of 8 or a difference of 100MB. This
estimate is also used to determine whether the disk has enough space, so
if my disk just had 50MB left, aptitude would have tried to install this
package and failed.

The actual problem
~~~~~~~~~~~~~~~~~~
Problems with Installed-Size are not exactly new as discussion in
http://bugs.debian.org/534408 (unit for Installed-Size) and
http://bugs.debian.org/630533 (usage of du --apparent-size) have shown.
So what is different this time? Installing the very same package on a
btrfs yields a size that is much closer to the listed Installed-Size. (I
don't have any numbers on this.) So whatever dpkg puts into this field,
it *will* be wrong somewhere. The policy already mentions that this
estimate cannot be accurate everywhere, but in fact it will be wrong by
a factor of at least 2.5 (=sqrt(8)) or a difference of at least 50MB
(=100MB/2) somewhere. Any attempt to change the computation of this
value thus cannot fix this bug.

Discussion
~~~~~~~~~~
In the example of libjs-mathjax the reason for the huge difference is
the inclusion of a large number of very small files. Some filesystems
allocate a block for each of these files and others are able to store
multiple files in a block. A simple approach could be to include an
additional field ("Installed-Files"?) that returns the number of files
in the package. A second estimate for the Installed-Size would then be
given by the number of files times the block size. The maximum of both
estimates could be used. It would solve the immediate symptoms with
libjs-mathjax. It is not without problems though. For instance I
did not explain what block size to use. An administrator may have
different file systems set up for / and /usr. Also the question remains
whether this feature is worth the associated effort.

To get discussion going I pull in debian-policy@l.d.o.

Helmut

Message #8 received at 650077-submitter@bugs.debian.org (full text, mbox, reply):

From: Johannes Schauer <j.schauer@email.de>

To: 650077-submitter@bugs.debian.org

Cc: debian-policy@lists.debian.org

Subject: Re: dpkg: The Installed-Size estimate can be wrong by a factor of 8 or a difference of 100MB

Date: Wed, 07 Jan 2015 12:22:47 +0100

Hi,

I'm reviving this old bug as this came recently up again in the context of
ReproducibleBuilds.

On Sat, 26 Nov 2011 12:06:42 +0100 Helmut Grohne <helmut@subdivi.de> wrote:
> The actual problem
> ~~~~~~~~~~~~~~~~~~
> Problems with Installed-Size are not exactly new as discussion in
> http://bugs.debian.org/534408 (unit for Installed-Size) and
> http://bugs.debian.org/630533 (usage of du --apparent-size) have shown.
> So what is different this time? Installing the very same package on a
> btrfs yields a size that is much closer to the listed Installed-Size. (I
> don't have any numbers on this.) So whatever dpkg puts into this field,
> it *will* be wrong somewhere. The policy already mentions that this
> estimate cannot be accurate everywhere, but in fact it will be wrong by
> a factor of at least 2.5 (=sqrt(8)) or a difference of at least 50MB
> (=100MB/2) somewhere. Any attempt to change the computation of this
> value thus cannot fix this bug.
> 
> Discussion
> ~~~~~~~~~~
> In the example of libjs-mathjax the reason for the huge difference is
> the inclusion of a large number of very small files. Some filesystems
> allocate a block for each of these files and others are able to store
> multiple files in a block. A simple approach could be to include an
> additional field ("Installed-Files"?) that returns the number of files
> in the package. A second estimate for the Installed-Size would then be
> given by the number of files times the block size. The maximum of both
> estimates could be used. It would solve the immediate symptoms with
> libjs-mathjax. It is not without problems though. For instance I
> did not explain what block size to use. An administrator may have
> different file systems set up for / and /usr. Also the question remains
> whether this feature is worth the associated effort.
> 
> To get discussion going I pull in debian-policy@l.d.o.

we did some brainstorming in #debian-reproducible over the past days. I'll try
to summarize the discussion and Helmut can chip in if I missed something.

The fundamental problem is, that there are many ways that the target file
system on which the binary package gets installed can influence the size that
installing the package requires. This includes but is not limited to:

 - support for sparse files
 - inlining data inside the inode
 - compression
 - block-level or file-level duduplication

Additionally, disk usage can even grow when files are removed due to:

 - snapshots
 - overlay file system

Helmut argues, that an additional field like Installed-Files can improve the
approximation for file systems with different block sizes or whether or not
they can store multiple small files in a single block.

This solution could be extended to storing groups of files with similar size in
exponentially growing intervals of size (like: 4^(n-1) <= size < 4^n) and then
storing the number of files and cumulative number of bytes occupied by these
files in each of these sets.

But this can still not account for sparse files, compression or deduplication.

It is also worth asking what functionality the Installed-Size field is supposed
to have when looking for a solution. It's primary purpose is probably to give
apt a clue of whether or not there is enough free space to install a certain
package.

Helmut notes that other uses of the Installed-Size field are made by
debian-reference, popularity-contest, deborphan and cupt.

I would argue, that the only way to reliably solve this problem is either by:

 1) an over approximation of the actual value which will be larger than the
    actual file system usage on any common file system

 2) a way of apt or dpkg to ask the file system if there is enough space to
    store a certain file/directory structure. Most file systems (if any) do not
    offer this, though.

I think that an over approximation would be the right way to go because it is
better to wrongly warn the user that a binary package might not be installable
due to not sufficient remaining disk space, than to install a package without
sufficient remaining disk space and only fail once there actually is no more
space.

The addition of the `--apparent-size` argument to the du call in dpkg as a
response to bug #630533 made the value of the Installed-Size field too small in
some situations as can be seen in this bug report. The bug report in #630533
argues, thaht --apparent-size should be used precisely because there are file
systems that can store many small files more efficient. Because of my argument
in the last paragraph, I'd argue the opposite. The change was then applied with
guillem arguing that --apparent-size should be used because of consistency
between package rebuilds.

But the --apparent-size argument is not sufficient to provide this consistency.
Running `du -k -s --apparent-size` (the command currently used by dpkg) on an
unpacked mathjax source in an ext4 and btrfs file system, will report different
values for them. This is detrimental to the goals of the ReproducibleBuilds
efforts.

I thus propose that dpkg implements something of the following functionality
which (if I didn't miss to test something) will give an overapproximation for
Installed-Size and at the same time will be reproducible across different file
systems:

	( find mathjax-2.4 -type f -print0 \
		| du --files0-from=- -b; \
		find mathjax-2.4 \! -type f -printf "1\n" ) \
	| awk '{total = total + int($1/4096) + 4096}END{print total}'

I'm not proposing this code to be part of dpkg but I'm posting it because code
is precise but words are not. The same can probably easily be implemented in
perl.

The above snippet will get the number of bytes from all regular files and treat
all non-regular file entries as being 1 byte small. The following awk call will
then round all these values up to multiples of (an arbitrarily picked) block
size of 4096 bytes.

I think that finding the right solution to this problem requires to define what
the purpose of the Installed-Size field is. If it is to prevent package
installations on systems where there is not enough space, then I think an
overapproximation is the right way to go. More complicated measures will still
not be able to give a good approximation, given the feature-rich-ness of
today's file systems.

What do you think?

Thanks!

cheers, josch

Message #13 received at 650077-quiet@bugs.debian.org (full text, mbox, reply):

From: Helmut Grohne <helmut@subdivi.de>

To: Johannes Schauer <j.schauer@email.de>, 650077-quiet@bugs.debian.org

Cc: debian-policy@lists.debian.org

Subject: Re: Bug#650077: dpkg: The Installed-Size estimate can be wrong by a factor of 8 or a difference of 100MB

Date: Wed, 7 Jan 2015 19:53:02 +0100

On Wed, Jan 07, 2015 at 12:22:47PM +0100, Johannes Schauer wrote:
> It is also worth asking what functionality the Installed-Size field is supposed
> to have when looking for a solution. It's primary purpose is probably to give
> apt a clue of whether or not there is enough free space to install a certain
> package.

This was/is a recurring question. The policy expends 4 on the field in
section 5.6.20. It fails however to clarify the purpose and thus the
preferred way of computing or using it.

> I think that an over approximation would be the right way to go because it is
> better to wrongly warn the user that a binary package might not be installable
> due to not sufficient remaining disk space, than to install a package without
> sufficient remaining disk space and only fail once there actually is no more
> space.

Consider Alice. She wants to install foo, which has a good approximation
for her filesystem. Unfortunately, it is too big to be installed. Thus
she looks at other packages and determines that she no longer needs bar.
Duly she issues "apt-get install foo bar-". Unfortunately, this command
fails unpacking foo as bar's approximation was bad and thus it does not
free the space advertised in Installed-Size.

> 	( find mathjax-2.4 -type f -print0 \
> 		| du --files0-from=- -b; \
> 		find mathjax-2.4 \! -type f -printf "1\n" ) \
> 	| awk '{total = total + int($1/4096) + 4096}END{print total}'

Slight improvement:

    find ... \( -type f -printf "%s\n" \) -o \
             \( ! -type f -printf "1\n" \) | ...

Helmut

Message #18 received at 650077@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Johannes Schauer <j.schauer@email.de>

Cc: 650077@bugs.debian.org, debian-policy@lists.debian.org

Subject: Re: dpkg: The Installed-Size estimate can be wrong by a factor of 8 or a difference of 100MB

Date: Wed, 7 Jan 2015 20:58:55 +0100

[ Reincluding dpkg-bugs. :) ]

Hi!

On Wed, 2015-01-07 at 12:22:47 +0100, Johannes Schauer wrote:
> On Sat, 26 Nov 2011 12:06:42 +0100 Helmut Grohne <helmut@subdivi.de> wrote:
> > Discussion
> > ~~~~~~~~~~
> > In the example of libjs-mathjax the reason for the huge difference is
> > the inclusion of a large number of very small files. Some filesystems
> > allocate a block for each of these files and others are able to store
> > multiple files in a block. A simple approach could be to include an
> > additional field ("Installed-Files"?) that returns the number of files
> > in the package. A second estimate for the Installed-Size would then be
> > given by the number of files times the block size. The maximum of both
> > estimates could be used. It would solve the immediate symptoms with
> > libjs-mathjax. It is not without problems though. For instance I
> > did not explain what block size to use. An administrator may have
> > different file systems set up for / and /usr. Also the question remains
> > whether this feature is worth the associated effort.
> > 
> > To get discussion going I pull in debian-policy@l.d.o.
> 
> we did some brainstorming in #debian-reproducible over the past days. I'll try
> to summarize the discussion and Helmut can chip in if I missed something.
> 
> The fundamental problem is, that there are many ways that the target file
> system on which the binary package gets installed can influence the size that
> installing the package requires. This includes but is not limited to:
> 
>  - support for sparse files
>  - inlining data inside the inode
>  - compression
>  - block-level or file-level duduplication
   - block and inode size

For example filesystems with 1KiB blocks or smaller, or 32KiB or
bigger.

> Additionally, disk usage can even grow when files are removed due to:
> 
>  - snapshots
>  - overlay file system

> Helmut argues, that an additional field like Installed-Files can improve the
> approximation for file systems with different block sizes or whether or not
> they can store multiple small files in a single block.
> 
> This solution could be extended to storing groups of files with similar size in
> exponentially growing intervals of size (like: 4^(n-1) <= size < 4^n) and then
> storing the number of files and cumulative number of bytes occupied by these
> files in each of these sets.

This seems to me would bloat both the control file and the Packages
files.

> But this can still not account for sparse files, compression or deduplication.

As filesystems improve there will be an increasing delta between the
actual size of files and the space it takes on disk, trying to find a
portable solution to this seems like a lost cause to me, or one that
would lead to much code for not much gain.

> It is also worth asking what functionality the Installed-Size field is supposed
> to have when looking for a solution. It's primary purpose is probably to give
> apt a clue of whether or not there is enough free space to install a certain
> package.

Personally I've always taken it as a small hint of the approx. size of
the package, but the most interesting case which is always accurate
is to spot size differences between the previous and next package
version, for example from aptitude TUI.

Take into account that usually other disk usage is not accounted, like
files generated at run-time, caches, logs and similar. There's the
Extra-Size substvar just for that but I don't think any package is
actually making use of it (at least according to codesearch.d.o), so…

Also apt (and cupt) only use that information to print the size deltas,
apt only actually checks for available disk size for downloaded data,
which is the only thing that actually makes sense doing.

> Helmut notes that other uses of the Installed-Size field are made by
> debian-reference, popularity-contest, deborphan and cupt.
> 
> I would argue, that the only way to reliably solve this problem is either by:

>  1) an over approximation of the actual value which will be larger than the
>     actual file system usage on any common file system

What's a common filesystem? To me this means making the provided
information even worse.

One of the reasons I was convinced of switching to use --apparent-size
was because the information is then more correct relative to the source.
But trying to approximate the destination system is never going to be
satisfactory.

>  2) a way of apt or dpkg to ask the file system if there is enough space to
>     store a certain file/directory structure. Most file systems (if any) do not
>     offer this, though.

As mentioned above, I think this is too much work for no much gain (if
at all).

> I think that an over approximation would be the right way to go because it is
> better to wrongly warn the user that a binary package might not be installable
> due to not sufficient remaining disk space, than to install a package without
> sufficient remaining disk space and only fail once there actually is no more
> space.

Given that (at least apt and cupt) are not actually comparing the
available disk space with the accumulated packages Installed-Size,
there will be no warning anyway, and to me just making dpkg fail
gracefully on ENOSPC is the best option, anything else will just be
wrong somewhere. It used to be the case that dpkg misbehaved on
ENOSPC, but that should have been fixed long time ago (AFAIR during
the dpkg 1.14.x cycle).

In any case, I'm open to be convinced with compelling arguments to
use some other approximation. But I've not yet seen any that would
do that. Hmm does anyone know what is done in the rpm-world, probably
by yum (otherwise I might take a look)?

> But the --apparent-size argument is not sufficient to provide this consistency.
> Running `du -k -s --apparent-size` (the command currently used by dpkg) on an
> unpacked mathjax source in an ext4 and btrfs file system, will report different
> values for them. This is detrimental to the goals of the ReproducibleBuilds
> efforts.

This is then a different issue to the one reported, and one that
should be fixed. ISTR that at some point I noticed that at least
directory entries where a problem due to their varying size over
different filesystems, but it also has the problem of needing to
use an arbitrary size.

> I thus propose that dpkg implements something of the following functionality
> which (if I didn't miss to test something) will give an overapproximation for
> Installed-Size and at the same time will be reproducible across different file
> systems:
> 
> 	( find mathjax-2.4 -type f -print0 \
> 		| du --files0-from=- -b; \
> 		find mathjax-2.4 \! -type f -printf "1\n" ) \
> 	| awk '{total = total + int($1/4096) + 4096}END{print total}'

Symlink sizes are also accurate, besides directories the rest just
store information in the inode. Directories are the trickiest as they
are also a shared resource between packages, and they change size
depending on their contents.

So I'd say, for nodes with content (regular and symlinks), use their
apparent size, for containers (dirs) and metadata only nodes just
either 0 or 1KiB as the minimal unit size. Which should give a
consistent size regardless of the build filesystem.

> I think that finding the right solution to this problem requires to define what
> the purpose of the Installed-Size field is. If it is to prevent package
> installations on systems where there is not enough space, then I think an
> overapproximation is the right way to go. More complicated measures will still
> not be able to give a good approximation, given the feature-rich-ness of
> today's file systems.

I think preventing package installation through Installed-Size would
be a bad idea and quite annoying because it could disallow possible
installations which would be even more confusing.

Thanks,
Guillem

Message #23 received at 650077@bugs.debian.org (full text, mbox, reply):

From: Russ Allbery <rra@debian.org>

To: Johannes Schauer <j.schauer@email.de>

Cc: 650077@bugs.debian.org, debian-policy@lists.debian.org

Subject: Re: dpkg: The Installed-Size estimate can be wrong by a factor of 8 or a difference of 100MB

Date: Wed, 07 Jan 2015 14:49:54 -0800

Guillem Jover <guillem@debian.org> writes:
> On Wed, 2015-01-07 at 12:22:47 +0100, Johannes Schauer wrote:

>> It is also worth asking what functionality the Installed-Size field is
>> supposed to have when looking for a solution. It's primary purpose is
>> probably to give apt a clue of whether or not there is enough free
>> space to install a certain package.

> Personally I've always taken it as a small hint of the approx. size of
> the package, but the most interesting case which is always accurate is
> to spot size differences between the previous and next package version,
> for example from aptitude TUI.

I consider the primary purpose to be to give a *human* a hint as to how
large the package is.  At least, that's how I've always used it.  For that
purpose, it really only needs to be accurate within a few hundred KB, and
overstating the package size is probably better than understating it for
packages with, say, large sparse files.  It helps avoid problems like "oh,
this game looks interesting -- ack, where did all of my disk space go?"

If it can also provide information to apt about whether a package will fit
on the disk, that's great, but I don't think we should put too much weight
on that.  Operating package managers very close to a full disk is a
problematic action anyway, since there are a bunch of intermediate steps
involved in installing a package, other files that get created
dynamically, and so forth, and it's very hard to predict how much space
will be required to successfully complete the installation.  Certainly,
Installed-Size doesn't do that.

So, from a Policy perspective, I'd be happy with language that says this
is an approximation of the size of the package on disk and leave it at
that.  It's tempting to generate it by just summing the sizes of the files
in the Debian package after rounding them up to some common file system
block size and not try to get too fancy.  Of course, if people want to
work on making the number more accurate, I won't stand in their way, but I
don't think the reasonable uses for the field require that.

> Take into account that usually other disk usage is not accounted, like
> files generated at run-time, caches, logs and similar. There's the
> Extra-Size substvar just for that but I don't think any package is
> actually making use of it (at least according to codesearch.d.o), so…

Yeah.  Because knowing that to put in that field is a rather hard problem.

> Also apt (and cupt) only use that information to print the size deltas,
> apt only actually checks for available disk size for downloaded data,
> which is the only thing that actually makes sense doing.

Yup.

> Given that (at least apt and cupt) are not actually comparing the
> available disk space with the accumulated packages Installed-Size, there
> will be no warning anyway, and to me just making dpkg fail gracefully on
> ENOSPC is the best option, anything else will just be wrong
> somewhere.

+1.

> I think preventing package installation through Installed-Size would be
> a bad idea and quite annoying because it could disallow possible
> installations which would be even more confusing.

+1.

-- 
Russ Allbery (rra@debian.org)               <http://www.eyrie.org/~eagle/>

Message #28 received at 650077@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Johannes Schauer <j.schauer@email.de>, 650077@bugs.debian.org, debian-policy@lists.debian.org

Subject: Re: dpkg: The Installed-Size estimate can be wrong by a factor of 8 or a difference of 100MB

Date: Thu, 8 Jan 2015 03:07:50 +0100

Hi!

On Wed, 2015-01-07 at 20:58:55 +0100, Guillem Jover wrote:
> Hmm does anyone know what is done in the rpm-world, probably
> by yum (otherwise I might take a look)?

Ok, I checked yum and rpm, and the disk space and inode usage delta
is tracked in rpm itself. It just adds a "problem" to the transaction
in case either of the above are depleted, which are reported on
install or erasures. It does the tracking per file as it has that
access itself, and compares against the different traversed mount
points, so it's more precise than Installed-Size (which is something
I also had in mind possibly doing in dpkg once it tracks the files
metadata, to report it back to frontends).

But rpm also does the comparison against the apparent file sizes, so
it has the same problem of over or underestimating the disk usage
depending on the target filesystem setup reported in this bug.

Thanks,
Guillem

Message #33 received at 650077-quiet@bugs.debian.org (full text, mbox, reply):

From: Matthias Urlichs <matthias@urlichs.de>

To: 650077-quiet@bugs.debian.org, debian-policy@lists.debian.org

Subject: Re: Bug#650077: dpkg: The Installed-Size estimate can be wrong by a factor of 8 or a difference of 100MB

Date: Thu, 8 Jan 2015 17:25:42 +0100

Hi,

> > 	| awk '{total = total + int($1/4096) + 4096}END{print total}'
> 
I don't understand this line. Counting file blocks works like this:

> > 	| awk '{total = total + int(($1+4095)/4096)}END{print total}'

except that we don't know what the destination block size is, so I wouldn't
bother. Instead I would count the number of files separately; the receiver
can multiply #files with LOCAL_BLOCKSIZEC, add #bytes, and warn if that is
smaller than the free space on /usr; dito for #files vs. number of free
inodes.

Having dpkg deal with ENOSPC gracefully is a good idea, but IMHO mostly
orthogonal.

-- 
-- Matthias Urlichs

Message #38 received at 650077@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Johannes Schauer <j.schauer@email.de>, 650077@bugs.debian.org, debian-policy@lists.debian.org

Subject: Re: dpkg: The Installed-Size estimate can be wrong by a factor of 8 or a difference of 100MB

Date: Thu, 22 Jan 2015 16:39:12 +0100

[Message part 1 (text/plain, inline)]

Hi!

On Thu, 2015-01-08 at 03:07:50 +0100, Guillem Jover wrote:
> Ok, I checked yum and rpm, and the disk space and inode usage delta
> is tracked in rpm itself. It just adds a "problem" to the transaction
> in case either of the above are depleted, which are reported on
> install or erasures. It does the tracking per file as it has that
> access itself, and compares against the different traversed mount
> points, so it's more precise than Installed-Size (which is something
> I also had in mind possibly doing in dpkg once it tracks the files
> metadata, to report it back to frontends).

I see this might seem to contradict what I said previously, that this
might not be worth it at all. My point is that *iff* we wanted more
detailed per-file vs mount-point information, then that should be done
in dpkg itself as it's the one with such access. But that information
will keep being just an approximation, maybe a slightly better one, but
still an approximation.


In any case, I've implemented now the change I mentioned in a previous
mail, and here's the patch I'm planning on pushing to dpkg. Which I
consider "solves" both reported problems: reproducibility and documenting
the current limitations of the field.

(The directories could probably get a better size accounting by
accumulating the shipped filename sizes within, but I don't think
that's worth the trouble either, and it makes the algorithm more
complicated for third-parties to implement.)

Thanks,
Guillem

[0001-dpkg-gencontrol-Rework-Installed-Size-field-default-.patch (text/x-diff, attachment)]

Message #43 received at 650077-submitter@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: 650077-submitter@bugs.debian.org

Subject: Bug#650077 marked as pending

Date: Thu, 29 Jan 2015 00:50:45 +0000

tag 650077 pending
thanks

Hello,

Bug #650077 reported by you has been fixed in the Git repository. You can
see the changelog below, and you can check the diff of the fix at:

    http://git.debian.org/?p=dpkg/dpkg.git;a=commitdiff;h=9ed7d4d

---
commit 9ed7d4d47b73ffe67e1f7d31f899a1dfd43d490b
Author: Guillem Jover <guillem@debian.org>
Date:   Tue Jan 20 02:37:20 2015 +0100

    dpkg-gencontrol: Rework Installed-Size field default value computation
    
    Switch from «du» to File::Find, and accumulate size usage per filesystem
    object, on 1 KiB units. Use the actual size only for regular files and
    symlinks, and just 1 KiB for any other filesystem object type.
    
    This guarantees a constant and reproducible size regardless of the
    build system filesystem being used.
    
    Document how the value is computed, and that it is just a size
    approximation.
    
    Closes: #650077

diff --git a/debian/changelog b/debian/changelog
index 59aae20..08c2bba 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -47,6 +47,10 @@ dpkg (1.18.0) UNRELEASED; urgency=low
   * Wrap file references in man page PO files.
   * Switch Dpkg::Checksums from using checksum programs to the more portable
     Digest modules. Obsolete the 'program' property, and add a 'name' one.
+  * Rework the Installed-Size field default value computation to make it
+    reproducible regardless of the build system filesystem, and document
+    how the value is computed and that it is just an approximation.
+    Closes: #650077
 
  -- Guillem Jover <guillem@debian.org>  Tue, 09 Dec 2014 23:53:18 +0100
 

Message #48 received at 650077-close@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: 650077-close@bugs.debian.org

Subject: Bug#650077: fixed in dpkg 1.18.0

Date: Mon, 18 May 2015 18:22:01 +0000

Source: dpkg
Source-Version: 1.18.0

We believe that the bug you reported is fixed in the latest version of
dpkg, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 650077@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Guillem Jover <guillem@debian.org> (supplier of updated dpkg package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Mon, 18 May 2015 15:08:31 +0200
Source: dpkg
Binary: libdpkg-dev dpkg dpkg-dev libdpkg-perl dselect
Architecture: source all
Version: 1.18.0
Distribution: unstable
Urgency: low
Maintainer: Dpkg Developers <debian-dpkg@lists.debian.org>
Changed-By: Guillem Jover <guillem@debian.org>
Description:
 dpkg       - Debian package management system
 dpkg-dev   - Debian package development tools
 dselect    - Debian package management front-end
 libdpkg-dev - Debian package management static library
 libdpkg-perl - Dpkg perl modules
Closes: 588505 616614 630342 650077 690361 693951 760741 767003 768842 769515 771752 772184 773398 773718 775124 775258 775379 776072 776551 777044 779467 780866 781074 781887 782019 782326 783014 784966 785096
Changes:
 dpkg (1.18.0) unstable; urgency=low
 .
   [ Guillem Jover ]
   * Only trim trailing “/” and “/.” from «dpkg-query --search» arguments if
     they are a pathname, and not a pattern or a substring match.
   * Switch C/C++ code to use a new set of C locale character type functions
     independent of the current locale.
   * Add support for arch-bits and arch-endian dpkg-gensymbols tags.
     Closes: #630342
   * Switch perl code from legacy File::Path functions to new ones.
   * Fix perl uninitialized value usage in dpkg-scansources when the Binary
     field is missing.
   * Use dpkg-query instead of dpkg for --search in dpkg-shlibdeps so that
     the subprocesses get the correct admindir. Closes: #775258
   * Rework the Installed-Size field default value computation to make it
     reproducible regardless of the build system filesystem, and document
     how the value is computed and that it is just an approximation.
     Closes: #650077
   * Use strftime() instead of «date -R» in dpkg-genchanges, as the latter
     is not specified by POSIX and is not widely portable.
   * Warn on obsolete '<' and '>' operators in dpkg --compare-versions.
   * Trim end of line whitespace from dpkg and dselect config file parsers.
     Reported by Christoph Biedl <debian.axhn@manchmal.in-ulm.de>.
   * Do not silently eat a standalone ‘-’ in the libdpkg command-line parser.
   * Fix short-lived memory leaks in dpkg-deb and libdpkg. Closes: #769515
   * Fix «dpkg-deb -b» filename generation when the package does not contain
     an Architecture field. Regression introduced in dpkg 1.16.2.
   * Fix «dpkg --audit» to report missing and empty architecture fields.
     Regression introduced in dpkg 1.16.2.
   * Add support to dpkg-deb for reading the archive from standard input,
     except for --raw-extract which does not yet support it. Closes: #616614
     Based on a patch by Johannes Schauer <j.schauer@email.de>.
   * Add ‘.mailmap’ to the default dpkg-source ignore lists.
   * Set the SE Linux context on «dpkg-statoverride --update». Closes: #690361
   * Do not fail on dpkg-query -W and -l when multiple arguments match the
     same package. Closes: #588505
   * Change dpkg-maintscript-helper to handle symlinks and pathnames ending in
     slash. For the former error out, for the latter strip it. Closes: #771752
   * Support moving a conffile not being shipped anymore. Closes: #767003
     Thanks to Mathias Behrle <mathiasb@m9s.biz>.
   * Add a new dpkg-buildflags sanitize feature area:
     - Add new “address”, “thread”, “leak” and “undefined” features, all
       disabled by default. Closes: #760741
   * Do not accept unknown user or group names on «dpkg-statoverride --add».
     Regression introduced in dpkg 1.17.11. Closes: #775124
   * Normalize dpkg-parsechangelog command-line parsing, so that «-ovalue»,
     «-o value», «--option=value» and «--option value» will all be accepted.
     Closes: #693951
   * Add dpkg --ctrl-tarfile forwarding command for dpkg-deb.
   * Disable dependency checks on dpkg-buildpackage -S -nc.
   * Make dependency checks fatal for dpkg-buildpackage -S.
   * Update amd64 GNU cpu regex in cputable to match amd64 too, in addition
     to x86_64. This is required for FreeBSD.
   * Use badusage() instead of ohshit() for command-line errors.
   * Use the original template symbols file when diffing in dpkg-gensymbols.
     We should not create a new template symbols file, because the output
     might change (different sorting order for example) relative to the
     original. Closes: #773718
   * Do not leak kvm descriptors in start-stop-daemon on GNU/kFreeBSD systems.
     Based on a patch by Jeff Epler <jepler@unpythonic.net>. Closes: #779467
   * Switch start-stop-daemon to use a monotonic clock if available. This
     makes the timeout checks resilient to abrupt system clock changes.
     Suggested by Jose M Calhariz <jose.calhariz@hds.com>. Closes: #783014
   * Fix perl warning in dpkg-genchanges when parsing BY-HAND file entries.
     Regression introduced in dpkg 1.17.7. Closes: #781074
   * Use the checksums files list order when building the Files field to match
     the other Checksum fields in dpkg-genchanges.
   * Skip files based on the architecture from the filename in dpkg-genchanges.
   * Allow binary packages not found in debian/control in dpkg-genchanges,
     which could allow injecting debug .debs for example.
   * Annotate any non-deb binary descriptions (not just udebs) with their
     package type in dpkg-genchanges.
   * Remove outdated local copy of the Debian README.mirrors.txt file from
     dselect ftp access method, and print a message pointing to the current
     URL instead. Closes: #784966
   * Cleanup default dpkg-shlibdeps shared library directory search list:
     - Do not add cross-root directories (/<triplet>/ and /usr/<triplet>/).
     - Remove ancient multilib /emul/ia32-linux/ paths.
     - Reorder directory precedence to:
       «dpkg-shlibdeps -l» > ENV{LD_LIBRARY_PATHS} > cross-multiarch >
       DEFAULT_LIBRARY_PATH > ld.so.conf > DEFAULT_MULTILIB_PATH
   * When upgrading, copy over the cached arch-qualified package name. This
     fixes wrong output when cross-grading.
   * Consider foreign packages ambiguous in need of an arch-qualifier.
   * Perl modules:
     - Rename and deprecate Dpkg::Gettext _g function with new g_.
     - Assume in Dpkg::Arch that the abitable is always present, and bump
       libdpkg-perl Depends on dpkg to 1.16.3, the version introducing the file.
     - Remove support for GCC_TARGET environment variable from Dpkg::Shlibs.
       This was a temporary workaround for very old gcc toolchains. See #453267.
     - Prefer multiarch paths to multilib ones in Dpkg::Shlibs.
       Thanks to Helmut Grohne <helmut@subdivi.de>. Closes: #772184
     - Enable sub-second timestamps in Dpkg::Source::Patch by using Time::HiRes.
     - Use TMPDIR instead of manually setting DIR on tempfile() call in
       Dpkg::Source::Package::V2.
     - Switch Dpkg::Checksums from using checksum programs to the more portable
       Digest modules. Obsolete the 'program' property, and add a 'name' one.
     - Add support for $DEFAULT_TEXT_DOMAIN to Dpkg::Gettext, so that the Dpkg
       perl modules can always produce localized messages.
     - Fix OpenPGP Armor Header Line parsing in Dpkg::Control::Hash. We should
       only accept [\r\t ] as trailing whitespace, although RFC4880 does not
       clarify what whitespace really maps to, we should really match the GnuPG
       implementation anyway, as that's what we use to verify the signatures.
       Reported by Jann Horn <jann@thejh.net>. Fixes CVE-2015-0840.
     - Pass PATCH_GET environment varialbe instead of -g0 to the patch command
       in Dpkg::Source::Patch. This allows using non-GNU patch programs, like
       FreeBSD's patch.
     - Accept an %opts argument for the Dpkg::Control::Info constructor, and
       accept either passing a filename option as undef, or a scalar undef.
       Closes: #782019
     - Do not print on undef filehandle in Dpkg::Control::Info output().
       Thanks to Roderich Schupp <roderich.schupp@gmail.com>. Closes: #781887
     - Always sort the Dpkg::Dist::Files files list on output, instead of
       preserving the insertion order, which is not reproducible with parallel
       builds. Reported by Jérémy Bobbio <lunar@debian.org>.
     - Add new filter() method to Dpkg::Substvars.
     - Kill the process when reaching timeout in Dpkg::IPC::wait_child().
   * Test suite:
     - Check perl code compilation, warnings and strictness.
     - Fix dpkg-divert unit test to work on BSD «rm -rf» that cannot traverse
       directories with mode 000.
     - Fix dpkg-divert unit test to work when there is no /dev/full.
     - Skip test cases when there is no c++filt available.
     - Add test cases for Dpkg::Conf and Dpkg::Checksums.
     - Handle libtool executables in progname unit test.
     - Do not use a timeout when testing cat I/O, speeds up test suite by 5s.
     - Reduce timeout test from 5 seconds to 1, to speed up test suite by 4s.
   * Build system:
     - Bump gettext version to 0.19:
       + Use --add-location=file in msgmerge and xgettext commands.
       + Use --porefs=noline for po4a command.
     - Wrap file references in man page PO files with po4a --porefs=wrap.
     - Fix support for cross-building dpkg:
       + Assume a working C99 snprintf on SUS >= v3.
       + Do not try to run the va_copy configure check, just check that the
         symbol is available.
     - Check that HAVE_DECL_SYS_SIGLIST is 0 instead of undefined, to fix a
       build failure on uclibc based systems. Closes: #777044
       Based on a patch by Alex Potapenko <opotapenko@gmail.com>.
     - Use single suffix rules instead of non-portable %-pattern rules.
     - Pass CC to the test suite, so that we can use a non-gcc compiler.
     - Call AM_PROG_AR to detect the correct system archiver to use.
     - Pass -Wall to automake in AM_INIT_AUTOMAKE, not implied by foreign.
   * Packaging:
     - Remove old trigger related Breaks and Conflicts from dpkg.
     - Only use stackprotectorstrong when building dpkg with gcc >= 4.9.
     - Switch to debhelper compatibility level 9.
     - Name each public-domain license with a different name.
     - Add missing public modules to dpkg-dev package description.
     - Get rid of backward compatibility pseudo-tags from bug reports.
     - Install doc/README.feature-removal-schedule only on affected packages
       and debian/usertags everywhere.
   * Documentation:
     - Document arch-qualifiers for dependency fields in deb-control(5) and
       deb-src-control(5). Reported by Johannes Schauer <j.schauer@email.de>.
       Closes: #768842
     - Document versioned Provides in deb-control(5).
     - Document the version when dpkg-deb --raw-extract got introduced.
     - Document dpkg --log format, add missing actions and describe the startup
       messages. Closes: #773398
     - Document when and how the dpkg-maintscript-helper package name argument
       is or should be arch-qualified. Closes: #776072
     - Fix and update Arch substvar description in deb-substvars(5).
     - Document that current build flag feature areas only work on Debian and
       derivatives in dpkg-buildflags(1).
     - Use “wildcard characters” instead of “wildchars” in dpkg-query(1).
     - Document dpkg-query --search behavior in man page. Closes: #775379
     - Document postinst “triggered” argument in debian/dpkg.postinst comment
       header.
     - Document Dpkg::IPC function signatures.
     - Document the obsolete --compare-versions '<' and '>' operators in the
       dpkg(1) man page. Thanks to Tomas Pospisek <tpo_hp@sourcepole.ch>.
       Closes: #776551
     - Move dpkg-divert, dpkg-statoverride and update-alternatives man pages
       from section 8 to 1, to match their installation path.
     - Capitalize dpkg-parsechangelog(1) option descriptions.
     - Mark dpkg-parsechangelog(1) --format values in bold.
     - Place short options before long ones in dpkg-mergechangelogs(1) and
       dselect(1) man pages.
     - Properly terminate a bold marking in dpkg(1).
     - Document in man pages the dpkg version when new features were introduced.
     - Document that timestamps are reset for files patched with source format
       “3.0 (quilt)” too.
     - Document in dpkg-buildpackage(1) that using dpkg-source is sometimes
       better than -S. Suggested by Johannes Schauer <j.schauer@email.de>.
     - Document dselect -? option.
     - Document in dpkg-buildflags(1) that DEB_VENDOR influences the execution.
     - Document that dpkg performs sanity checks on PATH.
     - Mark some words and commands as non-hyphenable in man pages.
     - Separate multi-line hanging tag paragraphs with .TQ in man pages.
     - Mark field names in bold in man pages.
     - Use various groff escape sequences for quoting characters.
     - Lowercase warning and note admonitions in start-stop-daemon(8).
     - Mark dselect(1) color attributes in bold.
     - Say output instead of display for dpkg-deb tar-file option in dpkg(1).
     - Say archive instead of filename for dpkg-deb option in dpkg(1).
     - Say control-field-name instead of control-file-field in dpkg-deb(1).
     - Fix option values and pathname markup in dpkg-deb(1).
     - Use .TQ to separate different but related options in dpkg(1).
     - Clarify that dpkg-buildpackage -jN forces parallel builds, regardless of
       the packaging or upstream build systems supporting them. Closes: #780866
     - Remove unneeded update-alternatives references from options descriptions.
     - Document that apt might expect Packages.xz too in dpkg-scanpackages(1).
     - Say METHODS instead of FUNCTIONS or OBJECT FOO in POD section titles.
     - Document dpkg version when perl module versions got bumped.
   * Output message fixes and improvements:
     - Remove trailing newline from string literal in warning calls.
     - Say “execute” instead of “exec” in Dpkg::Changelog::Parse error message.
     - Say “package” instead of “it” in dpkg-name warning message.
     - Uppercase field names in error messages.
     - Expand EOF and eof into “end of file” in error messages.
     - Use “<enter>” instead of “return” or “enter” in input prompts.
     - Say directory instead of dir in output messages.
     - Merge the same dpkg-scanpackages warning messages into a single line.
     - Clarify dpkg-genchanges changes description open error.
     - Add missing preposition in Dpkg::Source::Patch error message.
     - Improve available state sorting order strings in dselect panel.
     - Say “changelog-file” instead of “changelogfile” in --help output.
     - Say “command“ instead of “action“ for dselect.
     - Improve commands listing in «dselect --help» output, by printing them
       before options and listing them one on each line with a description.
     - Improve dselect color-spec --help output.
     - Move "(default)" annotations in scripts --help output after option
       description.
     - Consistently use proper quotation marks ("" or '', and not the
       unbalanced `' pair) all over the place.
     - Use syserr() instead of an ad-hoc error message in dpkg-scansources.
     - Say substvars instead of varlist in dpkg-shlibdeps error messages.
     - Fix error messages on invalid uid/git/mode command-line syntax errors
       in dpkg-statoverride, to not say they are from the statoverride file.
     - Fix error message on empty dsc file in dpkg-genchanges.
     - Mention “(^Z)” instead of “char” for the MSDOS end of file character.
 .
   [ Raphaël Hertzog ]
   * Drop myself from Uploaders.
 .
   [ Updated programs translations ]
   * Simplified Chinese (Zhou Mo). Closes: #782326
   * Turkish (Mert Dirik). Closes: #785096
   * Vietnamese (Trần Ngọc Quân).
 .
   [ Updated scripts translations ]
   * German (Helge Kreutzmann).
 .
   [ Updated manpages translations ]
   * German (Helge Kreutzmann).
Checksums-Sha1:
 158ab1c81d80eea7127bb8e0eb0d30f4d9df2c06 2014 dpkg_1.18.0.dsc
 ec82818dc1a3990a87f6c8257adaab0445058a9f 4330304 dpkg_1.18.0.tar.xz
 ffe4d125594e847b3bb6f5951eb115a55f6fdd2d 1408828 dpkg-dev_1.18.0_all.deb
 9051fad27d2b01180d5415b29517328724ede516 1102386 libdpkg-perl_1.18.0_all.deb
Checksums-Sha256:
 c8b9009b644236c86abcd87d47c932735641983b4a54f616b87655db7a8f12a5 2014 dpkg_1.18.0.dsc
 7b156fbeed5bbfb8aa546a18badb46e471b65fa00777ccd8d7e3bbce002f225a 4330304 dpkg_1.18.0.tar.xz
 9d33b1adefce8c65721e77bd92ac8a7ba645a3d268dfb324e62b951df4b94ddd 1408828 dpkg-dev_1.18.0_all.deb
 39798e363bdfd7620fedfab787042580a3550e6e696fe703f190182b45f47f8d 1102386 libdpkg-perl_1.18.0_all.deb
Files:
 22762915c7530b472077d5708228b962 2014 admin required dpkg_1.18.0.dsc
 22d1a0cc18910fc1275a4fbc22aa4982 4330304 admin required dpkg_1.18.0.tar.xz
 2d6f6b6381a343c0262acc4233accd0f 1408828 utils optional dpkg-dev_1.18.0_all.deb
 ef39f89dae3f3006e4711d84c3b64b95 1102386 perl optional libdpkg-perl_1.18.0_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVWfCZAAoJELlyvz6krlej7u8P+wflvoopOuHfc5xBsoz2EWSj
sLe3v0qtQv3ffZfZ+M8dVUBC32Bf/11NNbyarf35g702nAay1DL5T2YDypXLCid7
3KoMRWbaiSsvABuF+SOHmGVccagL9f5bd3xgMNA1JgvqEJ6ujFuG7plRP9ieCpOK
nhmKKhp1uhQtF7NDu4UFOtVrJJpZqmJ5JwVr5thYukB46fRNeWSEgY6n20+VmjFL
ThCHIbhxtXxyQaR+PURNfNXlmBO/zK1sA/9aUTcX9pafXg83rrBuMVhr+SOwScQ/
YWFjoao0nmi+3Z+lL4fg3hxNz4QirHk0t2H97ivLd1A39J7YOooTIz+SgmPuuQZH
UV+DDS8xBE+fiH3Vn529yXa84puRndYA7iD1UW0nJzVEkyJbb6HcA+Qi1hjDRaD/
QOxA5gFcMXFi4tMu8KESMztL0SZOHvyWEl17p4Ofu+oHCvmh4jsw+MEjOWcSFzaC
YW8t7DC9pP1cMOhMYFJBxjO0YjIu3nOQ6n2/HzlAYnF5FPCoQC8JKcWgIyurzJI1
+YcL5aSeB9xyVJlKPl7gJk+LgKU4S4QKT1rs4z4uVML7WRIbXfzmoPZCOntqBvF/
Yr8Pf77ZFdIwzmiyQBHKmUaxGP/+z3gupl9UfF+/5u+ty08+rfRgoASEr5m0ej8Q
CU8WUmb6vEHwytaqwMmy
=J4L8
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.