Debian Bug report logs - #1004557
update-alternatives: please make the mtimes reproducible if SOURCE_DATE_EPOCH is set

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Johannes Schauer Marin Rodrigues <josch@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: man-db: please make index.db installations reproducible

Date: Sun, 30 Jan 2022 14:27:05 +0100

[Message part 1 (text/plain, inline)]

Source: man-db
Version: 2.9.4-4
Severity: normal
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: timestamps
X-Debbugs-Cc: josch@debian.org, reproducible-bugs@lists.alioth.debian.org

Hi,

currently, the index.db files created by man-db -c are unreproducible
when creating a Debian chroot. This means that tools that attempt to
create reproducible system images delete all index.db files:

https://gitlab.tails.boum.org/tails/tails/-/blob/stable/config/chroot_local-hooks/99-zzzzzz_reproducible-builds-post-processing#L28
https://salsa.debian.org/live-team/live-build/-/blob/master/share/hooks/normal/0190-remove-temporary-files.hook.chroot#L6

This could be avoided if the index.db files after installation would be
bit-by-bit reproducible. The attached patch fixes the problem by
truncating the timestamp set in index.db to the value of
SOURCE_DATE_EPOCH if the variable is set.

This means that this patch does not change anything during normal
operation but only comes into play if a utility that sets
SOURCE_DATE_EPOCH is installing packages.

Thanks!

cheers, josch

[man-db.diff (text/plain, attachment)]

Message #10 received at 1004557@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: Johannes Schauer Marin Rodrigues <josch@debian.org>, 1004557@bugs.debian.org

Subject: Re: Bug#1004557: man-db: please make index.db installations reproducible

Date: Sun, 30 Jan 2022 14:03:30 +0000

On Sun, Jan 30, 2022 at 02:27:05PM +0100, Johannes Schauer Marin Rodrigues wrote:
> currently, the index.db files created by man-db -c are unreproducible
> when creating a Debian chroot. This means that tools that attempt to
> create reproducible system images delete all index.db files:
> 
> https://gitlab.tails.boum.org/tails/tails/-/blob/stable/config/chroot_local-hooks/99-zzzzzz_reproducible-builds-post-processing#L28
> https://salsa.debian.org/live-team/live-build/-/blob/master/share/hooks/normal/0190-remove-temporary-files.hook.chroot#L6
> 
> This could be avoided if the index.db files after installation would be
> bit-by-bit reproducible. The attached patch fixes the problem by
> truncating the timestamp set in index.db to the value of
> SOURCE_DATE_EPOCH if the variable is set.
> 
> This means that this patch does not change anything during normal
> operation but only comes into play if a utility that sets
> SOURCE_DATE_EPOCH is installing packages.

I'm a bit confused, because this seems to work at the wrong layer.
Debian packages are supposed to preserve timestamps from the source
package wherever possible, and failing that it would be possible to
ensure that the timestamp of generated manual pages in binary packages
is set to SOURCE_DATE_EPOCH.  Flattening timestamps to an epoch at mandb
time seems like the wrong place for this at first inspection, and I'd
like some clearer rationale for why you ended up with this approach.

I would suggest instead ensuring that mtimes of manual pages are
reproducible, after which mandb should produce reproducible databases
(and if it doesn't I'd consider that a bug).

Deliberately setting database timestamps that don't match the filesystem
will confuse mandb into doing unnecessary work in later runs, so I don't
like this approach.

-- 
Colin Watson (he/him)                              [cjwatson@debian.org]

Message #13 received at 1004557@bugs.debian.org (full text, mbox, reply):

From: Johannes Schauer Marin Rodrigues <josch@debian.org>

To: 1004557@bugs.debian.org, Colin Watson <cjwatson@debian.org>

Subject: Re: Bug#1004557: man-db: please make index.db installations reproducible

Date: Sun, 30 Jan 2022 17:35:37 +0100

[Message part 1 (text/plain, inline)]

Hi Colin,

thank you for your quick reply! :)

Quoting Colin Watson (2022-01-30 15:03:30)
> On Sun, Jan 30, 2022 at 02:27:05PM +0100, Johannes Schauer Marin Rodrigues wrote:
> > currently, the index.db files created by man-db -c are unreproducible
> > when creating a Debian chroot. This means that tools that attempt to
> > create reproducible system images delete all index.db files:
> > 
> > https://gitlab.tails.boum.org/tails/tails/-/blob/stable/config/chroot_local-hooks/99-zzzzzz_reproducible-builds-post-processing#L28
> > https://salsa.debian.org/live-team/live-build/-/blob/master/share/hooks/normal/0190-remove-temporary-files.hook.chroot#L6
> > 
> > This could be avoided if the index.db files after installation would be
> > bit-by-bit reproducible. The attached patch fixes the problem by
> > truncating the timestamp set in index.db to the value of
> > SOURCE_DATE_EPOCH if the variable is set.
> > 
> > This means that this patch does not change anything during normal
> > operation but only comes into play if a utility that sets
> > SOURCE_DATE_EPOCH is installing packages.
> 
> I'm a bit confused, because this seems to work at the wrong layer.
> Debian packages are supposed to preserve timestamps from the source
> package wherever possible, and failing that it would be possible to
> ensure that the timestamp of generated manual pages in binary packages
> is set to SOURCE_DATE_EPOCH.  Flattening timestamps to an epoch at mandb
> time seems like the wrong place for this at first inspection, and I'd
> like some clearer rationale for why you ended up with this approach.
> 
> I would suggest instead ensuring that mtimes of manual pages are
> reproducible, after which mandb should produce reproducible databases
> (and if it doesn't I'd consider that a bug).
> 
> Deliberately setting database timestamps that don't match the filesystem
> will confuse mandb into doing unnecessary work in later runs, so I don't like
> this approach.

My reasoning was, that tools that care about reproducible index.db will
"flatten" the mtimes to SOURCE_DATE_EPOCH in the tarball or image they produce,
so setting the timestamp in index.db to SOURCE_DATE_EPOCH for those timestamps
larger than SOURCE_DATE_EPOCH seemed like the approach that would result in a
consistent overall state.

But if that's the wrong approach, lets think of the alternative: making sure
that the mtimes of manual pages is reproducible. If I use gdbm_dump on the
index.db of two different chroots, then it looks like the following manual pages
have differing timestamps:

bash-builtins, which, dash, mawk, pager, awk, sh, more, nawk, builtins

Most of those seem to be symlinks into /etc/alternatives and those symlinks get
created by maintainer scripts using update-alternatives. Are you suggesting
that update-alternatives should gain support for setting the mtime of the files
it creates to SOURCE_DATE_EPOCH?

I'm puzzled by bash-builtins though because that one is not a symlink. So I
don't understand why the timestamp differs there.

Thanks!

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #18 received at 1004557@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: Johannes Schauer Marin Rodrigues <josch@debian.org>, 1004557@bugs.debian.org

Subject: Re: Bug#1004557: man-db: please make index.db installations reproducible

Date: Mon, 31 Jan 2022 02:28:07 +0000

On Sun, Jan 30, 2022 at 05:35:37PM +0100, Johannes Schauer Marin Rodrigues wrote:
> Quoting Colin Watson (2022-01-30 15:03:30)
> > I'm a bit confused, because this seems to work at the wrong layer.
> > Debian packages are supposed to preserve timestamps from the source
> > package wherever possible, and failing that it would be possible to
> > ensure that the timestamp of generated manual pages in binary packages
> > is set to SOURCE_DATE_EPOCH.  Flattening timestamps to an epoch at mandb
> > time seems like the wrong place for this at first inspection, and I'd
> > like some clearer rationale for why you ended up with this approach.
> > 
> > I would suggest instead ensuring that mtimes of manual pages are
> > reproducible, after which mandb should produce reproducible databases
> > (and if it doesn't I'd consider that a bug).
> > 
> > Deliberately setting database timestamps that don't match the filesystem
> > will confuse mandb into doing unnecessary work in later runs, so I don't like
> > this approach.
> 
> My reasoning was, that tools that care about reproducible index.db will
> "flatten" the mtimes to SOURCE_DATE_EPOCH in the tarball or image they produce,
> so setting the timestamp in index.db to SOURCE_DATE_EPOCH for those timestamps
> larger than SOURCE_DATE_EPOCH seemed like the approach that would result in a
> consistent overall state.

It might not be entirely impossible, but I'd really prefer not to break
the link between filesystem timestamps and database timestamps if we can
avoid it.  I know your implementation is more or less like "tar
--mtime=DATE --clamp-time", but the difference is that tar doesn't also
compare filesystem timestamps against the archive later.

> But if that's the wrong approach, lets think of the alternative: making sure
> that the mtimes of manual pages is reproducible. If I use gdbm_dump on the
> index.db of two different chroots, then it looks like the following manual pages
> have differing timestamps:
> 
> bash-builtins, which, dash, mawk, pager, awk, sh, more, nawk, builtins
> 
> Most of those seem to be symlinks into /etc/alternatives and those symlinks get
> created by maintainer scripts using update-alternatives. Are you suggesting
> that update-alternatives should gain support for setting the mtime of the files
> it creates to SOURCE_DATE_EPOCH?

I think that would at least be worth considering.  It doesn't seem any
less obvious a thing to do for reproducible installs than hacking mandb
would be, and it would deal with the problem closer to its source: for
instance, it would get you closer to being able to produce
bitwise-identical reproducible images by e.g. tarring up the filesystem,
which would preserve filesystem mtimes in the image.  (Though I guess
--clamp-mtime deals with that, but maybe not all image archiving tools
have something like that?)

Another approach might be to modify filesystem timestamps after
postinsts have finished running but before mandb runs to clamp
timestamps to SOURCE_DATE_EPOCH; a bit like your proposed patch, but
actually modifying the filesystem timestamps as well.  I'm not sure
where that could go, though.  It can't be in mandb because the postinst
deliberately doesn't run mandb as root; and of course mandb is itself
run from a postinst.  Maybe some kind of dpkg hook, or maybe it would be
simplest to just run a post-processing step that clamps all the
filesystem timestamps and then runs the equivalent of "sudo -u man mandb
-cq"?  (This might be more palatable with man-db 2.10.0, where this will
take more like 10 seconds rather than several minutes; see #1003089.)

> I'm puzzled by bash-builtins though because that one is not a symlink. So I
> don't understand why the timestamp differs there.

This puzzled me for a while too, but it's because
/usr/share/man/man7/builtins.7.gz is a symlink created by
update-alternatives and references bash-builtins in its NAME, which
provoked https://bugs.debian.org/691643.  I've now fixed that upstream:

  https://gitlab.com/cjwatson/man-db/-/commit/37ab864354c1d0ac09e27d2346a1221bf4628509

This may cause your comparisons to show more differences, but it should
mean that they're more reliably the *same* differences.  Previously, the
behaviour depended on directory iteration order (actually usually the
location of the first physical extent of each file on disk, since mandb
sorts by that for improved performance on rotational disk drives).

-- 
Colin Watson (he/him)                              [cjwatson@debian.org]

Message #21 received at 1004557@bugs.debian.org (full text, mbox, reply):

From: Johannes Schauer Marin Rodrigues <josch@debian.org>

To: 1004557@bugs.debian.org, Colin Watson <cjwatson@debian.org>, debian-dpkg@lists.debian.org

Subject: Re: Bug#1004557: man-db: please make index.db installations reproducible

Date: Mon, 31 Jan 2022 22:23:53 +0100

[Message part 1 (text/plain, inline)]

Quoting Colin Watson (2022-01-31 03:28:07)
> > But if that's the wrong approach, lets think of the alternative: making
> > sure that the mtimes of manual pages is reproducible. If I use gdbm_dump on
> > the index.db of two different chroots, then it looks like the following
> > manual pages have differing timestamps:
> > 
> > bash-builtins, which, dash, mawk, pager, awk, sh, more, nawk, builtins
> > 
> > Most of those seem to be symlinks into /etc/alternatives and those symlinks get
> > created by maintainer scripts using update-alternatives. Are you suggesting
> > that update-alternatives should gain support for setting the mtime of the files
> > it creates to SOURCE_DATE_EPOCH?
> 
> I think that would at least be worth considering.  It doesn't seem any
> less obvious a thing to do for reproducible installs than hacking mandb
> would be, and it would deal with the problem closer to its source: for
> instance, it would get you closer to being able to produce
> bitwise-identical reproducible images by e.g. tarring up the filesystem,
> which would preserve filesystem mtimes in the image.  (Though I guess
> --clamp-mtime deals with that, but maybe not all image archiving tools
> have something like that?)

I don't think we will ever not need --clamp-mtime when producing reproducible
chroot tarballs. It would mean that every maintainer script would have to be
extended so that every time after a file is created or changed its mtime is
adjusted. I don't think that can fly.

> Another approach might be to modify filesystem timestamps after postinsts
> have finished running but before mandb runs to clamp timestamps to
> SOURCE_DATE_EPOCH; a bit like your proposed patch, but actually modifying the
> filesystem timestamps as well.  I'm not sure where that could go, though.  It
> can't be in mandb because the postinst deliberately doesn't run mandb as
> root; and of course mandb is itself run from a postinst.  Maybe some kind of
> dpkg hook, or maybe it would be simplest to just run a post-processing step
> that clamps all the filesystem timestamps and then runs the equivalent of
> "sudo -u man mandb -cq"?  (This might be more palatable with man-db 2.10.0,
> where this will take more like 10 seconds rather than several minutes; see
> #1003089.)

I don't like the idea of moving functionality like that into chroot-creating
scripts. If we want the chroot to have a certain property, we should add that
to the packages involved using declarative methods.

So another way to fix this would be to add a "touch" call to every maintainer
script calling update-alternatives involving man pages and let them set the
symlink mtime to SOURCE_DATE_EPOCH if that variable is set. But I think that's
a bad idea and we should rather do this centrally.

> > I'm puzzled by bash-builtins though because that one is not a symlink. So I
> > don't understand why the timestamp differs there.
> 
> This puzzled me for a while too, but it's because
> /usr/share/man/man7/builtins.7.gz is a symlink created by
> update-alternatives and references bash-builtins in its NAME, which
> provoked https://bugs.debian.org/691643.  I've now fixed that upstream:
> 
>   https://gitlab.com/cjwatson/man-db/-/commit/37ab864354c1d0ac09e27d2346a1221bf4628509
> 
> This may cause your comparisons to show more differences, but it should
> mean that they're more reliably the *same* differences.  Previously, the
> behaviour depended on directory iteration order (actually usually the
> location of the first physical extent of each file on disk, since mandb sorts
> by that for improved performance on rotational disk drives).

Thanks for the fix!

I talked with Guillem about the possibility of changing update-alternatives to
produce reproducible mtimes. I'm adding debian-dpkg@lists.debian.org to discuss
having a reproducible index.db by changing unattended-upgrades.

Reading the commit you quote above it seems that using the symlink's mtime is
on purpose? I think the problem would not exist if the mtime of the link target
would be used. But there is probably a reason why this is not done already?

Guillem also brought up that using SOURCE_DATE_EPOCH is wrong in this context
because this is about runtime behaviour. I disagree with that assessment. The
idea would be to check whether SOURCE_DATE_EPOCH is set in unattended-upgrades
and only if it is, then change its behaviour. That means that the current
behaviour of unattended-upgrades would be unchanged without SOURCE_DATE_EPOCH
set. Only when building something that needs to be reproducible like a chroot
tarball or system image, SOURCE_DATE_EPOCH would be set. Since building a
chroot tarball or system images is essentially compiling a final artifact from
some other input I think this is completely in line with the idea that
SOURCE_DATE_EPOCH is there to allow creating reproducible build output. In that
sense, unattended-upgrades would be in line with many other tools that respect
SOURCE_DATE_EPOCH and thus differentiate between the scenario where they are
used in the context of some build process (here: creating a chroot tarball) or
normal operation. I don't think that it makes a difference that the input to
the build process here are binary packages and not sources. During normal
package building, build dependencies also do not always provide some human
readable source that is then recompiled but also just binary material that is
then integrated into the final build output.

Guillem was thinking about introducing a new variable in addition to
SOURCE_DATE_EPOCH to indicate that some software should produce reproducible
output in scenarios like this. This would mean that software that already
supports SOURCE_DATE_EPOCH and is called by maintainer scripts now has to be
patched to do the special casing for SOURCE_DATE_EPOCH as well as for the new
variable. I also don't think a new variable is a good idea because I think that
building a reproducible chroot tarball can be well described as some sort of
build process for which SOURCE_DATE_EPOCH makes perfect sense.

We also thought about letting unattended-upgrades use the mtime of the symlink
target as the mtime of the symlink. But this would be a bad idea because backup
software will likely not notice a change of the symlink in case the symlink
switches to a target with a lower mtime.

What do you think?

Thanks!

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #24 received at 1004557@bugs.debian.org (full text, mbox, reply):

From: Johannes Schauer Marin Rodrigues <josch@debian.org>

To: 1004557@bugs.debian.org, Colin Watson <cjwatson@debian.org>, debian-dpkg@lists.debian.org

Subject: Re: Bug#1004557: man-db: please make index.db installations reproducible

Date: Mon, 31 Jan 2022 22:38:09 +0100

[Message part 1 (text/plain, inline)]

Quoting Johannes Schauer Marin Rodrigues (2022-01-31 22:23:53)
> I talked with Guillem about the possibility of changing update-alternatives
> to produce reproducible mtimes. I'm adding debian-dpkg@lists.debian.org to
> discuss having a reproducible index.db by changing unattended-upgrades.
> 
> Reading the commit you quote above it seems that using the symlink's mtime is
> on purpose? I think the problem would not exist if the mtime of the link target
> would be used. But there is probably a reason why this is not done already?
> 
> Guillem also brought up that using SOURCE_DATE_EPOCH is wrong in this context
> because this is about runtime behaviour. I disagree with that assessment. The
> idea would be to check whether SOURCE_DATE_EPOCH is set in unattended-upgrades
> and only if it is, then change its behaviour. That means that the current
> behaviour of unattended-upgrades would be unchanged without SOURCE_DATE_EPOCH
> set. Only when building something that needs to be reproducible like a chroot
> tarball or system image, SOURCE_DATE_EPOCH would be set. Since building a
> chroot tarball or system images is essentially compiling a final artifact from
> some other input I think this is completely in line with the idea that
> SOURCE_DATE_EPOCH is there to allow creating reproducible build output. In that
> sense, unattended-upgrades would be in line with many other tools that respect
> SOURCE_DATE_EPOCH and thus differentiate between the scenario where they are
> used in the context of some build process (here: creating a chroot tarball) or
> normal operation. I don't think that it makes a difference that the input to
> the build process here are binary packages and not sources. During normal
> package building, build dependencies also do not always provide some human
> readable source that is then recompiled but also just binary material that is
> then integrated into the final build output.
> 
> Guillem was thinking about introducing a new variable in addition to
> SOURCE_DATE_EPOCH to indicate that some software should produce reproducible
> output in scenarios like this. This would mean that software that already
> supports SOURCE_DATE_EPOCH and is called by maintainer scripts now has to be
> patched to do the special casing for SOURCE_DATE_EPOCH as well as for the new
> variable. I also don't think a new variable is a good idea because I think that
> building a reproducible chroot tarball can be well described as some sort of
> build process for which SOURCE_DATE_EPOCH makes perfect sense.
> 
> We also thought about letting unattended-upgrades use the mtime of the symlink
> target as the mtime of the symlink. But this would be a bad idea because backup
> software will likely not notice a change of the symlink in case the symlink
> switches to a target with a lower mtime.

And of course all mentions of unattended-upgrades above should've been
update-alternatives instead. Sorry for that.

[signature.asc (application/pgp-signature, inline)]

Message #29 received at 1004557@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: Johannes Schauer Marin Rodrigues <josch@debian.org>

Cc: 1004557@bugs.debian.org, debian-dpkg@lists.debian.org

Subject: Re: Bug#1004557: man-db: please make index.db installations reproducible

Date: Tue, 1 Feb 2022 10:37:34 +0000

On Mon, Jan 31, 2022 at 10:23:53PM +0100, Johannes Schauer Marin Rodrigues wrote:
> Quoting Colin Watson (2022-01-31 03:28:07)
> > Another approach might be to modify filesystem timestamps after postinsts
> > have finished running but before mandb runs to clamp timestamps to
> > SOURCE_DATE_EPOCH; a bit like your proposed patch, but actually modifying the
> > filesystem timestamps as well.  I'm not sure where that could go, though.  It
> > can't be in mandb because the postinst deliberately doesn't run mandb as
> > root; and of course mandb is itself run from a postinst.  Maybe some kind of
> > dpkg hook, or maybe it would be simplest to just run a post-processing step
> > that clamps all the filesystem timestamps and then runs the equivalent of
> > "sudo -u man mandb -cq"?  (This might be more palatable with man-db 2.10.0,
> > where this will take more like 10 seconds rather than several minutes; see
> > #1003089.)
> 
> I don't like the idea of moving functionality like that into chroot-creating
> scripts. If we want the chroot to have a certain property, we should add that
> to the packages involved using declarative methods.
> 
> So another way to fix this would be to add a "touch" call to every maintainer
> script calling update-alternatives involving man pages and let them set the
> symlink mtime to SOURCE_DATE_EPOCH if that variable is set. But I think that's
> a bad idea and we should rather do this centrally.

Fair enough.

> > This puzzled me for a while too, but it's because
> > /usr/share/man/man7/builtins.7.gz is a symlink created by
> > update-alternatives and references bash-builtins in its NAME, which
> > provoked https://bugs.debian.org/691643.  I've now fixed that upstream:
> > 
> >   https://gitlab.com/cjwatson/man-db/-/commit/37ab864354c1d0ac09e27d2346a1221bf4628509
> > 
> > This may cause your comparisons to show more differences, but it should
> > mean that they're more reliably the *same* differences.  Previously, the
> > behaviour depended on directory iteration order (actually usually the
> > location of the first physical extent of each file on disk, since mandb sorts
> > by that for improved performance on rotational disk drives).
> 
> Thanks for the fix!
> 
> I talked with Guillem about the possibility of changing update-alternatives to
> produce reproducible mtimes. I'm adding debian-dpkg@lists.debian.org to discuss
> having a reproducible index.db by changing unattended-upgrades.
> 
> Reading the commit you quote above it seems that using the symlink's mtime is
> on purpose? I think the problem would not exist if the mtime of the link target
> would be used. But there is probably a reason why this is not done already?

I'm not saying there are no other ways things could work, but just
storing the mtime of the link target would definitely cause problems.
If mandb did that, then it would fail to detect when the symlink changed
to point to something else (at least in its current model where it
compares the mtime against the inode to see whether it needs to rescan a
page, and doesn't store the symlink data itself in the DB).

> Guillem also brought up that using SOURCE_DATE_EPOCH is wrong in this context
> because this is about runtime behaviour.
[...]
> Guillem was thinking about introducing a new variable in addition to
> SOURCE_DATE_EPOCH to indicate that some software should produce reproducible
> output in scenarios like this.

I don't think I have much of an opinion about the details of variable
naming there.

> We also thought about letting unattended-upgrades use the mtime of the symlink
> target as the mtime of the symlink. But this would be a bad idea because backup
> software will likely not notice a change of the symlink in case the symlink
> switches to a target with a lower mtime.

Yes, definitely a bad idea for that sort of reason.  Depending on the
exact details, it might also cause some backup software to think that
there's filesystem corruption (compare
https://bugs.launchpad.net/ubuntu/+source/man-db/+bug/1411633 /
https://bugs.debian.org/1004355).

-- 
Colin Watson (he/him)                              [cjwatson@debian.org]

Message #32 received at 1004557@bugs.debian.org (full text, mbox, reply):

From: Johannes Schauer Marin Rodrigues <josch@debian.org>

To: Colin Watson <cjwatson@debian.org>

Cc: 1004557@bugs.debian.org, debian-dpkg@lists.debian.org

Subject: Re: Bug#1004557: man-db: please make index.db installations reproducible

Date: Tue, 01 Feb 2022 16:56:27 +0100

[Message part 1 (text/plain, inline)]

Control: reassign -1 dpkg 1.21.1

Quoting Colin Watson (2022-02-01 11:37:34)
> > I talked with Guillem about the possibility of changing update-alternatives to
> > produce reproducible mtimes. I'm adding debian-dpkg@lists.debian.org to discuss
> > having a reproducible index.db by changing unattended-upgrades.
> > 
> > Reading the commit you quote above it seems that using the symlink's mtime is
> > on purpose? I think the problem would not exist if the mtime of the link target
> > would be used. But there is probably a reason why this is not done already?
> 
> I'm not saying there are no other ways things could work, but just
> storing the mtime of the link target would definitely cause problems.
> If mandb did that, then it would fail to detect when the symlink changed
> to point to something else (at least in its current model where it
> compares the mtime against the inode to see whether it needs to rescan a
> page, and doesn't store the symlink data itself in the DB).
> 
> > Guillem also brought up that using SOURCE_DATE_EPOCH is wrong in this context
> > because this is about runtime behaviour.
> [...]
> > Guillem was thinking about introducing a new variable in addition to
> > SOURCE_DATE_EPOCH to indicate that some software should produce reproducible
> > output in scenarios like this.
> 
> I don't think I have much of an opinion about the details of variable
> naming there.
> 
> > We also thought about letting unattended-upgrades use the mtime of the symlink
> > target as the mtime of the symlink. But this would be a bad idea because backup
> > software will likely not notice a change of the symlink in case the symlink
> > switches to a target with a lower mtime.
> 
> Yes, definitely a bad idea for that sort of reason.  Depending on the
> exact details, it might also cause some backup software to think that
> there's filesystem corruption (compare
> https://bugs.launchpad.net/ubuntu/+source/man-db/+bug/1411633 /
> https://bugs.debian.org/1004355).

Thanks for your input! It then seems that we should indeed adjust the mtime of
the symlinks instead.

I'm thus reassigning this bug to dpkg and attached a tested patch which fixes
the problem.

Thanks!

cheers, josch

[0001-utils-update-alternatives.c-respect-SOURCE_DATE_EPOC.patch (text/x-diff, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #43 received at 1004557-submitter@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: 1004557-submitter@bugs.debian.org

Subject: Bug#1004557 in package dpkg marked as pending

Date: Sun, 13 Mar 2022 17:50:01 +0100

Control: tag 1004557 pending

Hi!

Bug #1004557 in package dpkg reported by you has been fixed in
the dpkg/dpkg.git Git repository. You can see the changelog below, and
you can check the diff of the fix at:

    https://git.dpkg.org/cgit/dpkg/dpkg.git/diff/?id=272a5232b

---
u-a: Use the target timestamp when the symlink does not exist

Some programs track the symlink timestamps in their databases (such as
man-db). To be able to create reproducible installations we have to use
timestamps that are reproducible, at least on the first installation.

We also need to take into account that not using the current timestamp
when updating symlinks might cause backup systems to not work properly.

To cover these two cases, we will copy the timestamp from the target, if
the symlink did not exist.

Closes: #1004557

Message #50 received at 1004557-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 1004557-close@bugs.debian.org

Subject: Bug#1004557: fixed in dpkg 1.21.2

Date: Sun, 13 Mar 2022 20:49:48 +0000

Source: dpkg
Source-Version: 1.21.2
Done: Guillem Jover <guillem@debian.org>

We believe that the bug you reported is fixed in the latest version of
dpkg, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1004557@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Guillem Jover <guillem@debian.org> (supplier of updated dpkg package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Sun, 13 Mar 2022 20:17:35 +0100
Source: dpkg
Architecture: source
Version: 1.21.2
Distribution: unstable
Urgency: medium
Maintainer: Dpkg Developers <debian-dpkg@lists.debian.org>
Changed-By: Guillem Jover <guillem@debian.org>
Closes: 382307 533639 1001010 1001695 1001761 1003164 1003671 1003672 1003673 1003799 1004372 1004557 1007116
Changes:
 dpkg (1.21.2) unstable; urgency=medium
 .
   [ Guillem Jover ]
   * dpkg: Fix memory leak in remove-on-upgrade handling.
   * dpkg-deb: Fix unexpected end of file conditions on .deb extract.
   * Use anchor links for the dpkg FAQ URLs.
   * update-alternatives: Do not skip --config with a single entry.
     Reported by David Kalnischkies <donkult@debian.org>.
   * update-alternatives: Refactor alternative_install().
   * update-alternatives: Use intermediate variables when parsing actions.
   * update-alternatives: Clarify option parse errors by printing the wrong
     values.
   * update-alternatives: Clarify bad usage message by enclosing in angles and
     quoting arguments.
   * dpkg-buildpackage: Switch terse make from -s to --no-print-directory.
     Closes: #1003164
   * scripts: Unify deprecated command-line option warnings.
   * dpkg-source: Fix alternate changelog file usage with 2.0 and 3.0 formats.
     Reported by Umut <ue16@gmx.de> (on IRC).
   * update-alternatives: Use the target timestamp when the symlink does not
     exist. Closes: #1004557
   * dselect: Handle window resizes in help and menu screens. Closes: #382307
   * dselect: Do not beep on key press errors in method and package list
     windows. Closes: #533639
   * dselect: Add support for --instdir.
   * Perl modules:
     - Dpkg::Control::FieldsCore: Sort control type entries in %FIELD_ORDER.
     - Dpkg::Control::FieldsCore: Refactor Vcs fields into a common variable.
     - Dpkg::Control::FieldsCore: Refactor testsuite fields into a common
       variable.
     - Dpkg::Control::FieldsCore: Accept Architecture as a debian/tests/control
       field.
     - Dpkg::Control::FieldsCore: Add missing allowed fields for (In)Release
       files.
     - Dpkg::Control::FieldsCore: Add field order for all control types.
     - Dpkg::Index: Switch key function for control tests to be a stanza index.
       Reported by Paul Gevers <elbrus@debian.org>.
     - Dpkg::Source::Package: Only generate the patch header if needed.
       Prompted by Umut <ue16@gmx.de> (on IRC).
     - Dpkg::Source::Package: Use File::Spec instead of ad-hoc concatenation.
   * Documentation:
     - man: Mention on what actions triggers get processed in dpkg(1).
       Closes: #1001010
     - man: Clarify that dpkg-divert --list pattern is optional.
       Thanks to наб <nabijaczleweli@nabijaczleweli.xyz>. Closes: #1001761
     - man: Use «main» git branch in examples.
     - man: Markup each individual element independently.
     - man: Refer to the relevant maintscript actions explicitly.
     - man: Add a missing preposition to deb-preinst(5).
     - man: Do not hardcode DPKG_ADMINDIR in update-alternatives.
     - man: Document that update-alternatives honors DPKG_ROOT.
     - man: Clarify --admindir and --instdir default values.
       Reported by Johannes Schauer Marin Rodrigues <josch@debian.org>.
   * Code internals:
     - libdpkg: Check that cip is not NULL before dereferencing it.
     - libdpkg: Add missing symbols to the version map.
     - libdpkg: Use the amount of available memory instead phys_mem/2.
       Thanks to Sebastian Andrzej Siewior <sebastian@breakpoint.cc>.
     - libdpkg: Refactor liblzma memlimit and cputhreads getters.
     - libdpkg: Dynamically allocate the buffers for de/compression I/O.
     - libdpkg: Increase I/O memory buffers from 4 to 32 KiB.
     - libdpkg: Pass struct compress_params as the first argument.
     - libdpkg: Pass struct compress_params to decompressors.
     - libdpkg: Refactor pkg_format_print() out from pkg_format_show().
     - libdpkg: Do not restrict source:* virtual fields to installed packages.
       Closes: #1004372
     - libdpkg: Do not allow argv with no arguments.
     - update-alternatives: Refactor alternative_has_broken_symlink().
     - update-alternatives: Move symlink removal inside fsys_symlink().
     - update-alternatives: Move filename generation outside alternative setup.
     - dselect: Rework windows resize to be signal safe.
     - scripts: Expand long word list into one entry per line.
     - scripts/mk: Remove unneeded conditionals.
       Thanks to Nicolas Boulenguez <nicolas@debian.org>.
     - scripts/mk: Indent code in Makefile fragments.
       Thanks to Nicolas Boulenguez <nicolas.boulenguez@free.fr>.
   * Build system:
     - Terminate lists in variables with «# EOL».
     - Move build helper tools into build-aux/.
     - Reorganize dpkg programs source code under src/.
     - Move autotest suite under src/.
     - Move C test suite machinery into lib/dpkg/.
     - Fix relocated autotest test suite runner.
     - Fix gitignore for build-aux/ directory.
     - Namespace Config variable usage.
       Reported by Павловец Сергей Николаевич
       <s.pavlovets@ivcmf.by>.
     - Add gitlab CI test for shared library building.
     - Link all programs against libcompat.
       Prompted by Jörg Sonnenberger <joerg@NetBSD.org>.
     - Remove unused TESTDATA variable from autotest suite.
     - Rework TAP check hooking into the autotools machinery.
     - Move EXTRA_DIST close to the files it is acting on.
     - Refactor autotest dependencies into a new variable.
     - Refactor autotest machinery into an automake include file.
     - Fold autotest test suite machinery into src/ from src/at/.
     - Rename do_path_subst to do_make_subst.
     - Factor out installation variable substitution into a new subst.am file.
     - Move shell scripts into src/.
     - Make AS_TR_* calls more clear.
     - Support compression library names with dashes.
     - Add zlib-ng support.
     - Rename HAVE_LZMA_MT to HAVE_LZMA_MT_ENCODER.
     - Move zsh completion under a subdirectory.
     - Switch sed pseudo-in-place replace invocations with copy then move.
     - Add comment about «sed -i» not being portable.
       Prompted by Nicolas Boulenguez <nicolas@debian.org>.
     - Mark libcompat and libdpkg as internal components in changelog.
     - Support specifying previous and next tags to gen-changelog.
     - Use non-capturing groups in regex in gen-changelog.
     - Do not hardcode «main» section for title check in gen-changelog.
     - Use sort flag instead of hardcoding the section in gen-changelog.
     - Do not put localization entries on their own changelog group.
   * Packaging:
     - Ignore directories for the alternatives state fixup. Closes: #1001695
     - Update bug-script to clarify usrmerge systems are unsupported.
     - Install aclocal m4 files into libdpkg-dev.
     - Install optional localized man pages with dh_installman.
     - Add a new not-installed file.
     - Switch to use the dh sequencer.
     - Make TESTSUITEFLAGS extensible.
     - Pass -jN to autotools autotest test suite via TESTSUITEFLAGS.
     - Update lintian overrides.
     - Use dpkg-error.sh in postinst.
     - Warn in postinst about merged-/usr-via-aliased-dirs breakage.
   * Test suite:
     - Move AT_TESTED to testsuite.at.
     - Run the tools with --version.
     - Rename DPKG_GEN_FILE to DPKG_GEN_CTRL_FILE.
     - Refactor control file template generation.
     - Rewrite dpkg-realpath test from TAP to autotest.
     - Rewrite dpkg-divert test from TAP to autotest.
     - Remove now unused TAP tests support from src/.
     - Print field type name instead of id in test case description.
     - Add new pkg-format unit tests.
     - Fix test_command_exec program invocation.
       Thanks to Sören Tempel <soeren+git@soeren-tempel.net>.
     - Use a temporary file instead of unportable «sed -i».
       Prompted by Nicolas Boulenguez <nicolas@debian.org>.
     - Add test case for SOURCE_DATE_EPOCH.
       Thanks to Nicolas Boulenguez <nicolas@debian.org>.
     - Use «each» instead of «keys» and value fetching.
       Thanks to Nicolas Boulenguez <nicolas@debian.org>.
   * Localization:
     - Remove previous msgid for non-fuzzy translations.
     - Update Catalan translations.
     - Update Dutch translation. Closes: #1003671, #1003672, #1003673
       Thanks to Frans Spiesschaert <Frans.Spiesschaert@yucom.be>.
     - Update Swedish translations. Closes: #1003799, #1007116
       Thanks to Peter Krefting <peter@softwolves.pp.se>.
 .
   [ Sven Joachim ]
   * Localization:
     - Update German dselect translation.
     - Update German programs translation.
 .
   [ Helge Kreutzmann ]
   * Localization:
     - Update German man pages translation.
     - Update German scripts translation.
Checksums-Sha1:
 4e7705251e41a39f66105cb0ed7ec9fd0d009784 2120 dpkg_1.21.2.dsc
 8a4de9dd775ef22b6093798a770c61fb025bf785 5051548 dpkg_1.21.2.tar.xz
 a236f84c64d76ea9cefcd270776900b7c4c22b0a 7791 dpkg_1.21.2_amd64.buildinfo
Checksums-Sha256:
 c284379b51c65d9e7352505b40ee8369f307c60ec99b9783207a3808ad1cbb1c 2120 dpkg_1.21.2.dsc
 b8fc67fca696c6bea2f40f737c80574d53384db25202f72effc7e4de4662e1ac 5051548 dpkg_1.21.2.tar.xz
 6bfbbd70e074f2de152e680b6cbee92b2fab34a50617c0b5f1b6c28ad6196da1 7791 dpkg_1.21.2_amd64.buildinfo
Files:
 9c859df7d9dc8ddf9eb874c124c9fc86 2120 admin required dpkg_1.21.2.dsc
 81bb09c52de45ec823ade6f7cf2055f0 5051548 admin required dpkg_1.21.2.tar.xz
 47ed989d0fba971a16a0202ece12a051 7791 admin required dpkg_1.21.2_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEETz509DYFDBD1aWV0uXK/PqSuV6MFAmIuRtcACgkQuXK/PqSu
V6MdWg//S7UljDES3wTOO36xJhykAu1zhHInQomyli8sGcz7KU8pZNmLEL+/a0su
gwoA/C8o1JDxTp3nlut7k5JsIfyNp9DFIKSyopQYc8vsuWw+ZJX9gDbFBE12evj7
QXTzVjHsulFCqr/KX9LRco8p4iHols3nEStPB3034Vma8/ffIptp7EjWSDbM72Tz
CHU0AaMOMyvfpNrjh0JCf2rbRXzB0yG+FiNIJD97wLbwi9uPOyEfWiAkQZ4/1cfm
tVfqmpfF2WuEk2OMwBdTbSh4EDMY+iwxNEsWaxtnHpnuFt/u8O1RHO95KUOQprIu
OU+hn/FcShYk/eCQT/00sXoVSE9z/kZDJWe9vnAmbhdgd4w76A/smEW5Sa/v2jSC
cKigoZi8eKHUdU6zvsioCaP/RbMNYT2RpP8yvKVipzuYniPSTiUafL9fmecXHXNf
oZbjgYW1vTKSZ25k6lgBlPqBn9L1+Sfyj3QD+EvO+Vf2ZQSjxpQLABqhdj0Cnxvv
pzIV2gXPyxgJdC/z4QGqhZAE+Z5agAeJB4L5FqnNJH9jL7RkMXDb7AuRI7gpQ1xR
pK8VKMvFdgplKc7sRk1O5g2BvK0fkPLgMwa0MXzM0KJ/1kMiNE/GZsbPfVUOAvdX
9HN0sL+74zc5l11eFLGXO1IiM90W9LR22TlEkXcYN6IMoW4RtkA=
=31ol
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.