Debian Bug report logs - #963788
dpkg: run maintscripts in a deterministic order to make effects reproducible

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Johannes 'josch' Schauer <josch@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: systemd: please make user order and ids of systemd and systemd-timesyncd reproducible

Date: Sat, 27 Jun 2020 07:54:53 +0200

Source: systemd
Severity: normal
User: reproducible-builds@lists.alioth.debian.org
Usertags: randomness

Hi,

the packages systemd and systemd-timesyncd depend on each other. This
means, that they form a dependency cycle and it is impossible to figure
out whether the postinst script of systemd or systemd-timesyncd should
be executed first. But depending on which postinst script is executed
first, the systemd-{journal,network,resolve} users from the systemd
postinst and the systemd-timesync user from the systemd-timesyncd
package will end up with differing user ids because they are chosen
sequentially, starting with 101. In addition to the user ids, the order
of users and their associated groups will differ in /etc/shadow,
/etc/passwd, /etc/group and associated files.

This is problem for reproducible installations because the exact same
package set, consisting of systemd and systemd-timesyncd can result in a
different system after installation.

Thanks!

cheers, josch

Message #10 received at 963788@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: "Johannes Schauer" <josch@debian.org>, 963788@bugs.debian.org, "Reproducible builds folks" <reproducible-builds@lists.alioth.debian.org>

Subject: Re: Bug#963788: systemd: please make user order and ids of systemd and systemd-timesyncd reproducible

Date: Mon, 29 Jun 2020 22:48:03 -0000

[adding reproducible-builds@lists.alioth.debian.org to CC]

Johannes 'josch' Schauer wrote:

> This is problem for reproducible installations because the exact same
> package set, consisting of systemd and systemd-timesyncd can result in a
> different system after installation.

I remember working on related issues in Tails which releases
bit-for-bit reproducible ISO images.

In the end, we went with a horrible post-build script that swapped
group IDs that were assigned non-deterministically due to the
arbitrary execution order of the postinst scripts.

I mention this here to encourage us exploring an archive-wide solution
rather than fixing every time it comes up.

This is a particularly good candidate for a general solution as, in my
hard-won experience:

 a) The non-determinism can happen infrequently and thus does not appear
    even in extensive testing.

 b) There was no way to flush out the problem in CI (compare using
    disorderfs to reverse your filesystem ordering to
    deterministically flush out non-deterministic behaviour or similar
    tricks.)

 c) Each test build run can take a significant amount of time.

 d) The packages could be entirely unrelated. As in, it could have
    been between entirely different packages that could not have been
    fixed by adding a relationship.

(Tails also has unrelated reasons for having persistent GIDs across
builds with different inputs. I would immediately concede that
these are out of scope for Debian itself to resolve.)

I'm not sure exactly where this change could be made (dpkg? apt?) as I
lack a confident understanding of the exact roles of those two tools,
but I am assuming that one of these is *eventually* deciding whether to
run the postinst for systemd or systemd-timesyncd first.


Regards,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `- 

Message #15 received at 963788@bugs.debian.org (full text, mbox, reply):

From: Balint Reczey <balint.reczey@canonical.com>

To: Chris Lamb <lamby@debian.org>, 963788@bugs.debian.org

Cc: Johannes Schauer <josch@debian.org>, Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>

Subject: Re: Bug#963788: systemd: please make user order and ids of systemd and systemd-timesyncd reproducible

Date: Tue, 30 Jun 2020 12:55:31 +0200

Control: reassign -1 dpkg 1.20.3
Control: affects -1 systemd

Hi All,

I agree with Chris, it would be better to find an archive-wide
solution than adding a hack to the two binary packages of systemd.
As I see the postinst scripts themselves are correct and the
nondeterminism comes from the order of executing them which is done by
dpkg, thus I'm reassigning this bug.

Cheers,
Balint

On Tue, Jun 30, 2020 at 12:51 AM Chris Lamb <lamby@debian.org> wrote:
>
> [adding reproducible-builds@lists.alioth.debian.org to CC]
>
> Johannes 'josch' Schauer wrote:
>
> > This is problem for reproducible installations because the exact same
> > package set, consisting of systemd and systemd-timesyncd can result in a
> > different system after installation.
>
> I remember working on related issues in Tails which releases
> bit-for-bit reproducible ISO images.
>
> In the end, we went with a horrible post-build script that swapped
> group IDs that were assigned non-deterministically due to the
> arbitrary execution order of the postinst scripts.
>
> I mention this here to encourage us exploring an archive-wide solution
> rather than fixing every time it comes up.
>
> This is a particularly good candidate for a general solution as, in my
> hard-won experience:
>
>  a) The non-determinism can happen infrequently and thus does not appear
>     even in extensive testing.
>
>  b) There was no way to flush out the problem in CI (compare using
>     disorderfs to reverse your filesystem ordering to
>     deterministically flush out non-deterministic behaviour or similar
>     tricks.)
>
>  c) Each test build run can take a significant amount of time.
>
>  d) The packages could be entirely unrelated. As in, it could have
>     been between entirely different packages that could not have been
>     fixed by adding a relationship.
>
> (Tails also has unrelated reasons for having persistent GIDs across
> builds with different inputs. I would immediately concede that
> these are out of scope for Debian itself to resolve.)
>
> I'm not sure exactly where this change could be made (dpkg? apt?) as I
> lack a confident understanding of the exact roles of those two tools,
> but I am assuming that one of these is *eventually* deciding whether to
> run the postinst for systemd or systemd-timesyncd first.
>
>
> Regards,
>
> --
>       ,''`.
>      : :'  :     Chris Lamb
>      `. `'`      lamby@debian.org chris-lamb.co.uk
>        `-
>


-- 
Balint Reczey
Ubuntu & Debian Developer

Send a report that this bug log contains spam.