Debian Bug report logs - #950585
binutils-dev: included log files introduce reproducibility issues

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: binutils-dev: included log files introduce reproducibility issues

Date: Mon, 03 Feb 2020 13:13:04 -0800

[Message part 1 (text/plain, inline)]

Package: binutils-dev
Severity: normal
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: toolchain randomness buildpath hostname timestamps username
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org

The log files included in /usr/share/doc/binutils/tests/ include many
variable things (timestamps, timings, PIDs, temporary filenames,
usernames, addresses) that vary between builds.

I'm not sure what the rationale for including these test logs in the
package is, but from a reproducible builds perspective, ideally these
would simply not be included at all.

When running the build with DEB_BUILD_OPTIONS=nocheck, the build logs
are excluded, and results in a reproducible build. Perhaps you could
create a build profile that doesn't include these logs, and use it in
builds in which you want the logs?

If the logs must be included in the packages, the attached patch
attempts to sanitize the log file by removing everything that varies
between builds; I'm not sure if this removes too much data to be useful,
but it should make binutils reproducible at least in bullseye
(unstable/experimental also varies build path in the reproducible builds
test infrastructure, which triggers other issues). This approach does
feel a bit prone to playing whack-a-mole, so long-term this may not be
viable.

Please consider removing or sanitizing the logs so that this part of the
essential build toolchain can itself be built reproducibly!


Thanks for maintaining binutils!


live well,
  vagrant

[0001-Sanitize-build-logs-to-ensure-reproducible-builds-by.patch (text/x-diff, inline)]

From ec5b5081300f275991c4b7dad7401d7ac882a6aa Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Mon, 3 Feb 2020 20:44:51 +0000
Subject: [PATCH] Sanitize build logs to ensure reproducible builds by
 stripping out embedded timestamps, timings, PIDs, temporary filenames,
 usernames, function addresses, etc.

---
 debian/rules | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/debian/rules b/debian/rules
index 1acbe57..0b202e3 100755
--- a/debian/rules
+++ b/debian/rules
@@ -1445,7 +1445,7 @@ binary.%: stamps/install.% install
 
 ifeq ($(with_check),yes)
 	: # remove user and date from test-summary for reproducible builds
-	sed -i -e '/Test Run By/Id' test-summary-$*
+	sed -i -e '/Test run by/d' test-summary-$*
 	$(install_file) test-summary-$* \
 	  $(D_CROSS)/$(PF)/share/doc/$(P_CROSS)/test-summary
 	gzip -9nf $(D_CROSS)/$(PF)/share/doc/$(P_CROSS)/test-summary
@@ -1668,7 +1668,7 @@ endif
 ifeq ($(DEB_BUILD_GNU_TYPE),$(DEB_HOST_GNU_TYPE))
 ifeq ($(with_check),yes)
 	: # remove user and date from test-summary for reproducible builds
-	sed -i -e '/Test Run By/Id' $(pwd)/test-summary
+	sed -i -e '/Test run by/d' $(pwd)/test-summary
 	$(install_dir) $(d_nat)/$(PF)/share/doc/$(p_bin)
 	$(install_file) test-summary \
 	  $(d_nat)/$(PF)/share/doc/$(p_bin)/test-summary-$(DEB_HOST_ARCH)
@@ -1680,8 +1680,27 @@ ifeq ($(with_check),yes)
 	for i in $$(find builddir-single -name '*.sum'); do \
 	  b=$$(basename $$i .sum); \
 	  $(install_file) $$i $(d_dev)/$(PF)/share/doc/$(p_bin)/tests/$$b.sum; \
+	  sed -i -e '/Test run by/d' $(d_dev)/$(PF)/share/doc/$(p_bin)/tests/$$b.sum; \
 	  xz -9v $(d_dev)/$(PF)/share/doc/$(p_bin)/tests/$$b.sum; \
 	  $(install_file) $${i%.sum}.log $(d_dev)/$(PF)/share/doc/$(p_bin)/tests/$$b.log; \
+	  sed -i -e '/Test run by/d' \
+	    -e '/time stamp.*:/d' \
+	    -e 's,completed in.*seconds,completed,g' \
+	    -e 's,runtest completed.*,runtest completed,g' \
+	    -e 's,completed at.*,completed,g' \
+	    -e 's,/tmp/cc.......,/tmp/ccXX.,g' \
+	    -e 's,tmpdir/compiler[0-9]*,tmpdir/compilerXX,g' \
+	    -e 's,tmpdir/plt[0-9]*,tmpdir/pltXX,g' \
+	    -e 's,tmpdir/nopie[0-9]*,tmpdir/nopieXX,g' \
+	    -e 's,tmpdir/int128[0-9]*,tmpdir/int128XX,g' \
+	    -e 's,tmpdir/gnu2_tls[0-9]*,tmpdir/gnu2_tlsXX,g' \
+	    -e 's,tmpdir/gnu[0-9]*,tmpdir/gnuXX,g' \
+	    -e 's,tmpdir/static[0-9]*,tmpdir/staticXX,g' \
+	    -e 's,tmpdir/dl_avail_test[0-9]*,tmpdir/dl_avail_testXX,g' \
+	    -e 's,tmpdir/ifunc[0-9]*,tmpdir/ifuncXX,g' \
+	    -e 's,func@0.*,func@ADDRESS_REDACTED_FOR_REPRODUCIBILITY,g' \
+	    -e "s,$(CURDIR),BUILD_DIR/,g" \
+	    $(d_dev)/$(PF)/share/doc/$(p_bin)/tests/$$b.log; \
 	  xz -9v $(d_dev)/$(PF)/share/doc/$(p_bin)/tests/$$b.log; \
 	done
 endif
-- 
2.20.1

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 950585@bugs.debian.org (full text, mbox, reply):

From: Matthias Klose <doko@debian.org>

To: 950585@bugs.debian.org, Vagrant Cascadian <vagrant@reproducible-builds.org>

Subject: Re: binutils-dev: included log files introduce reproducibility issues

Date: Sat, 22 Feb 2020 10:12:32 +0100

> I'm not sure what the rationale for including these test logs in the
> package is, but from a reproducible builds perspective, ideally these
> would simply not be included at all.

that's not an option.  this is all useful information for debugging purposes,
and you'll find that in the GCC packages as well.  Having it turned off by
default defeats the purpose.  I think I filed a bug report that builds should be
able to generate their own artifacts, as the autopkg tests are doing that,
however that probably will take a while to implement.

Or you could add a override database for files which are expected to differ.

Matthias

Message #15 received at 950585@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Matthias Klose <doko@debian.org>

Cc: reproducible-builds@lists.alioth.debian.org, 950585@bugs.debian.org

Subject: Re: binutils-dev: included log files introduce reproducibility issues

Date: Mon, 24 Feb 2020 14:09:23 -0800

[Message part 1 (text/plain, inline)]

On 2020-02-22, Matthias Klose wrote:
>> I'm not sure what the rationale for including these test logs in the
>> package is, but from a reproducible builds perspective, ideally these
>> would simply not be included at all.
>
> that's not an option.  this is all useful information for debugging purposes,
> and you'll find that in the GCC packages as well.  Having it turned off by
> default defeats the purpose.

So, my attempt at sanitizing them removed too much information; ok.

Why can't these test logs be output to the build log instead of being
embedded in the package? What's the use-case that needs them to be in
the package itself?

From what I recall, binutils was reproducible in debian testing for a
while before these logs were added. While GCC was not ever reproducible
thus far, it is another core part of the toolchain that I would hope
could one day be made reproducible.


> I think I filed a bug report that builds should be
> able to generate their own artifacts, as the autopkg tests are doing that,
> however that probably will take a while to implement.

I would be very interested in this bug report; against what package?


> Or you could add a override database for files which are expected to differ.

This is considerably more complicated than running a checksum on the
resulting .deb files and is another opportunity for bugs to lead to
incorrect reproducibility results... which I think has actually happened
when trying this kind of approach in the past, though I don't have a
reference off the top of my head.

Exploring avenues to put files like this into some separate artifact for
things that are not reproducible might be one avenue; I know that the
debian-installer packages ship some artifacts which are not
.deb/.dbgsym/.udeb... but this still makes it more difficult to compare
the resulting objects... worried about how to get that right, but maybe
we, as reproducible builds, need to explore that an an option?


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 950585@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: "Vagrant Cascadian" <vagrant@reproducible-builds.org>, "Matthias Klose" <doko@debian.org>

Cc: 950585@bugs.debian.org, "Reproducible builds folks" <reproducible-builds@lists.alioth.debian.org>

Subject: Re: binutils-dev: included log files introduce reproducibility issues

Date: Tue, 25 Feb 2020 01:57:04 -0000

Vagrant Cascadian wrote:

> > Or you could add a override database for files which are expected to differ.
> 
> This is considerably more complicated than running a checksum on the
> resulting .deb files and is another opportunity for bugs to lead to
> incorrect reproducibility results...

I would very much underline Vagrant's hesitation regarding a
centralised database. Such overrides would get out of date (or at
least out of sync) amongst many many other concerns including it,
albeit at a slight stretch, being a possible attack vector.

The ability to check reproducibility with no other knowledge or tools
other than cmp(1) or sha256sum(1) etc. does not seem to be that
important as it might initially appear but it extremely valuable as it
is so simple, engendering trust, lowering the barrier to entry,
reducing mistakes, etc. etc.

> which I think has actually happened when trying this kind of approach
> in the past, though I don't have a reference off the top of my head.

(Vagrant, are you perchance thinking of RPM? If I recall correctly, the
signatures in question there are embedded in the .rpm itself so you
need a special tool to even extract them.)


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Message #25 received at 950585@bugs.debian.org (full text, mbox, reply):

From: Matthias Klose <doko@debian.org>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>, 950585@bugs.debian.org

Cc: reproducible-builds@lists.alioth.debian.org

Subject: Re: Bug#950585: binutils-dev: included log files introduce reproducibility issues

Date: Fri, 28 Feb 2020 14:48:26 +0100

On 2/24/20 11:09 PM, Vagrant Cascadian wrote:
> On 2020-02-22, Matthias Klose wrote:
>>> I'm not sure what the rationale for including these test logs in the
>>> package is, but from a reproducible builds perspective, ideally these
>>> would simply not be included at all.
>>
>> that's not an option.  this is all useful information for debugging purposes,
>> and you'll find that in the GCC packages as well.  Having it turned off by
>> default defeats the purpose.
> 
> So, my attempt at sanitizing them removed too much information; ok.
> 
> Why can't these test logs be output to the build log instead of being
> embedded in the package? What's the use-case that needs them to be in
> the package itself?
> 
> From what I recall, binutils was reproducible in debian testing for a
> while before these logs were added. While GCC was not ever reproducible
> thus far, it is another core part of the toolchain that I would hope
> could one day be made reproducible.

cat'ing to stdout is possible, yes, however that will add to the log size.
gcc-N-test-results is a 13MB compressed package. This doesn't scale.

>> I think I filed a bug report that builds should be
>> able to generate their own artifacts, as the autopkg tests are doing that,
>> however that probably will take a while to implement.
> 
> I would be very interested in this bug report; against what package?

hmm, that was Ubuntu only:
https://bugs.launchpad.net/launchpad/+bug/1845159

Feel free to forward that. sbuild, pbuilder?

>> Or you could add a override database for files which are expected to differ.
> 
> This is considerably more complicated than running a checksum on the
> resulting .deb files and is another opportunity for bugs to lead to
> incorrect reproducibility results... which I think has actually happened
> when trying this kind of approach in the past, though I don't have a
> reference off the top of my head.
> 
> Exploring avenues to put files like this into some separate artifact for
> things that are not reproducible might be one avenue; I know that the
> debian-installer packages ship some artifacts which are not
> .deb/.dbgsym/.udeb... but this still makes it more difficult to compare
> the resulting objects... worried about how to get that right, but maybe
> we, as reproducible builds, need to explore that an an option?
> 
> 
> live well,
>   vagrant
> 

Message #30 received at 950585@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: 950585@bugs.debian.org, Matthias Klose <doko@debian.org>, Vagrant Cascadian <vagrant@reproducible-builds.org>, reproducible-builds <reproducible-builds@lists.alioth.debian.org>

Subject: Re: binutils-dev: included log files introduce reproducibility issues

Date: Sat, 16 May 2020 08:38:23 +0800

[Message part 1 (text/plain, inline)]

On Mon, 24 Feb 2020 14:09:23 -0800 Vagrant Cascadian wrote:

> Exploring avenues to put files like this into some separate artifact for
> things that are not reproducible might be one avenue

There is already the BYHAND (and automatic BYHAND) mechanisms for files
that get installed outside of pool/ in the Debian apt repository. Each
one needs support from dak too though.

https://salsa.debian.org/ftp-team/dak/-/tree/master/scripts/debian/
https://codesearch.debian.net/search?q=byhand+path%3Adebian&literal=0

The other option would be to put the files in a test results .deb file,
but that would still mean repro-builds folks would compare them, unless
there were a naming convention that could be used to ignore them.

It strikes me that these files are most similar to .buildinfo or the
build logs in that they are data *about* the builds. I've wanted
maintainers to be able to also upload build logs with their binary
builds and started a WIP patch for that.

https://salsa.debian.org/pabs/dak/-/commits/maintainer-build-logs

It looks like dpkg-genchanges can already add more files through the
debian/files input file. I managed to get my build log included in my
.changes file using this mechanism. I then attempted to upload it to
the Debian archive, queued gladly accepted the upload but dak rejected
it saying that it looks like a BYHAND package:

   whowatch_1.8.6-2_amd64.changes: whowatch_1.8.6-1_amd64.build.log looks like a byhand package, but is in section build

I suggest that the dpkg-dev maintainer and the ftp-masters should be
talked to about this topic. Probably the right mechanism is to have a
convention in debian/files similar to how dbgsyms are represented and
similar to byhand but the files go into the pool like .deb files do.

   whowatch_1.8.6-1_amd64.build build optional automatic=yes
   whowatch-dbgsym_1.8.6-2_amd64.deb debug optional automatic=yes

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]

Message #35 received at 950585@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: "Paul Wise" <pabs@debian.org>, 950585@bugs.debian.org, "Matthias Klose" <doko@debian.org>, "Vagrant Cascadian" <vagrant@reproducible-builds.org>, "Reproducible builds folks" <reproducible-builds@lists.alioth.debian.org>

Subject: Re: binutils-dev: included log files introduce reproducibility issues

Date: Sat, 16 May 2020 22:58:05 -0000

Hi Paul,

> There is already the BYHAND (and automatic BYHAND) mechanisms for files
> that get installed outside of pool/ in the Debian apt repository. Each
> one needs support from dak too though.
[..]
> It strikes me that these files are most similar to .buildinfo or the
> build logs in that they are data *about* the builds. I've wanted
> maintainers to be able to also upload build logs with their binary
> builds and started a WIP patch for that.

My gut feeling is that this is the avenue we want to explore. Having a
separate mechanism to capture this build-specific metadata would be an
elegant solution and, as you imply, having the logs would have QA
advantages as well as permit reproducible builds. The system could be
generic enough for future use-cases that we cannot envisage too.

> The other option would be to put the files in a test results .deb file,
> but that would still mean repro-builds folks would compare them, unless
> there were a naming convention that could be used to ignore them.

This sort of thinking always makes me a little uneasy. We have taken
great pains over the years that no special knowledge, tools or tricks
are required to compare the artifacts of a Debian build with respect
to reproducibility.

All you need right now is sha1sum(1) or any cmp(1)-like command-line
tool. This is partly due to aesthetics but I have further observed
that distributions and platforms that have not adopted this fail-safe
approach are always at a marked disadvantage. This would appear to be
a superficial niggle, but it makes a huge difference in mindset with
many 1st and 2nd effects.


Best wishes,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `- 

Message #40 received at 950585@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: Chris Lamb <lamby@debian.org>, 950585@bugs.debian.org, Matthias Klose <doko@debian.org>, Vagrant Cascadian <vagrant@reproducible-builds.org>, Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>

Subject: Re: binutils-dev: included log files introduce reproducibility issues

Date: Sun, 17 May 2020 09:09:42 +0800

[Message part 1 (text/plain, inline)]

On Sat, 2020-05-16 at 22:58 +0000, Chris Lamb wrote:

> My gut feeling is that this is the avenue we want to explore. Having a
> separate mechanism to capture this build-specific metadata would be an
> elegant solution and, as you imply, having the logs would have QA
> advantages as well as permit reproducible builds. The system could be
> generic enough for future use-cases that we cannot envisage too.

Agreed that this is the best option. In order to standardise the
naming, structure and organisation of the data in order to make it work
across all Debian derivatives, probably a new conversation needs to be
started between ftp-master and the dpkg maintainer.

> We have taken great pains over the years that no special knowledge,
> tools or tricks are required to compare the artifacts of a Debian
> build with respect to reproducibility.

I'm not involved in repro builds enough so your statement leads me to
wonder how you deal with ignoring the inevitable differences between
the buildinfo files, which would record the inevitable differences in
build environment between different builders. I'm guessing you just
ignore all differences in buildinfo files and would have to add to that
ignoring differences between build logs and other build metadata?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]

Message #45 received at 950585@bugs.debian.org (full text, mbox, reply):

From: Matthias Klose <doko@debian.org>

To: Chris Lamb <lamby@debian.org>, Paul Wise <pabs@debian.org>, 950585@bugs.debian.org, Vagrant Cascadian <vagrant@reproducible-builds.org>, Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>

Subject: Re: binutils-dev: included log files introduce reproducibility issues

Date: Sun, 17 May 2020 13:12:10 +0200

On 5/17/20 12:58 AM, Chris Lamb wrote:
> Hi Paul,
> 
>> There is already the BYHAND (and automatic BYHAND) mechanisms for files
>> that get installed outside of pool/ in the Debian apt repository. Each
>> one needs support from dak too though.
> [..]
>> It strikes me that these files are most similar to .buildinfo or the
>> build logs in that they are data *about* the builds. I've wanted
>> maintainers to be able to also upload build logs with their binary
>> builds and started a WIP patch for that.

I thought I also had filed a bug report for Debian, but there's only one in LP,
describing build artifacts for both successful and failing builds. See
https://bugs.launchpad.net/launchpad/+bug/1845159

Any idea which package this should be filed for?

Matthias

Message #50 received at 950585@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: Matthias Klose <doko@debian.org>, Chris Lamb <lamby@debian.org>, 950585@bugs.debian.org, Vagrant Cascadian <vagrant@reproducible-builds.org>, Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>

Subject: Re: binutils-dev: included log files introduce reproducibility issues

Date: Sun, 17 May 2020 20:55:11 +0800

[Message part 1 (text/plain, inline)]

On Sun, 2020-05-17 at 13:12 +0200, Matthias Klose wrote: 

> Any idea which package this should be filed for?

I think this is going to require a bunch of different changes in
various places including dpkg-dev, debhelper, reprepro, dak, launchpad
and any other apt repos that have an incoming system. As long as the
files are mentioned in the .changes files dupload/dput won't change. So
probably there needs to be a discussion on debian-devel CCed to the
debian-dpkg list and to the ftp-masters and other folks.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.