Debian Bug report logs - #926242
jenkins.debian.org: Please test reproducibility status of Debian Installer images

Package: jenkins.debian.org; Maintainer for jenkins.debian.org is Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>;

Reported by: "Chris Lamb" <lamby@debian.org>

Date: Tue, 2 Apr 2019 13:33:02 UTC

Severity: wishlist

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox

Report forwarded to debian-bugs-dist@lists.debian.org, debian-boot@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Tue, 02 Apr 2019 13:33:04 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
New Bug report received and forwarded. Copy sent to debian-boot@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Tue, 02 Apr 2019 13:33:04 GMT) (full text, mbox, link).

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: submit@bugs.debian.org
Subject: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Tue, 02 Apr 2019 09:30:58 -0400
Package: jenkins.debian.org
Severity: wishlist
X-Debbugs-CC: debian-boot@lists.debian.org

Hi,

(CC'ing the Installer Team as they likely have input.)

In a number of commits and bugs (#920631, #920676, etc. etc.) I've
managed to make the Debian Installer images reproducible, at least on
the architectures and environment variations I have easy access to.

However, it would be great if we had some continuous testing of this,
with the usual bells-and-whistles of running/publishing diffoscope
reports, etc.

Due to the d-i testing release cycle it would be great to find any
regressions so they can get merged in time for whatever the next
alpha/beta is particularly as we (completely correctly!) get more
conservative with any changes respect to the upcoming release of
"buster."

Whilst I think of it, there is also the separate issue of ensuring we
generate a .buildinfo (or .buildinfo-like) build attestation document so
that others can reproduce the build at a later date and further ensure
this is published or otherwise available somewhere for official
releases… but I was hoping it would become more obvious what we needed
(without guessing) once we have testing.

Mattia, is this something you could proof-of-concept…?


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Tue, 02 Apr 2019 14:12:02 GMT) (full text, mbox, link).

Message #8 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>
To: Chris Lamb <lamby@debian.org>, 926242@bugs.debian.org
Cc: debian-boot@lists.debian.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Tue, 2 Apr 2019 16:09:18 +0200
[Message part 1 (text/plain, inline)]
user jenkins.debian.org@packages.debian.org
usertags 926242 reproducible
thanks

On Tue, Apr 02, 2019 at 09:30:58AM -0400, Chris Lamb wrote:
> However, it would be great if we had some continuous testing of this,
> with the usual bells-and-whistles of running/publishing diffoscope
> reports, etc.
> 
> Due to the d-i testing release cycle it would be great to find any
> regressions so they can get merged in time for whatever the next
> alpha/beta is particularly as we (completely correctly!) get more
> conservative with any changes respect to the upcoming release of
> "buster."

Does the installer need anything special?  I thought d-i was just like
any other package when it came to regular building it.

> Whilst I think of it, there is also the separate issue of ensuring we
> generate a .buildinfo (or .buildinfo-like) build attestation document so
> that others can reproduce the build at a later date and further ensure
> this is published or otherwise available somewhere for official
> releases… but I was hoping it would become more obvious what we needed
> (without guessing) once we have testing.

If d-i is like I assume it be, there should a regular buildinfo as well,
see https://buildinfo.debian.net/sources/debian-installer for what I
mean.

> Mattia, is this something you could proof-of-concept…?

So the reason src:debian-installer does not build at this moment, it's
because it *is* a bit of a snowflake as it wants to access a debian
archive while it builds.  Currently pbuilder doesn't have a "whistlist"
method to block all networking but to a particular site, which is what
would be needed to solve this nicely.

One way to workaround this problem of src:debian-installer, would be for
our building script to instruct pbuilder to not block the network when
it's building this special package.  I think that's acceptable, given
that d-i is not a random package... :)


Am I missing some other detail?

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 09:15:05 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 03 Apr 2019 09:15:05 GMT) (full text, mbox, link).

Message #13 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Mattia Rizzolo" <mattia@debian.org>, 926242@bugs.debian.org
Cc: debian-boot@lists.debian.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 03 Apr 2019 05:01:44 -0400
Hey Mattia et al.,

> Does the installer need anything special?  I thought d-i was just like
> any other package when it came to regular building it.

For some reason I thought that src:debian-installer was a special-case
package (or simply used for its build-depends) and it does not
"really" build the images. Obviously, I hope you are absolutely right
and this is much easier than I thought to resolve.

(I think I got some of this preconception from various README files
which imply one should build via the Makefile, not via building
src:debian-installer.)

Thinking about it, a d-i *release* might trigger or other go through
some other codepaths — perhaps even manual ones — that might lead to
an unreproducible package.. and thus we would definitely want to
exercise these.

> If d-i is like I assume it be, there should a regular buildinfo as well,
> see https://buildinfo.debian.net/sources/debian-installer for what I
> mean.

Nod. However, we would need to double check that the src:debian-
installer .buildinfo includes or otherwise takes into consideration:

 a) The actual generated ISOs/.img/netboot images etc. and not just
    the binary packages as usual. (I mean we only really care about
    the former here, right?)

 b) Any sources downloaded directly from the archive.

As above, perhaps there is some kind of "release" codepath we need to
definitely and specifically test too? -boot can you chime in here on
this angle?

> So the reason src:debian-installer does not build at this moment [..]

For completeness (and/or people following along):

  http://tests.reproducible-builds.org/debian-installer

… specifically:

  make[2]: Entering directory '/build/1st/debian-installer-20190118/build'
  WARNING: mirror 'http://deb.debian.org/debian' appears to be invalid; skipping
  WARNING: mirror 'http://deb.debian.org/debian' appears to be invalid; skipping
  […]
  Building dependency tree...
  E: Unable to locate package acpi-modules-4.19.0-1-amd64-di
  E: Couldn't find any package by glob 'acpi-modules-4.19.0-1-amd64-di'
  E: Couldn't find any package by regex 'acpi-modules-4.19.0-1-amd64-di'
  […]

> One way to workaround this problem of src:debian-installer, would be for
> our building script to instruct pbuilder to not block the network when
> it's building this special package.

I think that's fine, especially if we add this to the log and that
it's well-commented as a magic exception. You happy to go-ahead with
this?


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 09:21:03 GMT) (full text, mbox, link).

Acknowledgement sent to Samuel Thibault <sthibault@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 03 Apr 2019 09:21:03 GMT) (full text, mbox, link).

Message #18 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Samuel Thibault <sthibault@debian.org>
To: Chris Lamb <lamby@debian.org>
Cc: Mattia Rizzolo <mattia@debian.org>, 926242@bugs.debian.org, debian-boot@lists.debian.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 3 Apr 2019 11:17:57 +0200
Hello,

Chris Lamb, le mer. 03 avril 2019 05:01:44 -0400, a ecrit:
> > Does the installer need anything special?  I thought d-i was just like
> > any other package when it came to regular building it.
> 
> For some reason I thought that src:debian-installer was a special-case
> package (or simply used for its build-depends) and it does not
> "really" build the images.

It doesn't build the netinst/CD/DVD iso images indeed (debian-cd handles
that). But it builds the initrd used there (and the netboot mini.iso).

Samuel

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 09:27:06 GMT) (full text, mbox, link).

Acknowledgement sent to Cyril Brulebois <kibi@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 03 Apr 2019 09:27:06 GMT) (full text, mbox, link).

Message #23 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>
To: Chris Lamb <lamby@debian.org>, Mattia Rizzolo <mattia@debian.org>, 926242@bugs.debian.org, debian-boot@lists.debian.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 3 Apr 2019 11:25:29 +0200
[Message part 1 (text/plain, inline)]
Samuel Thibault <sthibault@debian.org> (2019-04-03):
> Hello,
> 
> Chris Lamb, le mer. 03 avril 2019 05:01:44 -0400, a ecrit:
> > > Does the installer need anything special?  I thought d-i was just like
> > > any other package when it came to regular building it.
> > 
> > For some reason I thought that src:debian-installer was a special-case
> > package (or simply used for its build-depends) and it does not
> > "really" build the images.
> 
> It doesn't build the netinst/CD/DVD iso images indeed (debian-cd handles
> that). But it builds the initrd used there (and the netboot mini.iso).

Right. Check the tarball (!) produced by building src:debian-installer;
that's what gets installed in installer-$arch directories in the
archive; then consumed by debian-cd to produce “full-blown” installation
images.

(You might have seen some special casing in dak; that's us!)


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 09:51:06 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 03 Apr 2019 09:51:06 GMT) (full text, mbox, link).

Message #28 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Cyril Brulebois" <kibi@debian.org>, "Mattia Rizzolo" <mattia@debian.org>, 926242@bugs.debian.org, debian-boot@lists.debian.org, "Samuel Thibault" <sthibault@debian.org>
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 03 Apr 2019 05:50:26 -0400
Hey Cyril & Samuel,

> > It doesn't build the netinst/CD/DVD iso images indeed (debian-cd handles
> > that). But it builds the initrd used there (and the netboot mini.iso).
> 
> Right. Check the tarball (!) produced by building src:debian-installer;
> that's what gets installed in installer-$arch directories in the
> archive; then consumed by debian-cd to produce “full-blown” installation
> images.

TIL. However, as these generated files do not appear in the binary
debian-installer package it is likely that that our testing framework
will (after the mooted networking exception is made) entirely-
correctly report that the src:debian-installer package is reproducible
as its declared artifects contain only documentation. This will be
somewhat misleading about the true reproducibility status of our installer.

Just throwing out ideas here but perhaps this binary package could
contain at least the hashes of the generated files you mention? That way,
at least if they vary between our test builds then we will implicitly see
that in the package's reproducibility status.

However, as this would not incorporate anything that debian-cd does
with them to produce the "full-blown" images I suspect that this will
not be enough to cover everything. Just to underline this point in a
silly way we would not be aware of, for example, debian-cd running
"echo $RANDOM >> /target/ somefile.txt", even with the above hack.

Thanks for your input btw. :)


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 09:57:03 GMT) (full text, mbox, link).

Acknowledgement sent to Holger Levsen <holger@layer-acht.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 03 Apr 2019 09:57:03 GMT) (full text, mbox, link).

Message #33 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>
To: Chris Lamb <lamby@debian.org>
Cc: Cyril Brulebois <kibi@debian.org>, Mattia Rizzolo <mattia@debian.org>, 926242@bugs.debian.org, debian-boot@lists.debian.org, Samuel Thibault <sthibault@debian.org>
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 3 Apr 2019 09:54:59 +0000
[Message part 1 (text/plain, inline)]
Hi,

kudos for the progress so far!

On Wed, Apr 03, 2019 at 05:50:26AM -0400, Chris Lamb wrote:
> Just throwing out ideas here but perhaps this binary package could
> contain at least the hashes of the generated files you mention? 

or we setup debian-cd builds as well..!


-- 
tschau,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 10:06:02 GMT) (full text, mbox, link).

Message #36 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>
To: Chris Lamb <lamby@debian.org>
Cc: Cyril Brulebois <kibi@debian.org>, 926242@bugs.debian.org, debian-boot@lists.debian.org, Samuel Thibault <sthibault@debian.org>
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 3 Apr 2019 12:01:50 +0200
[Message part 1 (text/plain, inline)]
On Wed, Apr 03, 2019 at 05:50:26AM -0400, Chris Lamb wrote:
> Hey Cyril & Samuel,
> 
> > > It doesn't build the netinst/CD/DVD iso images indeed (debian-cd handles
> > > that). But it builds the initrd used there (and the netboot mini.iso).
> > 
> > Right. Check the tarball (!) produced by building src:debian-installer;
> > that's what gets installed in installer-$arch directories in the
> > archive; then consumed by debian-cd to produce “full-blown” installation
> > images.
> 
> TIL. However, as these generated files do not appear in the binary
> debian-installer package it is likely that that our testing framework
> will (after the mooted networking exception is made) entirely-
> correctly report that the src:debian-installer package is reproducible
> as its declared artifects contain only documentation. This will be
> somewhat misleading about the true reproducibility status of our installer.

It doesn't contain only documentation.
src:debian-installer also builds a
debian-installer-images_$(VERSION)_$(ARCH).tar.gz that does contain
binary stuff, including the initrd, kernel image, mini.iso, etc etc.

> However, as this would not incorporate anything that debian-cd does
> with them to produce the "full-blown" images I suspect that this will
> not be enough to cover everything. Just to underline this point in a
> silly way we would not be aware of, for example, debian-cd running
> "echo $RANDOM >> /target/ somefile.txt", even with the above hack.

Right, to cover this we would need to do a full build of the cd image.
I have no idea whatsoever how that's done (and, as others said, it's
outside of the debian-boot office, it's within debian-cd).

In the meantime I did what I mentioned,
https://salsa.debian.org/qa/jenkins.debian.net/commit/e3117ca244b230c04e324814e20c02032026a5cf
and the build for unstable/arm64 is running.

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 10:24:03 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 03 Apr 2019 10:24:03 GMT) (full text, mbox, link).

Message #41 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Mattia Rizzolo" <mattia@debian.org>
Cc: "Cyril Brulebois" <kibi@debian.org>, 926242@bugs.debian.org, debian-boot@lists.debian.org, "Samuel Thibault" <sthibault@debian.org>
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 03 Apr 2019 06:21:39 -0400
Dear Mattia,

> > TIL. However, as these generated files do not appear in the binary
> > debian-installer package it is likely that that our testing framework
> > will (after the mooted networking exception is made) entirely-
> > correctly report that the src:debian-installer package is reproducible
> > as its declared artifects contain only documentation. This will be
> > somewhat misleading about the true reproducibility status of our installer.
> 
> It doesn't contain only documentation.
> src:debian-installer also builds a
> debian-installer-images_$(VERSION)_$(ARCH).tar.gz that does contain
> binary stuff, including the initrd, kernel image, mini.iso, etc etc.

Whilst it may build indeed these files they do not appear in the
binary package:

    $ find | head
    .
    ./usr
    ./usr/share
    ./usr/share/doc
    ./usr/share/doc/debian-installer
    ./usr/share/doc/debian-installer/talks
    ./usr/share/doc/debian-installer/talks/fosdem07
    ./usr/share/doc/debian-installer/talks/fosdem07/README
    ./usr/share/doc/debian-installer/talks/fosdem07/fosdem1.tgz
    ./usr/share/doc/debian-installer/talks/d-i_debconf7

    $ find | grep images

    $ 

Thus, they cannot affect the reproducibility status of the debian-
installer source package. This prompted my paragraph regarding at
least including these files' hashes, etc.

> Right, to cover this we would need to do a full build of the cd image.
> I have no idea whatsoever how that's done (and, as others said, it's
> outside of the debian-boot office, it's within debian-cd).

I see I've been conflating the -boot and -cd work here, sorry. Well,
if full debian-cd rebuilds are out of scope for now (!) we should
at-least cover any "debian-boot" related stuff.

Now, I am certain I am missing something but would this latter
approach just (!) involve adding another Jenkins job that tracks the
HEAD of debian- installer.git (ooh, another advantage; time) which
calls "make -C build release" in two environments?


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 10:36:03 GMT) (full text, mbox, link).

Message #44 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>
To: Chris Lamb <lamby@debian.org>
Cc: Cyril Brulebois <kibi@debian.org>, 926242@bugs.debian.org, debian-boot@lists.debian.org, Samuel Thibault <sthibault@debian.org>
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 3 Apr 2019 12:32:34 +0200
[Message part 1 (text/plain, inline)]
On Wed, Apr 03, 2019 at 06:21:39AM -0400, Chris Lamb wrote:
> Whilst it may build indeed these files they do not appear in the
> binary package:

> Thus, they cannot affect the reproducibility status of the debian-
> installer source package. This prompted my paragraph regarding at
> least including these files' hashes, etc.

Yes they do!

The tricky part is that that "tarball" we have been talking about is not
saved anywhere.  Once it is uploaded to ftp-master it's unpacked and
then discarded, so you can't quite get your hands on it without doing a
local build of debian-installer.

Furthermore now d-i as released wouldn't build anyway because it tries
to look up linux 4.19.0-1 instead of 4.19.0-4... And after changing that
it tries to look for the non-existent hyperv-modules-4.19.0-4-amd64-di,
and here I don't want to go digging in d-i… :( - so I can't show you
that tarball I'm talking about right now...

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 10:48:02 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 03 Apr 2019 10:48:03 GMT) (full text, mbox, link).

Message #49 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Mattia Rizzolo" <mattia@debian.org>
Cc: "Cyril Brulebois" <kibi@debian.org>, 926242@bugs.debian.org, debian-boot@lists.debian.org, "Samuel Thibault" <sthibault@debian.org>
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 03 Apr 2019 06:45:28 -0400
Hi Mattia,

> > Whilst it may build indeed these files they do not appear in the
> > binary package:
> 
> > Thus, they cannot affect the reproducibility status of the debian-
> > installer source package. […]
>
> Yes they do!

Oh, hang on, does src:debian-installer's .changes file include these
extra files? That might be what I'm missing.

My conception is that as we call diffoscope on the two .changes files
it will report they are reproducible as, well, the binary package will
(likely…) be identical.

> The tricky part is that that "tarball" we have been talking about is not
> saved anywhere.

Nod, I guess we would want that for true jenkins.debian.org "please
save the build artifacts" support but for now getting a red/green
light would be interesting in/of itself.


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 10:51:02 GMT) (full text, mbox, link).

Acknowledgement sent to Cyril Brulebois <kibi@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 03 Apr 2019 10:51:02 GMT) (full text, mbox, link).

Message #54 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>
To: Chris Lamb <lamby@debian.org>
Cc: Mattia Rizzolo <mattia@debian.org>, 926242@bugs.debian.org, debian-boot@lists.debian.org, Samuel Thibault <sthibault@debian.org>
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 3 Apr 2019 12:47:52 +0200
[Message part 1 (text/plain, inline)]
Chris Lamb <lamby@debian.org> (2019-04-03):
> Hi Mattia,
> 
> > > Whilst it may build indeed these files they do not appear in the
> > > binary package:
> > 
> > > Thus, they cannot affect the reproducibility status of the debian-
> > > installer source package. […]
> >
> > Yes they do!
> 
> Oh, hang on, does src:debian-installer's .changes file include these
> extra files? That might be what I'm missing.

Yes:

    kibi@armor:~/debian-installer$ dcmd debian-installer_20190118_amd64.changes
    debian-installer-images_20190118_amd64.tar.gz
    debian-installer_20190118_amd64.buildinfo
    debian-installer_20190118_amd64.deb
    debian-installer_20190118_amd64.changes

The tarball is uploaded alongside the other files; “just” treated
“somewhat differently” by dak.
 

Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 10:51:04 GMT) (full text, mbox, link).

Acknowledgement sent to Samuel Thibault <sthibault@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 03 Apr 2019 10:51:04 GMT) (full text, mbox, link).

Message #59 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Samuel Thibault <sthibault@debian.org>
To: Chris Lamb <lamby@debian.org>
Cc: Mattia Rizzolo <mattia@debian.org>, Cyril Brulebois <kibi@debian.org>, 926242@bugs.debian.org, debian-boot@lists.debian.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 3 Apr 2019 12:48:51 +0200
Chris Lamb, le mer. 03 avril 2019 06:45:28 -0400, a ecrit:
> > > Whilst it may build indeed these files they do not appear in the
> > > binary package:
> > 
> > > Thus, they cannot affect the reproducibility status of the debian-
> > > installer source package. […]
> >
> > Yes they do!
> 
> Oh, hang on, does src:debian-installer's .changes file include these
> extra files? That might be what I'm missing.

It seems so, see in

https://buildd.debian.org/status/fetch.php?pkg=debian-installer&arch=amd64&ver=20190118&stamp=1547858681&raw=0

Checksums-Sha1:
 726c95c9add3222b6c4b744f8cf1e18afd44437e 528261542 debian-installer-images_20190118_amd64.tar.gz
 324185c8187548dd86e301910751e1f83413ae5c 10730 debian-installer_20190118_amd64.buildinfo
 870d941832bcdd7fbde0b04974d5e46c812d76b2 734340 debian-installer_20190118_amd64.deb
Checksums-Sha256:
 698b1c32e0f3d1da174ed96af64040d65eac955ccec110d7b651b615ff3c74ae 528261542 debian-installer-images_20190118_amd64.tar.gz
 633371431ea7a425163f7a8360d56e1fa401fffcfff8612fe80078cdb40add4b 10730 debian-installer_20190118_amd64.buildinfo
 7e82f489df6d36a37bc9c810cb24d4cfbdc1eb95e5aa539d19691c28d24c6dfa 734340 debian-installer_20190118_amd64.deb
Files:
 843efa1eba8e58e04bd174647431fbc6 528261542 raw-installer - debian-installer-images_20190118_amd64.tar.gz
 9daa2e9db4355b189c60132a97eb4103 10730 devel optional debian-installer_20190118_amd64.buildinfo
 4c9e4745b8dd0e673af2548f11bc5796 734340 devel optional debian-installer_20190118_amd64.deb

Samuel

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 03 Apr 2019 11:00:03 GMT) (full text, mbox, link).

Message #62 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>
To: Chris Lamb <lamby@debian.org>
Cc: Cyril Brulebois <kibi@debian.org>, 926242@bugs.debian.org, debian-boot@lists.debian.org, Samuel Thibault <sthibault@debian.org>
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 3 Apr 2019 12:56:56 +0200
[Message part 1 (text/plain, inline)]
On Wed, Apr 03, 2019 at 06:45:28AM -0400, Chris Lamb wrote:
> My conception is that as we call diffoscope on the two .changes files
> it will report they are reproducible as, well, the binary package will
> (likely…) be identical.

Yup.  It would.

> > The tricky part is that that "tarball" we have been talking about is not
> > saved anywhere.
> 
> Nod, I guess we would want that for true jenkins.debian.org "please
> save the build artifacts" support but for now getting a red/green
> light would be interesting in/of itself.

I think it would work out of the box already.

Now, regarding building d-i as a normal package, I hit a bit of a
readblock because it fails while trying to download the files in the
nodes running in the future.
What d-i does it copying the url from the host's (or, well, the
chroot's) /etc/apt/sources.list, but it seems it doesn't also pick up an
eventual [check-valid-until=no] placed on the same line :\

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Sat, 18 May 2019 10:48:02 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Sat, 18 May 2019 10:48:02 GMT) (full text, mbox, link).

Message #67 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Mattia Rizzolo" <mattia@debian.org>
Cc: 926242@bugs.debian.org, debian-boot@lists.debian.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Sat, 18 May 2019 11:45:20 +0100
[dropping CCs already subscribed to debian-boot]

Hi Mattia,

> Now, regarding building d-i as a normal package, I hit a bit of a
> readblock because it fails while trying to download the files in the
> nodes running in the future.

Sounds about right:

  https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/debian-installer.html

    get-packages udeb  
    make[10]: 'sources.list.udeb' is up to date.
    Ign:1 copy:/build/debian-installer-20190410/2nd/build localudebs/ InRelease
    Ign:2 copy:/build/debian-installer-20190410/2nd/build localudebs/ Release
    Ign:3 copy:/build/debian-installer-20190410/2nd/build localudebs/ Packages
    Ign:3 copy:/build/debian-installer-20190410/2nd/build localudebs/ Packages
    Ign:3 copy:/build/debian-installer-20190410/2nd/build localudebs/ Packages
    Get:3 copy:/build/debian-installer-20190410/2nd/build localudebs/ Packages [20 B]
    Get:4 http://cdn-fastly.deb.debian.org/debian buster InRelease [163 kB]
    Reading package lists...
    E: Release file for http://deb.debian.org/debian/dists/buster/InRelease is expired (invalid since 391d 12h 5min 44s). Updates for this repository will not be applied.

> What d-i does it copying the url from the host's (or, well, the
> chroot's) /etc/apt/sources.list, but it seems it doesn't also pick up an
> eventual [check-valid-until=no] placed on the same line :\

Nod. Any further thoughts on this? It would be great to see where we
are at here...


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Sat, 25 May 2019 02:00:04 GMT) (full text, mbox, link).

Acknowledgement sent to Vagrant Cascadian <vagrant@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Sat, 25 May 2019 02:00:04 GMT) (full text, mbox, link).

Message #72 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@debian.org>
To: Chris Lamb <lamby@debian.org>, 926242@bugs.debian.org, Mattia Rizzolo <mattia@debian.org>
Cc: debian-boot@lists.debian.org, Colin Watson <cjwatson@debian.org>
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Fri, 24 May 2019 18:56:35 -0700
[Message part 1 (text/plain, inline)]
On 2019-05-18, Chris Lamb wrote:
>> Now, regarding building d-i as a normal package, I hit a bit of a
>> readblock because it fails while trying to download the files in the
>> nodes running in the future.
>
> Sounds about right:
>
>   https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/debian-installer.html
>
>     get-packages udeb  
>     make[10]: 'sources.list.udeb' is up to date.
>     Get:3 copy:/build/debian-installer-20190410/2nd/build localudebs/ Packages [20 B]
...
>     Get:4 http://cdn-fastly.deb.debian.org/debian buster InRelease [163 kB]
>     Reading package lists...
>     E: Release file for http://deb.debian.org/debian/dists/buster/InRelease is expired (invalid since 391d 12h 5min 44s). Updates for this repository will not be applied.
>
>> What d-i does it copying the url from the host's (or, well, the
>> chroot's) /etc/apt/sources.list, but it seems it doesn't also pick up an
>> eventual [check-valid-until=no] placed on the same line :\
>
> Nod. Any further thoughts on this? It would be great to see where we
> are at here...

Colin Watson removed it back in 2011:

  https://salsa.debian.org/installer-team/debian-installer/commit/fa965c32ca8bfa2ff14886c6f0dca131532815c7

commit fa965c32ca8bfa2ff14886c6f0dca131532815c7
Author: Colin Watson <cjwatson@debian.org>
Date:   Mon Mar 14 18:08:25 2011 +0000

    Skip the option field in sources.list lines, if present.                                                   

diff --git a/build/util/gen-sources.list.udeb b/build/util/gen-sources.list.udeb                               
index e86b4fa66..9f140100e 100755
--- a/build/util/gen-sources.list.udeb
+++ b/build/util/gen-sources.list.udeb
@@ -36,6 +36,7 @@ get_mirrors() {
                [ -s $file ] || continue
                grep '^deb[[:space:]]' $file | \
                   grep -v '^deb[[:space:]]\+cdrom:' | \                                                       
+                  sed 's,^deb \[[^]]*\] ,deb ,' | \
                   grep -v '\(security.debian.org\|volatile.debian.\(net\|org\)\)' | \                         
                   grep '[[:space:]]main' | \
                   awk '{print $1 " " $2}' | \


Happen to remember what it breaks to have these present? If it was a
workaround, perhaps it is no longer needed?


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Sat, 25 May 2019 11:33:03 GMT) (full text, mbox, link).

Acknowledgement sent to Colin Watson <cjwatson@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Sat, 25 May 2019 11:33:03 GMT) (full text, mbox, link).

Message #77 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>
To: Vagrant Cascadian <vagrant@debian.org>
Cc: Chris Lamb <lamby@debian.org>, 926242@bugs.debian.org, Mattia Rizzolo <mattia@debian.org>, debian-boot@lists.debian.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Sat, 25 May 2019 12:30:42 +0100
On Fri, May 24, 2019 at 06:56:35PM -0700, Vagrant Cascadian wrote:
> Colin Watson removed it back in 2011:
> 
>   https://salsa.debian.org/installer-team/debian-installer/commit/fa965c32ca8bfa2ff14886c6f0dca131532815c7
> 
> commit fa965c32ca8bfa2ff14886c6f0dca131532815c7
> Author: Colin Watson <cjwatson@debian.org>
> Date:   Mon Mar 14 18:08:25 2011 +0000
> 
>     Skip the option field in sources.list lines, if present.                                                   
> 
> diff --git a/build/util/gen-sources.list.udeb b/build/util/gen-sources.list.udeb                               
> index e86b4fa66..9f140100e 100755
> --- a/build/util/gen-sources.list.udeb
> +++ b/build/util/gen-sources.list.udeb
> @@ -36,6 +36,7 @@ get_mirrors() {
>                 [ -s $file ] || continue
>                 grep '^deb[[:space:]]' $file | \
>                    grep -v '^deb[[:space:]]\+cdrom:' | \                                                       
> +                  sed 's,^deb \[[^]]*\] ,deb ,' | \
>                    grep -v '\(security.debian.org\|volatile.debian.\(net\|org\)\)' | \                         
>                    grep '[[:space:]]main' | \
>                    awk '{print $1 " " $2}' | \
> 
> 
> Happen to remember what it breaks to have these present? If it was a
> workaround, perhaps it is no longer needed?

I'm not certain even after going through my IRC and email logs around
that time, but given the timing I suspect that it was a workaround for
multiarch systems where sources.list contained some lines with
[arch=...] options to limit them to only some architectures.  I don't
remember exactly how this broke the d-i build; I assume it must have
done at the time or I wouldn't have made that change.

It would be worth somebody trying out a d-i build on a system with this
kind of configuration to see if it still breaks, but otherwise I suspect
we can drop this.

-- 
Colin Watson                                       [cjwatson@debian.org]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Sun, 26 May 2019 10:09:02 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Sun, 26 May 2019 10:09:02 GMT) (full text, mbox, link).

Message #82 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Colin Watson" <cjwatson@debian.org>, "Vagrant Cascadian" <vagrant@debian.org>
Cc: 926242@bugs.debian.org, "Mattia Rizzolo" <mattia@debian.org>, debian-boot@lists.debian.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Sun, 26 May 2019 11:05:40 +0100
Dear Colin,

> >   https://salsa.debian.org/installer-team/debian-installer/commit/fa965c32ca8bfa2ff14886c6f0dca131532815c7
[…]
> I'm not certain even after going through my IRC and email logs around
> that time, but given the timing I suspect that it was a workaround for
> multiarch systems where sources.list contained some lines with
> [arch=...] options to limit them to only some architectures.

Thank for looking into this.

> It would be worth somebody trying out a d-i build on a system with this
> kind of configuration to see if it still breaks
  ^^^^^^^^^^^^^^^^^^^^^

Just to clarify, building d-i on a system with [arch=...] foo in its
/etc/apt/sources.list?


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Sun, 26 May 2019 13:27:03 GMT) (full text, mbox, link).

Acknowledgement sent to Colin Watson <cjwatson@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Sun, 26 May 2019 13:27:03 GMT) (full text, mbox, link).

Message #87 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>
To: Chris Lamb <lamby@debian.org>
Cc: Vagrant Cascadian <vagrant@debian.org>, 926242@bugs.debian.org, Mattia Rizzolo <mattia@debian.org>, debian-boot@lists.debian.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Sun, 26 May 2019 14:24:41 +0100
On Sun, May 26, 2019 at 11:05:40AM +0100, Chris Lamb wrote:
> Dear Colin,
> > It would be worth somebody trying out a d-i build on a system with this
> > kind of configuration to see if it still breaks
>   ^^^^^^^^^^^^^^^^^^^^^
> 
> Just to clarify, building d-i on a system with [arch=...] foo in its
> /etc/apt/sources.list?

This is all from dubious memory, but I suspect my setup at the time was
roughly an amd64 system with:

  deb [arch=amd64] <local partial mirror>
  deb <official mirror>

... on the grounds that my local partial mirror didn't have space for
both amd64 and i386.

-- 
Colin Watson                                       [cjwatson@debian.org]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 05 Jun 2019 14:21:02 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 05 Jun 2019 14:21:02 GMT) (full text, mbox, link).

Message #92 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Colin Watson" <cjwatson@debian.org>
Cc: "Vagrant Cascadian" <vagrant@debian.org>, 926242@bugs.debian.org, "Mattia Rizzolo" <mattia@debian.org>, debian-boot@lists.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 05 Jun 2019 15:19:40 +0100
[adding rb-general@lists.reproducible-builds.org to CC]

Hi Colin,

> This is all from dubious memory, but I suspect my setup at the time was
> roughly an amd64 system with:
> 
>   deb [arch=amd64] <local partial mirror>
>   deb <official mirror>
> 
> ... on the grounds that my local partial mirror didn't have space for
> both amd64 and i386.

Apologies for the delay in getting back to you all here.

I've got this working locally here although we require the following
change to the gen-sources.list.udeb script. Basically, we need print
three columns if we have "[options]", otherwise we just print two:

diff --git a/build/util/gen-sources.list.udeb b/build/util/gen-sources.list.udeb
index 539345a45..ac416266a 100755
--- a/build/util/gen-sources.list.udeb
+++ b/build/util/gen-sources.list.udeb
@@ -36,10 +36,9 @@ get_mirrors() {
 		[ -s $file ] || continue
 		grep '^deb[[:space:]]' $file | \
 		   grep -v '^deb[[:space:]]\+cdrom:' | \
-		   sed 's,^deb \[[^]]*\] ,deb ,' | \
 		   grep -v '\(security.debian.org\|volatile.debian.\(net\|org\)\)' | \
 		   grep '[[:space:]]main' | \
-		   awk '{print $1 " " $2}' | \
+		   awk '{ print (substr($2, 0, 1) == "[") ? $1 " " $2 " " $3 : $1 " " $2 }' | \
 		   sed 's,^deb file,deb copy,' | \
 		   sed 's,/* *$,,'
 	done

How does this look to you? Shell "golf" suggestions welcome,
naturally. (I tried a few sed variants but it got a bit messy.)


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 05 Jun 2019 16:57:03 GMT) (full text, mbox, link).

Acknowledgement sent to Cyril Brulebois <kibi@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 05 Jun 2019 16:57:03 GMT) (full text, mbox, link).

Message #97 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>
To: Chris Lamb <lamby@debian.org>
Cc: Colin Watson <cjwatson@debian.org>, Vagrant Cascadian <vagrant@debian.org>, 926242@bugs.debian.org, Mattia Rizzolo <mattia@debian.org>, debian-boot@lists.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 5 Jun 2019 18:53:06 +0200
[Message part 1 (text/plain, inline)]
Hi,

Chris Lamb <lamby@debian.org> (2019-06-05):
> Apologies for the delay in getting back to you all here.
> 
> I've got this working locally here although we require the following
> change to the gen-sources.list.udeb script. Basically, we need print
> three columns if we have "[options]", otherwise we just print two:
> 
> diff --git a/build/util/gen-sources.list.udeb b/build/util/gen-sources.list.udeb
> index 539345a45..ac416266a 100755
> --- a/build/util/gen-sources.list.udeb
> +++ b/build/util/gen-sources.list.udeb
> @@ -36,10 +36,9 @@ get_mirrors() {
>  		[ -s $file ] || continue
>  		grep '^deb[[:space:]]' $file | \
>  		   grep -v '^deb[[:space:]]\+cdrom:' | \
> -		   sed 's,^deb \[[^]]*\] ,deb ,' | \
>  		   grep -v '\(security.debian.org\|volatile.debian.\(net\|org\)\)' | \
>  		   grep '[[:space:]]main' | \
> -		   awk '{print $1 " " $2}' | \
> +		   awk '{ print (substr($2, 0, 1) == "[") ? $1 " " $2 " " $3 : $1 " " $2 }' | \
>  		   sed 's,^deb file,deb copy,' | \
>  		   sed 's,/* *$,,'
>  	done
> 
> How does this look to you? Shell "golf" suggestions welcome,
> naturally. (I tried a few sed variants but it got a bit messy.)

I'm a little wary with possibly merging this late in the release cycle,
so I'd rather get see that looked at after Buster is out. With extra
apologies since I've just broken the context of your patch by removing
the volatile references (in both debian-installer{,-netboot-images}.git)
since volatile disappeared with Squeeze and it's about time we dealt
with it… (https://www.debian.org/volatile/)


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 05 Jun 2019 20:33:03 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 05 Jun 2019 20:33:03 GMT) (full text, mbox, link).

Message #102 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Cyril Brulebois" <kibi@debian.org>
Cc: "Colin Watson" <cjwatson@debian.org>, "Vagrant Cascadian" <vagrant@debian.org>, 926242@bugs.debian.org, "Mattia Rizzolo" <mattia@debian.org>, debian-boot@lists.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 05 Jun 2019 21:29:16 +0100
Cyril Brulebois wrote:

> I'm a little wary with possibly merging this late in the release cycle,
> so I'd rather get see that looked at after Buster is out.

I naturally understand your hesitation but on the other hand I would
truly love to see this in buster. Indeed, we may actually have done
enough work to boast about having reproducible installer images for
the upcoming release (!) although without testing on our more-
comprehensive testing framework it is difficult to tell at the
moment...

Devil's advocate: this is surely unlikely to break the release of
buster itself? I mean, for the "final" official buster builds, that is?

> With extra apologies since I've just broken the context of your patch
> by removing the volatile references

No problem, updated patch is:

commit c27a34ba97e028c5b57a35470f6ecb82ad1d9ffb
Author: Chris Lamb <lamby@debian.org>
Date:   Wed Jun 5 21:20:34 2019 +0100

    Include arguments from sources.list(5) such as [check-valid-until=no], etc. (Re: #926242)
    
    We print three columns if we have "[options]", otherwise we just print
    two as before.

diff --git a/build/util/gen-sources.list.udeb b/build/util/gen-sources.list.udeb
index 7fa40ac5a..f8415fb05 100755
--- a/build/util/gen-sources.list.udeb
+++ b/build/util/gen-sources.list.udeb
@@ -36,10 +36,9 @@ get_mirrors() {
 		[ -s $file ] || continue
 		grep '^deb[[:space:]]' $file | \
 		   grep -v '^deb[[:space:]]\+cdrom:' | \
-		   sed 's,^deb \[[^]]*\] ,deb ,' | \
 		   grep -v 'security.debian.org' | \
 		   grep '[[:space:]]main' | \
-		   awk '{print $1 " " $2}' | \
+		   awk '{ print (substr($2, 0, 1) == "[") ? $1 " " $2 " " $3 : $1 " " $2 }' | \
 		   sed 's,^deb file,deb copy,' | \
 		   sed 's,/* *$,,'
 	done


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 05 Jun 2019 21:09:02 GMT) (full text, mbox, link).

Acknowledgement sent to Cyril Brulebois <kibi@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 05 Jun 2019 21:09:02 GMT) (full text, mbox, link).

Message #107 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>
To: Chris Lamb <lamby@debian.org>
Cc: Colin Watson <cjwatson@debian.org>, Vagrant Cascadian <vagrant@debian.org>, 926242@bugs.debian.org, Mattia Rizzolo <mattia@debian.org>, debian-boot@lists.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 5 Jun 2019 23:04:33 +0200
[Message part 1 (text/plain, inline)]
Chris Lamb <lamby@debian.org> (2019-06-05):
> I naturally understand your hesitation but on the other hand I would
> truly love to see this in buster. Indeed, we may actually have done
> enough work to boast about having reproducible installer images for
> the upcoming release (!) although without testing on our more-
> comprehensive testing framework it is difficult to tell at the
> moment...
> 
> Devil's advocate: this is surely unlikely to break the release of
> buster itself? I mean, for the "final" official buster builds, that is?

That's exactly the point: I don't think it's unlikely. And it can't be
tested until it reaches the buildds. At which point, seeing breakages
isn't exactly what we want when a release is about to happen.

As a middle ground, how does the following sound? First test this in
unstable with the first alpha for bullseye, and possible backport it
in a point release?


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 05 Jun 2019 21:27:03 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 05 Jun 2019 21:27:03 GMT) (full text, mbox, link).

Message #112 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Cyril Brulebois" <kibi@debian.org>
Cc: "Colin Watson" <cjwatson@debian.org>, "Vagrant Cascadian" <vagrant@debian.org>, 926242@bugs.debian.org, "Mattia Rizzolo" <mattia@debian.org>, debian-boot@lists.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 05 Jun 2019 22:23:55 +0100
Hi Cyril,

> > Devil's advocate: this is surely unlikely to break the release of
> > buster itself? I mean, for the "final" official buster builds, that is?
> 
> That's exactly the point: I don't think it's unlikely. […]

Could you elaborate on this bit? sources.list(5) options aren't /that/
common, in my experience. I ACK the lack of visibility until it hits
buildds, thanks for explaining that.

> As a middle ground, how does the following sound? First test this in
> unstable with the first alpha for bullseye, and possible backport it
> in a point release?

Ah, a point release might be the thing. Indeed, we might need some
other tiny changes to be reproducible that only become visible once we
can do the aforementioned extensive testing.

In the meantime, Mattia: I wonder if we could hack something specific
for src:debian-installer in addition to the network access exception?

For example, my reading of:

     683         if [ "$(MIRROR)x" != "x" ]; then \
     684                 echo "deb $(MIRROR) $(USE_UDEBS_FROM) $(UDEB_COMPONENTS)"; \
     685                 if [ "$(USE_UNRELEASED)" = 1 ]; then \
     686                         echo "deb $(MIRROR) unreleased $(UDEB_COMPONENTS)"; \
     687                 fi \
     688         else \
     689                 gen-sources.list.udeb "$(SYSTEM_SOURCES_LIST)" $(USE_UDEBS_FROM) $(UDEB_COMPONENTS) $(USE_PROPOSED_UPDATES); \
     690                 if [ "$(USE_UNRELEASED)" = 1 ]; then \
     691                         gen-sources.list.udeb "$(SYSTEM_SOURCES_LIST)" unreleased $(UDEB_COMPONENTS); \
     692                 fi \
     693         fi) > $@

… suggests that if we could export "MIRROR" pointing to (literally)
"[check-valid-until=no] http://deb.debian.org/debian" (!).


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 05 Jun 2019 22:45:02 GMT) (full text, mbox, link).

Acknowledgement sent to Philip Hands <phil@hands.com>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 05 Jun 2019 22:45:02 GMT) (full text, mbox, link).

Message #117 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Philip Hands <phil@hands.com>
To: Chris Lamb <lamby@debian.org>, Colin Watson <cjwatson@debian.org>
Cc: Vagrant Cascadian <vagrant@debian.org>, 926242@bugs.debian.org, Mattia Rizzolo <mattia@debian.org>, debian-boot@lists.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Thu, 06 Jun 2019 00:32:07 +0200
[Message part 1 (text/plain, inline)]
"Chris Lamb" <lamby@debian.org> writes:

> [adding rb-general@lists.reproducible-builds.org to CC]
>
> Hi Colin,
>
>> This is all from dubious memory, but I suspect my setup at the time was
>> roughly an amd64 system with:
>> 
>>   deb [arch=amd64] <local partial mirror>
>>   deb <official mirror>
>> 
>> ... on the grounds that my local partial mirror didn't have space for
>> both amd64 and i386.
>
> Apologies for the delay in getting back to you all here.
>
> I've got this working locally here although we require the following
> change to the gen-sources.list.udeb script. Basically, we need print
> three columns if we have "[options]", otherwise we just print two:
>
> diff --git a/build/util/gen-sources.list.udeb b/build/util/gen-sources.list.udeb
> index 539345a45..ac416266a 100755
> --- a/build/util/gen-sources.list.udeb
> +++ b/build/util/gen-sources.list.udeb
> @@ -36,10 +36,9 @@ get_mirrors() {
>  		[ -s $file ] || continue
>  		grep '^deb[[:space:]]' $file | \
>  		   grep -v '^deb[[:space:]]\+cdrom:' | \
> -		   sed 's,^deb \[[^]]*\] ,deb ,' | \
>  		   grep -v '\(security.debian.org\|volatile.debian.\(net\|org\)\)' | \
>  		   grep '[[:space:]]main' | \
> -		   awk '{print $1 " " $2}' | \
> +		   awk '{ print (substr($2, 0, 1) == "[") ? $1 " " $2 " " $3 : $1 " " $2 }' | \
>  		   sed 's,^deb file,deb copy,' | \
>  		   sed 's,/* *$,,'
>  	done
>
> How does this look to you? Shell "golf" suggestions welcome,
> naturally. (I tried a few sed variants but it got a bit messy.)

I failed to resist that, so I _think_ this sed command implements the
same effect as those greps/seds & awk:

  sed -ne '/^deb[[:space:]]\+cdrom/d;
           /\(security.debian.org\|volatile.debian.\(net\|org\)\)/d;
           /^deb[[:space:]]\+.*[[:space:]]\+main/{
             s,\(deb[[:space:]]\+\(\[[^]]*\][[:space:]]\+\|\)[^[:space:]]*\).*$,\1,;
             s,^deb file,deb copy,;
             s,/* *$,,;
             p
           }'

I only tested that with a few fairly simple test cases, so hopefully
you've got some nice test data.

BTW I note that in the original (and therefore in this too) that the
exclusion of cdrom: and the 'deb file' to 'deb copy' bits only work if
there's no [option] bit in the line -- was that an oversight?

Also, I'm asking myself: Why all the [[:space:]]\+ stuff if one then
seds for '^deb file' and '/* *$' at the end? -- I think one should
choose to do one or the other of those throughout.

As you say though, the sed for handling the optional bit is rather nasty
to read.  I'm now wondering if the whole thing might not be better done
in a single awk invocation...

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/    http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,    GERMANY

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 05 Jun 2019 22:48:02 GMT) (full text, mbox, link).

Acknowledgement sent to Philip Hands <phil@hands.com>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 05 Jun 2019 22:48:02 GMT) (full text, mbox, link).

Message #122 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Philip Hands <phil@hands.com>
To: Chris Lamb <lamby@debian.org>, Colin Watson <cjwatson@debian.org>
Cc: Vagrant Cascadian <vagrant@debian.org>, 926242@bugs.debian.org, Mattia Rizzolo <mattia@debian.org>, debian-boot@lists.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Thu, 06 Jun 2019 00:45:16 +0200
[Message part 1 (text/plain, inline)]
Philip Hands <phil@hands.com> writes:
...
> I failed to resist that, so I _think_ this sed command implements the
> same effect as those greps/seds & awk:
>
>   sed -ne '/^deb[[:space:]]\+cdrom/d;
>            /\(security.debian.org\|volatile.debian.\(net\|org\)\)/d;
>            /^deb[[:space:]]\+.*[[:space:]]\+main/{
>              s,\(deb[[:space:]]\+\(\[[^]]*\][[:space:]]\+\|\)[^[:space:]]*\).*$,\1,;
>              s,^deb file,deb copy,;
>              s,/* *$,,;
>              p
>            }'

I seem to have neglected to type the line I had intended saying "If
this change needs to wait for post-buster anyway, then we could cut down
the number of sed/grep invocations".

This was not meant as a suggestion that we should do this sort of thing
this late, particularly since the suggested patch may actually be broken
if it's supposed to be able to deal with, e.g.:

  deb [arch=amd64] file:///...

because that won't currently end up as:

  deb [arch=amd64] copy:///...

so I completely sympathise with Cyril's caution.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/    http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,    GERMANY

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 12 Jun 2019 10:21:07 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 12 Jun 2019 10:21:07 GMT) (full text, mbox, link).

Message #127 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Philip Hands" <phil@hands.com>, "Colin Watson" <cjwatson@debian.org>
Cc: "Vagrant Cascadian" <vagrant@debian.org>, 926242@bugs.debian.org, "Mattia Rizzolo" <mattia@debian.org>, debian-boot@lists.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 12 Jun 2019 11:19:14 +0100
Philip Hands wrote:

> BTW I note that in the original (and therefore in this too) that the
> exclusion of cdrom: and the 'deb file' to 'deb copy' bits only work if
> there's no [option] bit in the line -- was that an oversight?

I would guess so, yes.

In light of that (and whilst my shell is a little rusty) but how about
we just make this all more explicit instead of abusing sed/awk?

For example:

get_mirrors () {
	local file
	local line
	for file in $@; do
		while read line
		do
			if ! echo "$line" | grep -qs '^deb[[:space:]]'; then
				continue
			fi

			local options=
			local uri="$(echo "$line" | cut -d' ' -f2)"
			local dist="$(echo "$line" | cut -d' ' -f3)"
			local components="$(echo "$line" | cut -d' ' -f4-)"

			case "$uri" in
			\[*)
				options="$uri "
				uri="$(echo "$line" | cut -d' ' -f3)"
				dist="$(echo "$line" | cut -d' ' -f4)"
				components="$(echo "$line" | cut -d' ' -f5-)"
				;;
			esac

			case "$uri" in
			cdrom:*|*security.debian.org*|*volatile.debian.*)
				continue
				;;
			file:*)
				uri="$(echo "${uri}" | sed 's,^file,copy,')"
				;;
			esac

			echo "deb ${options}${uri} ${dist} ${components}"
		done < $file
	done	
}


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Mon, 08 Jul 2019 13:30:03 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Mon, 08 Jul 2019 13:30:03 GMT) (full text, mbox, link).

Message #132 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Chris Lamb" <lamby@debian.org>, "Philip Hands" <phil@hands.com>, "Colin Watson" <cjwatson@debian.org>
Cc: "Vagrant Cascadian" <vagrant@debian.org>, debian-boot@lists.debian.org, 926242@bugs.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Mon, 08 Jul 2019 10:27:02 -0300
Chris Lamb wrote:

> In light of that (and whilst my shell is a little rusty) but how about
> we just make this all more explicit instead of abusing sed/awk?
> 
> For example:

[…]

So, I heard a vague rumour that this "buster" thing was released? I
was thus wondering whether we could apply my patch from:

  https://bugs.debian.org/926242#127
  
:)


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Mon, 08 Jul 2019 13:42:03 GMT) (full text, mbox, link).

Acknowledgement sent to Holger Levsen <holger@layer-acht.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Mon, 08 Jul 2019 13:42:03 GMT) (full text, mbox, link).

Message #137 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>
To: Chris Lamb <lamby@debian.org>
Cc: Philip Hands <phil@hands.com>, Colin Watson <cjwatson@debian.org>, Vagrant Cascadian <vagrant@debian.org>, debian-boot@lists.debian.org, 926242@bugs.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Mon, 8 Jul 2019 13:39:30 +0000
[Message part 1 (text/plain, inline)]
On Mon, Jul 08, 2019 at 10:27:02AM -0300, Chris Lamb wrote:
> So, I heard a vague rumour that this "buster" thing was released? I
> was thus wondering whether we could apply my patch from:
>   https://bugs.debian.org/926242#127

https://bugs.debian.org/926242#117 makes me think this is not to be
applied against jenkins.debian.net.git? So, a.) a full (git) patch is nicer
than just some inline code in a bug report because b.) this also reveals 
where to apply against.


-- 
tschau,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Mon, 08 Jul 2019 14:51:05 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Mon, 08 Jul 2019 14:51:05 GMT) (full text, mbox, link).

Message #142 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: "Holger Levsen" <holger@layer-acht.org>
Cc: "Philip Hands" <phil@hands.com>, "Colin Watson" <cjwatson@debian.org>, "Vagrant Cascadian" <vagrant@debian.org>, debian-boot@lists.debian.org, 926242@bugs.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Mon, 08 Jul 2019 11:47:34 -0300
Hi Holger,

> > So, I heard a vague rumour that this "buster" thing was released? I
> > was thus wondering whether we could apply my patch from:
[…]
> https://bugs.debian.org/926242#117 makes me think this is not to be
> applied against jenkins.debian.net.git?

Pre-buster, perhaps? I don't quite see why we cannot and should not
fix it "upstream" in d-i instead? Indeed, fixing it on our Jenkins
instance would surely be rather ugly and essentially involve special-
casing, hardcoding a patch, etc. etc. Ew.

(If we should fix it in d-i, I'll create a MR or similar but I'll
save that until there is some form of ACK...)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Mon, 08 Jul 2019 14:51:06 GMT) (full text, mbox, link).

Acknowledgement sent to Holger Levsen <holger@layer-acht.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Mon, 08 Jul 2019 14:51:06 GMT) (full text, mbox, link).

Message #147 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>
To: Chris Lamb <lamby@debian.org>
Cc: Philip Hands <phil@hands.com>, Colin Watson <cjwatson@debian.org>, Vagrant Cascadian <vagrant@debian.org>, debian-boot@lists.debian.org, 926242@bugs.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Mon, 8 Jul 2019 14:49:59 +0000
[Message part 1 (text/plain, inline)]
On Mon, Jul 08, 2019 at 11:47:34AM -0300, Chris Lamb wrote:
> > > So, I heard a vague rumour that this "buster" thing was released? I
> > > was thus wondering whether we could apply my patch from:
> […]
> > https://bugs.debian.org/926242#117 makes me think this is not to be
> > applied against jenkins.debian.net.git?
> Pre-buster, perhaps? I don't quite see why we cannot and should not
> fix it "upstream" in d-i instead? 

yes, though #926242 is a bug against jenkins.d.o|n and not against d-i/.

> Indeed, fixing it on our Jenkins
> instance would surely be rather ugly and essentially involve special-
> casing, hardcoding a patch, etc. etc. Ew.

yeah

> (If we should fix it in d-i, I'll create a MR or similar but I'll
> save that until there is some form of ACK...)

*nods*


-- 
tschau,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Tue, 09 Jul 2019 05:45:03 GMT) (full text, mbox, link).

Acknowledgement sent to Cyril Brulebois <kibi@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Tue, 09 Jul 2019 05:45:03 GMT) (full text, mbox, link).

Message #152 received at 926242@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>
To: Chris Lamb <lamby@debian.org>
Cc: Philip Hands <phil@hands.com>, Colin Watson <cjwatson@debian.org>, Vagrant Cascadian <vagrant@debian.org>, debian-boot@lists.debian.org, 926242@bugs.debian.org, rb-general@lists.reproducible-builds.org
Subject: Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Tue, 9 Jul 2019 07:40:20 +0200
[Message part 1 (text/plain, inline)]
Chris Lamb <lamby@debian.org> (2019-07-08):
> Chris Lamb wrote:
> 
> > In light of that (and whilst my shell is a little rusty) but how about
> > we just make this all more explicit instead of abusing sed/awk?
> > 
> > For example:
> 
> […]
> 
> So, I heard a vague rumour that this "buster" thing was released? I
> was thus wondering whether we could apply my patch from:
> 
>   https://bugs.debian.org/926242#127
>   
> :)

My current plan is (1) breathing a little, (2) getting the needed
bugfixes into 10.1.


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Thu, 02 Jan 2020 00:00:02 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Thu, 02 Jan 2020 00:00:02 GMT) (full text, mbox, link).

Message #157 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: debian-boot@lists.debian.org, 926242@bugs.debian.org, rb-general <rb-general@lists.reproducible-builds.org>
Subject: Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 01 Jan 2020 23:57:04 -0000
[trimming CCs to just the mailing lists and #926242]

Hey all,

> > So, I heard a vague rumour that this "buster" thing was released? I
> > was thus wondering whether we could apply my patch from:
> > 
> >   https://bugs.debian.org/926242#127
> >   
> > :)
> 
> My current plan is (1) breathing a little, (2) getting the needed
> bugfixes into 10.1.

Whoops, I'm afraid I totally neglected to followup on this so I
apologise this got stalled. Anyway, anything I can do to help?


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Fri, 28 Feb 2020 21:57:03 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Fri, 28 Feb 2020 21:57:03 GMT) (full text, mbox, link).

Message #162 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: debian-boot@lists.debian.org, 926242@bugs.debian.org, rb-general <rb-general@lists.reproducible-builds.org>
Subject: Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Fri, 28 Feb 2020 13:52:52 -0800
Chris Lamb wrote:

> > > So, I heard a vague rumour that this "buster" thing was released? I
> > > was thus wondering whether we could apply my patch from:
> > > 
> > >   https://bugs.debian.org/926242#127
> > >   
> > > :)
> > 
> > My current plan is (1) breathing a little, (2) getting the needed
> > bugfixes into 10.1.
> 
> Whoops, I'm afraid I totally neglected to followup on this so I
> apologise this got stalled. Anyway, anything I can do to help?

I've made an initial step of taking my patch from:

  https://bugs.debian.org/926242#127

… and submitting it as a MR on salsa here:

  https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/13


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 19 Aug 2020 16:57:02 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 19 Aug 2020 16:57:02 GMT) (full text, mbox, link).

Message #167 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: rb-general <rb-general@lists.reproducible-builds.org>, debian-boot@lists.debian.org, 926242@bugs.debian.org
Subject: Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 19 Aug 2020 16:52:03 -0000
Chris Lamb wrote:

> > > My current plan is (1) breathing a little, (2) getting the needed
> > > bugfixes into 10.1.
> >
> > Whoops, I'm afraid I totally neglected to followup on this so I
> > apologise this got stalled. Anyway, anything I can do to help?
>
> I've made an initial step of taking my patch from:
>
>   https://bugs.debian.org/926242#127
>
> … and submitting it as a MR on salsa here:
>
>   https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/13

May I make a gentle request to get this MR merged? It's been open for
about 5 months now, only affects the build system and is only to
handle cases where we have the stranger [foo=bar] arguments in
sources.list(5) entries, which is unlikely to be the case for any
official builds.

As I write in my latest comment on the MR, it is not *strictly*
blocking testing whether d-i images are reproducible, but it is making
it really rather difficult -- I'm using awful 140-line local shell
script, rather using our far-superior testing framework, and we have
likely been accumulating regressions since last time I was seriously
working on this.


Regards,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 02 Sep 2020 23:03:22 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 02 Sep 2020 23:03:22 GMT) (full text, mbox, link).

Message #172 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: rb-general <rb-general@lists.reproducible-builds.org>, debian-boot@lists.debian.org, 926242@bugs.debian.org, "Cyril Brulebois" <kibi@debian.org>
Subject: Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 02 Sep 2020 22:58:04 -0000
Chris Lamb wrote:

> > I've made an initial step of taking my patch from:
> >
> >   https://bugs.debian.org/926242#127
> >
> > … and submitting it as a MR on salsa here:
> >
> >   https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/13
>
> May I make a gentle request to get this MR merged? It's been open for
> about 5 months now, only affects the build system and is only to
> handle cases where we have the stranger [foo=bar] arguments in
> sources.list(5) entries, which is unlikely to be the case for any
> official builds.

It has been a little disheartening to go through a large number of
minor revisions to this MR, only for it to then go unacknowledged for
almost half a year.

Please let me know if debian-boot has an alternative or preferred
method of performing reviews of this kind — I would, of course, be
more than willing to follow that if pointed in the right direction.

Without that, however, I remain at a loss at how I can effectively
contribute to this project.


Regards,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>:
Bug#926242; Package jenkins.debian.org. (Wed, 06 Jan 2021 11:21:02 GMT) (full text, mbox, link).

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Jenkins Team <qa-jenkins-dev@lists.alioth.debian.org>. (Wed, 06 Jan 2021 11:21:02 GMT) (full text, mbox, link).

Message #177 received at 926242@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>
To: 926242@bugs.debian.org, reproducible-bugs@lists.alioth.debian.org
Subject: Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images
Date: Wed, 06 Jan 2021 11:16:23 +0000
Chris Lamb wrote:

> > I've made an initial step of taking my patch from:
> >
> >   https://bugs.debian.org/926242#127
> >
> > … and submitting it as a MR on salsa here:
> >
> >   https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/13

Alas, this didn't seem to land in the Bullseye Alpha 3 release of d-i.
Are we in time to merge it into the next alpha/beta, perhaps? :)


Best wishes,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Send a report that this bug log contains spam.

Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Wed May 17 09:26:40 2023; Machine Name: buxtehude

Debian Bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

Copyright © 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson, 2005-2017 Don Armstrong, and many other contributors.