Debian Bug report logs - #862073
ftp.debian.org: Please POST .buildinfo files to buildinfo.debian.net

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: submit@bugs.debian.org

Subject: ftpmaster.debian.org: Please POST .buildinfo files to buildinfo.debian.net

Date: Mon, 08 May 2017 09:32:15 +0100

[Message part 1 (text/plain, inline)]

Package: ftpmaster.debian.org
Severity: wishlist
Tags: patch

Hi,

Attached is a patch submit .buildinfo files to buildinfo.debian.net,
our experimental system for centrally storing .buildinfo files for
analysis, retrieval, etc.  We almost have 2,000,000 files there.

This patch supplements the existing filesystem archiving and simply
performs a POST on the .buildinfo file itself.

As a deployment note, this will actually mean that — right now — most
.buildinfo files will be rejected by buildinfo.debian.net as it only
accepts signed .buildinfo files. However, we intend to fix that
separately via #862059 ("sbuild: please sign buildinfo files").

Also note that this patch enables this for the main archive only.
Please clarify whether I should enable this for the security archive
too; I would not want it to leak the fact we have rebuilt a package
there if itwere embargoed, etc.


  commit a85df018d210c054e7ae0b5a6fe037a537b62e7a
  Author: Chris Lamb <lamby@debian.org>
  Date:   Mon May 8 01:06:03 2017 +0200
  
      Upload buildinfo files to buildinfo.debian.net.
      
      Signed-off-by: Chris Lamb <lamby@debian.org>
  
   config/debian/dak.conf |  5 +++++
   dak/process_upload.py  | 52 ++++++++++++++++++++++++++++++++++++++++++++++++--
   2 files changed, 55 insertions(+), 2 deletions(-)


Alternatively you can merge from the:

  upload-buildinfo-files-to-buildinfo-debian-net

branch on <https://github.com/lamby/dak.git>.


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

[0001-Upload-buildinfo-files-to-buildinfo.debian.net.patch (text/x-diff, attachment)]

Message #10 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 862073@bugs.debian.org

Cc: reproducible-bugs@lists.alioth.debian.org

Subject: Re: ftpmaster.debian.org: Please POST .buildinfo files to buildinfo.debian.net

Date: Mon, 08 May 2017 09:37:33 +0100

reassign 862073 ftp.debian.org
user reproducible-builds@lists.alioth.debian.org
usertag 862073 + toolchain
thanks

Chris Lamb wrote:

> Attached is a patch submit .buildinfo files to buildinfo.debian.net,
> our experimental system for centrally storing .buildinfo files for
> analysis, retrieval, etc.  We almost have 2,000,000 files there.

Re-assigning to the correct package & setting usertags; apologies for
the noise.


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #21 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: 862073@bugs.debian.org, 862538@bugs.debian.org

Subject: Re: Processed: cloning

Date: Sun, 14 May 2017 12:01:42 +0000

[Message part 1 (text/plain, inline)]

Hi,

I've cloned #862073 as bug #862538 because of the issue of not sending embargoed
builds to buildinfo.debian.net before they can be made public…

That said, it would be really really cool to have .buildinfo files for security
updates for stretch during stretch's lifetime already. We'd be the first
distribution to have reproducible security updates! (for those 93% of the
packages which are reproducibly already…)


-- 
cheers,
	Holger

[signature.asc (application/pgp-signature, inline)]

Message #24 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Ansgar Burchardt <ansgar@debian.org>

To: Chris Lamb <lamby@debian.org>

Cc: 862073@bugs.debian.org

Subject: Re: Bug#862073: ftpmaster.debian.org: Please POST .buildinfo files to buildinfo.debian.net

Date: Mon, 15 May 2017 21:28:02 +0200

Hi,

Chris Lamb writes:
> Attached is a patch submit .buildinfo files to buildinfo.debian.net,
> our experimental system for centrally storing .buildinfo files for
> analysis, retrieval, etc.  We almost have 2,000,000 files there.
>
> This patch supplements the existing filesystem archiving and simply
> performs a POST on the .buildinfo file itself.

I don't think dak should push things to external services while
processing uploads: the code runs as the privileged user (and ideally
doesn't talk to the external world) and we still need a second point
where .buildinfo files are pushed (in case the PUT fails for any
reason).

So we could implement only the second point and push .buildinfo files
asynchronous and as an unprivileged user.

Ansgar

Message #29 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Ansgar Burchardt <ansgar@debian.org>

Cc: 862073@bugs.debian.org

Subject: Re: Bug#862073: ftpmaster.debian.org: Please POST .buildinfo files to buildinfo.debian.net

Date: Tue, 16 May 2017 16:38:42 +0100

Hi Ansgar,

> push .buildinfo files asynchronous and as an unprivileged user.

How about this alternative: as we are already archiving them on
ftp-master, how about a cronjob that would go through that archive,
uploading those that have not been (successfully) uploaded yet?

That would solve both of the problems of running as a privileged user and
be idempotent to boot.

(As a minor bonus, it would also ensure that the *existing* set of
archived .buildinfo files would be sent to external services too...)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #34 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: 862073@bugs.debian.org

Subject: Re: Bug#862073: ftpmaster.debian.org: Please POST .buildinfo files to buildinfo.debian.net

Date: Tue, 16 May 2017 15:50:05 +0000

[Message part 1 (text/plain, inline)]

On Tue, May 16, 2017 at 04:38:42PM +0100, Chris Lamb wrote:
> That would solve both of the problems of running as a privileged user and
> be idempotent to boot.
> 
> (As a minor bonus, it would also ensure that the *existing* set of
> archived .buildinfo files would be sent to external services too...)

and fourth, it can also be used for security.debian.org to only upload
public .buildinfo files \o/ 


-- 
cheers,
	Holger

[signature.asc (application/pgp-signature, inline)]

Message #39 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@debian.org>

To: "David A. Wheeler" <dwheeler@dwheeler.com>, rb-general <rb-general@lists.reproducible-builds.org>

Cc: 862073@bugs.debian.org

Subject: Uploading buildinfo files to buildinfo.debian.net

Date: Thu, 25 Oct 2018 12:56:20 -0700

[Message part 1 (text/plain, inline)]

On 2018-10-23, Vagrant Cascadian wrote:
> Main blocker that comes to mind is publishing of buildinfo files
> submitted to the archive in a way that people can actually download them
> who are not Debian developers:
>
>   https://bugs.debian.org/763822
>   https://bugs.debian.org/862073

Thanks for bringing this up, it was the final straw inciting me to
action on this particular point!

I started the process of uploading all the .buildinfo files available on
ftp.debian.org to buildinfo.debian.net.

Then I hope to set up a cron job to do uploads at least daily with a
little better error-handling.... Would be more ideal to have something
more formally integrated into infrastructure, but maybe I can work out a
proof-of-concept implementation as a basis for something that can be
integrated.

Still a lot of work to do to make those buildinfo files useable, but at
least it puts the data somewhere where anyone can work on solving the
remaining issues of validating what's actually in the archive.


Since I've got some numbers handy...

All the individual .buildinfo files for 2017 came to:

  4574MB    2017

Compressed into monthly tarballs of .buildinfo files:

  594MB

So that's about an 87% compression rate! Individual .buildinfo files
compressed to around 50%.

For the first 3 months or so of 2017, many of the .buildinfo files
weren't signed, so are probably much smaller. Reasonably
buildinfo.debian.net won't accept unsigned buildinfo files.

The Debian archive accepts some signatures types (elliptic curve) that
buildinfo.debian.net doesn't yet:

  https://github.com/lamby/buildinfo.debian.net/issues/51

So presumably any signed with such keys won't yet get accepted either.


For 2018 the numbers were similar, though a bit larger(more consistant
use of tools that produce .buildinfo files?). Sorry I don't have the
exact details on hand at the moment.


So I'm guessing the overall storage burden would be around 5GB per year
uncompressed and growing slightly as the archive grows and architectures
are added (minus the occasional removed architecture)...


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #44 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: Vagrant Cascadian <vagrant@debian.org>, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Cc: 862073@bugs.debian.org

Subject: Re: [rb-general] Uploading buildinfo files to buildinfo.debian.net

Date: Fri, 15 Feb 2019 21:11:54 +0000

[Message part 1 (text/plain, inline)]

Hi Vagrant,

I've just been re-reading this old and joyful thread... :)

On Thu, Oct 25, 2018 at 12:56:20PM -0700, Vagrant Cascadian wrote:
> I started the process of uploading all the .buildinfo files available on
> ftp.debian.org to buildinfo.debian.net.
> 
> Then I hope to set up a cron job to do uploads at least daily with a
> little better error-handling.... Would be more ideal to have something
> more formally integrated into infrastructure, but maybe I can work out a
> proof-of-concept implementation as a basis for something that can be
> integrated.

did you manage to setup this cron job?


-- 
tschau,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

[signature.asc (application/pgp-signature, inline)]

Message #49 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@debian.org>

To: Holger Levsen <holger@layer-acht.org>, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Cc: 862073@bugs.debian.org

Subject: Re: [rb-general] Uploading buildinfo files to buildinfo.debian.net

Date: Fri, 15 Feb 2019 13:51:40 -0800

[Message part 1 (text/plain, inline)]

On 2019-02-15, Holger Levsen wrote:
> I've just been re-reading this old and joyful thread... :)

I come back to it now and then myself... :)


> On Thu, Oct 25, 2018 at 12:56:20PM -0700, Vagrant Cascadian wrote:
>> I started the process of uploading all the .buildinfo files available on
>> ftp.debian.org to buildinfo.debian.net.
>> 
>> Then I hope to set up a cron job to do uploads at least daily with a
>> little better error-handling.... Would be more ideal to have something
>> more formally integrated into infrastructure, but maybe I can work out a
>> proof-of-concept implementation as a basis for something that can be
>> integrated.
>
> did you manage to setup this cron job?

I had thought I left more detail about the current status, but
apparently not! Thanks for the nudge.

I have a cron job running on coccia.debian.org since November, as my own
"vagrant" user:

  coccia.debian.org:~vagrant/rb-buildinfos/upload-buildinfos

Logs for various upload passes in are the same directory, which should
probably be migrated to sqlite or some real database. The script is
checked into it's own git repository, but not properly pushed
anywhere.

The cron job runs several times per day, checking the queues for
buildinfos uploaded both the current day yesterday to make sure we don't
miss a .buildinfo file uploaded in the middle of a processing run. If
coccia were down for longer than 24 hours, it might need to manually be
run to check for missing ones.

The vast majority of buildinfo files uploaded to the archive should be
present in buildinfo.debian.net since November 2018. I also "manually"
uploaded all the available buildinfo files from 2017-2018 (most of the
very small number from 2016 failed for one reason or another).

There are a small number of buildinfo uploads that buildinfo.debian.net
rejects for some reason probably related to ed25519 signing keys:

  https://github.com/lamby/buildinfo.debian.net/issues/51

There are a few individual developers uploading unsigned .buildinfo
files, as well as a few buildds for non-release architectures
(e.g. hurd-i386, kfreebsd-*). To hadle those, I actually had a
legitimate use for the technique described in:

  https://xkcd.com/1181/

Which basically means I don't even bother attempting to upload unsigned
buildinfo files.


So, it's working, but we probably would need a little more work on it to
integrate into debian's infrastructure.


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #52 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: Vagrant Cascadian <vagrant@debian.org>, 862073@bugs.debian.org

Cc: General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Subject: Re: Bug#862073: [rb-general] Uploading buildinfo files to buildinfo.debian.net

Date: Sat, 16 Feb 2019 10:49:08 +0100

[Message part 1 (text/plain, inline)]

On Fri, Feb 15, 2019 at 01:51:40PM -0800, Vagrant Cascadian wrote:
> There are a few individual developers uploading unsigned .buildinfo
> files, as well as a few buildds for non-release architectures
> (e.g. hurd-i386, kfreebsd-*). To hadle those, I actually had a
> legitimate use for the technique described in:
> 
>   https://xkcd.com/1181/

Do you think you could provide more info about the kbsd and hurd
buildinfo that are unsigned?

James, who manages the current kbsd buildd, reads ftp.d.o bugs so should
be receiving this email (and I can otherwise nudge him), and I could
poke youpi for the hurd buildds.  But I would expect both of them to be
running unstable, and so have up-to-date software.

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Message #57 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@debian.org>

To: Mattia Rizzolo <mattia@debian.org>, 862073@bugs.debian.org

Cc: General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Subject: Re: [rb-general] Bug#862073: Uploading buildinfo files to buildinfo.debian.net

Date: Sat, 16 Feb 2019 08:48:27 -0800

[Message part 1 (text/plain, inline)]

On 2019-02-16, Mattia Rizzolo wrote:
> On Fri, Feb 15, 2019 at 01:51:40PM -0800, Vagrant Cascadian wrote:
>> There are a few individual developers uploading unsigned .buildinfo
>> files, as well as a few buildds for non-release architectures
>> (e.g. hurd-i386, kfreebsd-*). To hadle those, I actually had a
>> legitimate use for the technique described in:
>> 
>>   https://xkcd.com/1181/
>
> Do you think you could provide more info about the kbsd and hurd
> buildinfo that are unsigned?

Looks like kfreebsd were fixed at some point.

I'm not sure what more information is needed ... the .buildinfo files
available on coccia are unsigned. The only current ones still failing
are hurd-i386 or the infrequent developer uploads, for example:

NOSIGNATURE: ['/srv/ftp-master.debian.org/buildinfo/2019/02/16/gcc-9_9-20190215-1_hurd-i386.buildinfo',
'/srv/ftp-master.debian.org/buildinfo/2019/02/16/acl_2.2.52-5_hurd-i386.buildinfo',
'/srv/ftp-master.debian.org/buildinfo/2019/02/16/attr_2.4.47-4_hurd-i386.buildinfo',
'/srv/ftp-master.debian.org/buildinfo/2019/02/16/usepackage_1.13-4_hurd-i386.buildinfo',
'/srv/ftp-master.debian.org/buildinfo/2019/02/16/libocas_0.97+dfsg-5_hurd-i386.buildinfo',
'/srv/ftp-master.debian.org/buildinfo/2019/02/16/lxqt-metapackages_28_hurd-i386.buildinfo',
'/srv/ftp-master.debian.org/buildinfo/2019/02/16/ocaml-mm_0.4.0-1_hurd-i386.buildinfo',
'/srv/ftp-master.debian.org/buildinfo/2019/02/16/postfix_3.3.2-2_amd64.buildinfo']

I've privately emailed some of the individual developer uploads when
I've noticed, but haven't done a systematic look. Maybe it's time to do that.


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #60 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: Vagrant Cascadian <vagrant@debian.org>, 862073@bugs.debian.org

Cc: General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>, sthibault@debian.org

Subject: Re: Bug#862073: Uploading buildinfo files to buildinfo.debian.net

Date: Sat, 16 Feb 2019 19:02:05 +0100

[Message part 1 (text/plain, inline)]

On Sat, Feb 16, 2019 at 08:48:27AM -0800, Vagrant Cascadian wrote:
> On 2019-02-16, Mattia Rizzolo wrote:
> > Do you think you could provide more info about the kbsd and hurd
> > buildinfo that are unsigned?
> 
> Looks like kfreebsd were fixed at some point.
> 
> I'm not sure what more information is needed ... the .buildinfo files
> available on coccia are unsigned. The only current ones still failing
> are hurd-i386 or the infrequent developer uploads, for example:

ACK, I've asked youpi to have a look at his signing script (remember
that kbsd and hurd uploads are all manually signed by a DD).
Hopefully this will greatly reduce the number of unsigned buildinfo :)

> I've privately emailed some of the individual developer uploads when
> I've noticed, but haven't done a systematic look. Maybe it's time to do that.

Maybe after youpi confirms he improved his script.

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Message #67 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Samuel Thibault <sthibault@debian.org>

To: Mattia Rizzolo <mattia@debian.org>, 862073@bugs.debian.org

Cc: Vagrant Cascadian <vagrant@debian.org>, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>, sbuild@packages.debian.org

Subject: Re: Bug#862073: Uploading buildinfo files to buildinfo.debian.net

Date: Thu, 7 Mar 2019 23:49:51 +0100

Control: clone -1 -2
Control: reassign -2 sbuild
Control: retitle -2 Should also send the buildinfo in the build mail

Hello,

Mattia Rizzolo, le sam. 16 févr. 2019 19:02:05 +0100, a ecrit:
> On Sat, Feb 16, 2019 at 08:48:27AM -0800, Vagrant Cascadian wrote:
> > On 2019-02-16, Mattia Rizzolo wrote:
> > > Do you think you could provide more info about the kbsd and hurd
> > > buildinfo that are unsigned?
> > 
> > Looks like kfreebsd were fixed at some point.
> > 
> > I'm not sure what more information is needed ... the .buildinfo files
> > available on coccia are unsigned. The only current ones still failing
> > are hurd-i386 or the infrequent developer uploads, for example:
> 
> ACK, I've asked youpi to have a look at his signing script (remember
> that kbsd and hurd uploads are all manually signed by a DD).
> Hopefully this will greatly reduce the number of unsigned buildinfo :)
> 
> > I've privately emailed some of the individual developer uploads when
> > I've noticed, but haven't done a systematic look. Maybe it's time to do that.
> 
> Maybe after youpi confirms he improved his script.

I had a look, and as mentioned on IRC the .build part is not in the mail
sent by sbuild. kbsd builds are signed by hand with debsign so that
doesn't matter, but I use a script in my mailer to sign easily progressively,
so I need the .buildinfo in the mail send by sbuild.

Samuel

Message #74 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Cc: 862073@bugs.debian.org

Subject: Re: [rb-general] Uploading buildinfo files to buildinfo.debian.net

Date: Mon, 29 Apr 2019 16:53:16 +0000

[Message part 1 (text/plain, inline)]

Hi,

On Fri, Feb 15, 2019 at 01:51:40PM -0800, Vagrant Cascadian wrote:
> On 2019-02-15, Holger Levsen wrote:
> > I've just been re-reading this old and joyful thread... :)
> I come back to it now and then myself... :)

and here we go again...

I'll also kind of top-post now, because I want to emphasize one
important bit of information:

we now have two similar implementations of a buildinfo server for Debian
.buildinfo files:

- https://buildinfo.debian.net - this service exists since Oct. 2016
- https://buildinfos.debian.net - this service exist since March 2019

I've set up the latter because the former doesnt provide an easy means
to only get .buildinfo files for .debs from ftp.debian.org and because
it was easy to set it up. (The code lives in jenkins.debian.net.git)

I'm not really happy about the duplication of services and the very
similar names, but I couldnt come up with a better name and this is all
prototyping land anyway, so - for now - I just went ahead.

I'm also not sure how to proceed with those two services in future. For
now I think it's ok to have both and let us experiment with both.


> > On Thu, Oct 25, 2018 at 12:56:20PM -0700, Vagrant Cascadian wrote:
> >> I started the process of uploading all the .buildinfo files available on
> >> ftp.debian.org to buildinfo.debian.net.
> >> 
> >> Then I hope to set up a cron job to do uploads at least daily with a
> >> little better error-handling.... Would be more ideal to have something
> >> more formally integrated into infrastructure, but maybe I can work out a
> >> proof-of-concept implementation as a basis for something that can be
> >> integrated.
> >
> > did you manage to setup this cron job?
> 
> I had thought I left more detail about the current status, but
> apparently not! Thanks for the nudge.
> 
> I have a cron job running on coccia.debian.org since November, as my own
> "vagrant" user:

I have a similar cronjob on coccia...

>   coccia.debian.org:~vagrant/rb-buildinfos/upload-buildinfos
> 
> Logs for various upload passes in are the same directory, which should
> probably be migrated to sqlite or some real database. The script is
> checked into it's own git repository, but not properly pushed
> anywhere.
> 
> The cron job runs several times per day, checking the queues for
> buildinfos uploaded both the current day yesterday to make sure we don't
> miss a .buildinfo file uploaded in the middle of a processing run. If
> coccia were down for longer than 24 hours, it might need to manually be
> run to check for missing ones.
> 
> The vast majority of buildinfo files uploaded to the archive should be
> present in buildinfo.debian.net since November 2018. I also "manually"
> uploaded all the available buildinfo files from 2017-2018 (most of the
> very small number from 2016 failed for one reason or another).

buildinfos.debian.net has all .buildinfo files since December 2016.

> There are a small number of buildinfo uploads that buildinfo.debian.net
> rejects for some reason probably related to ed25519 signing keys:
>   https://github.com/lamby/buildinfo.debian.net/issues/51

see above :) (=buildinfos.d.n has those.)

> There are a few individual developers uploading unsigned .buildinfo
> files, as well as a few buildds for non-release architectures
> (e.g. hurd-i386, kfreebsd-*). To hadle those, I actually had a
> legitimate use for the technique described in:
>   https://xkcd.com/1181/
> Which basically means I don't even bother attempting to upload unsigned
> buildinfo files.

see above :) (=buildinfos.d.n has those.)
 
> So, it's working, but we probably would need a little more work on it to
> integrate into debian's infrastructure.

this is also true for buildinfos.d.n.


-- 
tschau,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

In Europe there are people prosecuted by courts because they saved other people
from drowning in the  Mediterranean Sea.  That is almost as absurd  as if there
were people being prosecuted because they save humans from drowning in the sea.

[signature.asc (application/pgp-signature, inline)]

Message #79 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Cc: 862073@bugs.debian.org

Subject: Re: Bug#862073: [rb-general] Uploading buildinfo files to buildinfo.debian.net

Date: Mon, 29 Apr 2019 17:03:24 +0000

[Message part 1 (text/plain, inline)]

On Mon, Apr 29, 2019 at 04:53:16PM +0000, Holger Levsen wrote:
> - https://buildinfos.debian.net - this service exist since March 2019

to expand on this:

provides two views:

https://buildinfos.debian.net/ftp-master.debian.org/buildinfo/ - this
has the same contents as /srv/ftp-master.debian.org/buildinfo/ on
(ftp-master|coccia).debian.org.

https://buildinfos.debian.net/buildinfo-pool/ - this is a pool view on
the same data. (implemented using links to the former.)


-- 
tschau,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

[signature.asc (application/pgp-signature, inline)]

Message #84 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Holger Levsen <holger@layer-acht.org>, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Cc: 862073@bugs.debian.org

Subject: Re: [rb-general] Uploading buildinfo files to buildinfo.debian.net

Date: Mon, 29 Apr 2019 10:39:30 -0700

[Message part 1 (text/plain, inline)]

On 2019-04-29, Holger Levsen wrote:
> On Fri, Feb 15, 2019 at 01:51:40PM -0800, Vagrant Cascadian wrote:
>> On 2019-02-15, Holger Levsen wrote:

> we now have two similar implementations of a buildinfo server for Debian
> .buildinfo files:
>
> - https://buildinfo.debian.net - this service exists since Oct. 2016
> - https://buildinfos.debian.net - this service exist since March 2019

I really like that it provides a view in a "pool" style, e.g.:

  https://buildinfos.debian.net/buildinfo-pool/u/u-boot/


I almost wonder if we shouldn't try to coordinate archiving this data
with:

  https://www.softwareheritage.org/

It might be a slight stretch of their mission to call .buildinfo files
"source code" ... but I wouldn't mind making the case that .buildinfo
files should be considered source code.


>> The vast majority of buildinfo files uploaded to the archive should be
>> present in buildinfo.debian.net since November 2018. I also "manually"
>> uploaded all the available buildinfo files from 2017-2018 (most of the
>> very small number from 2016 failed for one reason or another).
>
> buildinfos.debian.net has all .buildinfo files since December 2016.

Very cool! It is definitely a much simpler approach and catches many
corner cases (unsigned, signatures, etc.) that my method doesn't!


It seems to be missing the .buildinfo.N, which in some cases are the
actual .buildinfo files built by the buildd's and the corresponding .deb
files shipped in the archive.

The .buildinfo without a numbered increment is frequently provided by
developers who follow best practices and do source-only uploads that
include a signed .buildinfo file. I'll take a look at the code and see
if I can't propose a simple fix.

The presence of multiple .buildinfo* files does make it harder to know
which .buildinfo to use to reproduce a build from the archive if they
differ, unfortunately.


>> There are a few individual developers uploading unsigned .buildinfo
>> files, as well as a few buildds for non-release architectures
>> (e.g. hurd-i386, kfreebsd-*). To hadle those, I actually had a
>> legitimate use for the technique described in:
>>   https://xkcd.com/1181/
>> Which basically means I don't even bother attempting to upload unsigned
>> buildinfo files.
>
> see above :) (=buildinfos.d.n has those.)

Unsigned .buildinfo files are of limited usefulness, if we're really
trying to establish a chain of verification... though perhaps it's still
better than no .buildinfo at all, since the archive verifies the
.changes file before including it... though obviously a compromised
archive could inject malicious unsigned .buildinfo files more easily and
requires some trust needed in specific parties.


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #89 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: Holger Levsen <holger@layer-acht.org>, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Cc: 862073@bugs.debian.org

Subject: Re: [rb-general] Uploading buildinfo files to buildinfo.debian.net

Date: Mon, 29 Apr 2019 10:43:11 -0700

[Message part 1 (text/plain, inline)]

On 2019-04-29, Vagrant Cascadian wrote:
> It seems to be missing the .buildinfo.N, which in some cases are the
> actual .buildinfo files built by the buildd's and the corresponding .deb
> files shipped in the archive.
>
> The .buildinfo without a numbered increment is frequently provided by
> developers who follow best practices and do source-only uploads that
> include a signed .buildinfo file. I'll take a look at the code and see
> if I can't propose a simple fix.

It seems like you did in fact catch these, and named them as
ARCH-source.buildinfo. Nice!

live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #94 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: rb-general@lists.reproducible-builds.org

Cc: 862073@bugs.debian.org

Subject: Re: [rb-general] Uploading buildinfo files to buildinfo.debian.net

Date: Thu, 2 May 2019 15:27:56 +0000

[Message part 1 (text/plain, inline)]

Hi,

please note that https://buildinfos.debian.net/buildinfo-pool/ is
currently being recreated and thus not have all data again yet...

(I've added support for 3 (and more) .buildinfo files with the same
name and noticed that the numbered links were created in the wrong
order, thus the recreation.)

On Mon, Apr 29, 2019 at 10:39:30AM -0700, Vagrant Cascadian wrote:
> I really like that it provides a view in a "pool" style, e.g.:
>   https://buildinfos.debian.net/buildinfo-pool/u/u-boot/

thanks. (and me too)

> I almost wonder if we shouldn't try to coordinate archiving this data
> with:
>   https://www.softwareheritage.org/

thats an interesting idea!

> It might be a slight stretch of their mission to call .buildinfo files
> "source code" ... but I wouldn't mind making the case that .buildinfo
> files should be considered source code.

indeed.

> > buildinfos.debian.net has all .buildinfo files since December 2016.
> Very cool! It is definitely a much simpler approach and catches many
> corner cases (unsigned, signatures, etc.) that my method doesn't!

yup. 

> It seems to be missing the .buildinfo.N, which in some cases are the
> actual .buildinfo files built by the buildd's and the corresponding .deb
> files shipped in the archive.

it has them. see eg dpkg (once the pool structure is back).

> The .buildinfo without a numbered increment is frequently provided by
> developers who follow best practices and do source-only uploads that
> include a signed .buildinfo file. I'll take a look at the code and see
> if I can't propose a simple fix.

please do.

> The presence of multiple .buildinfo* files does make it harder to know
> which .buildinfo to use to reproduce a build from the archive if they
> differ, unfortunately.

the one with the correct hash for the .deb

> Unsigned .buildinfo files are of limited usefulness, if we're really
> trying to establish a chain of verification... though perhaps it's still
> better than no .buildinfo at all, since the archive verifies the
> .changes file before including it... though obviously a compromised
> archive could inject malicious unsigned .buildinfo files more easily and
> requires some trust needed in specific parties.

yes. unsigned .buildinfo files should not exist. for this to happen, as
a first step, I plan to record this state in the db (in a new table...)


-- 
tschau,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

Our civilization is being sacrificed for the opportunity of a very small number
of people to continue making enormous amounts of money...  It is the sufferings
of the many  which pay  for the luxuries  of the few...  You say  you love your
children  above all else,  and yet  you are stealing  their future  in front of 
their very eyes...

[signature.asc (application/pgp-signature, inline)]

Message #99 received at 862073@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: rb-general@lists.reproducible-builds.org

Cc: 862073@bugs.debian.org

Subject: Re: [rb-general] Uploading buildinfo files to buildinfo.debian.net

Date: Thu, 2 May 2019 15:37:23 +0000

[Message part 1 (text/plain, inline)]

On Mon, Apr 29, 2019 at 10:43:11AM -0700, Vagrant Cascadian wrote:
> On 2019-04-29, Vagrant Cascadian wrote:
> > It seems to be missing the .buildinfo.N, which in some cases are the
> > actual .buildinfo files built by the buildd's and the corresponding .deb
> > files shipped in the archive.
> >
> > The .buildinfo without a numbered increment is frequently provided by
> > developers who follow best practices and do source-only uploads that
> > include a signed .buildinfo file. I'll take a look at the code and see
> > if I can't propose a simple fix.
> It seems like you did in fact catch these, and named them as
> ARCH-source.buildinfo. Nice!

in the pool structure the filenames are changed so that the original
filename $package_$version_$arch.buildinfo is replaced with one where
the $arch part is replaced like this:

ARCHITECTURE=$(grep ^Architecture: $FILE | cut -d ' ' -f2-|sed 's# #-#g')

or in plain English: with the Architecture: field from inside the file,
where the architectures are concated with hyphons, so that

Architecture: all source amd64

results in a $package_$version_all-source-amd64.buildinfo file.

And then strangely there are a few hundred cases of identically named
$package_$version_$arch.buildinfo files (out of allmost a million) in
the yyyy/mm/dd structure, these get a .0 suffix. And then there are 4
cases with 3 identically named files, which get a .1 suffix.
(This needs to be investigated why this happens...)


-- 
tschau,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

we'll all die. make a difference while you can. disobey. smile.

[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.