Debian Bug report logs - #846164
dpkg-dev: Have dpkg-genchanges support a source+buildinfo-only upload after a binary build

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: dpkg-dev: Have dpkg-genchanges support a source+buildinfo-only upload

Date: Mon, 28 Nov 2016 22:00:32 +0100

Package: dpkg-dev
Version: 1.18.15
Severity: wishlist

Dear Maintainer,

Currently one can do

$ dpkg-buildpackage --changes-option=-S

to do a binary build locally, but only upload the source package so that buildds build on all arches. Note that this command *does* generate a buildinfo file, but does not include it in the changes file.

It would be good to be able to do something like

$ dpkg-buildpackage --changes-option=-SB

to do a binary build locally, but only upload the source package *and* the buildinfo file.

This was one idea we had near the beginning of the R-B project. The idea being developers could do this, then the buildds could try to match what they built, as an extra check. Another advantage is that the upload itself would be reduced in size.

(Something other than -SB would also be fine.)

X

-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'stable'), (300, 'unstable'), (200, 'experimental'), (1, 'experimental-debug')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.6.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages dpkg-dev depends on:
ii  binutils      2.27.51.20161108-1
ii  bzip2         1.0.6-8
ii  libdpkg-perl  1.18.15
ii  make          4.1-9
ii  patch         2.7.5-1
pn  perl:any      <none>
ii  tar           1.29b-1.1
ii  xz-utils      5.2.2-1.2

Versions of packages dpkg-dev recommends:
ii  build-essential          12.2
ii  clang-3.5 [c-compiler]   1:3.5.2-5
ii  fakeroot                 1.21-2
ii  gcc [c-compiler]         4:6.1.1-1
ii  gcc-6 [c-compiler]       6.2.0-13
ii  gnupg                    2.1.16-2
ii  gnupg2                   2.1.16-2
ii  gpgv                     2.1.16-2
ii  libalgorithm-merge-perl  0.08-3

Versions of packages dpkg-dev suggests:
ii  debian-keyring  2016.09.04

-- no debconf information

Message #12 received at 846164@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Ximin Luo <infinity0@debian.org>, 846164@bugs.debian.org

Cc: Ansgar Burchardt <ansgar@debian.org>, ftpmaster@debian.org

Subject: Re: Bug#846164: dpkg-dev: Have dpkg-genchanges support a source+buildinfo-only upload

Date: Wed, 30 Nov 2016 05:54:40 +0100

Hi!

On Mon, 2016-11-28 at 22:00:32 +0100, Ximin Luo wrote:
> Package: dpkg-dev
> Version: 1.18.15
> Severity: wishlist

> Currently one can do
> 
> $ dpkg-buildpackage --changes-option=-S
> 
> to do a binary build locally, but only upload the source package
> so that buildds build on all arches. Note that this command *does*
> generate a buildinfo file, but does not include it in the changes
> file.

This is actually a bug, which I mentioned in:

  <https://lists.debian.org/debian-dpkg/2016/11/msg00056.html>

but probably failed to make this more clear there.

> It would be good to be able to do something like
> 
> $ dpkg-buildpackage --changes-option=-SB
> 
> to do a binary build locally, but only upload the source package
> *and* the buildinfo file.

This was the intention from the beginning. But when I mentioned this
on IRC the other day, Ansgar said ftp-masters might not be too happy
about accepting buildinfo files referencing artifacts that we might
not be able to generated. Because there's no distinction between
packages that are reproducible and ones that are not.

> This was one idea we had near the beginning of the R-B project. The
> idea being developers could do this, then the buildds could try to
> match what they built, as an extra check. Another advantage is that
> the upload itself would be reduced in size.

Those are the reasons I gave to Ansgar, but I'd like his and
ftp-masters input on this.

Right now I'm feeling a bit stuck with this because I'm not sure I
can fulfill all desires: me fixing the bug, making repro peple happy,
and making ftp-masters happy. :)

Thanks,
Guillem

Message #17 received at 846164@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org>

To: Guillem Jover <guillem@debian.org>, 846164@bugs.debian.org

Cc: Ansgar Burchardt <ansgar@debian.org>, ftpmaster@debian.org

Subject: Re: Bug#846164: dpkg-dev: Have dpkg-genchanges support a source+buildinfo-only upload

Date: Wed, 30 Nov 2016 12:42:00 +0000

Guillem Jover:
> On Mon, 2016-11-28 at 22:00:32 +0100, Ximin Luo wrote:
>> It would be good to be able [..]
>> to do a binary build locally, but only upload the source package
>> *and* the buildinfo file.
> 
> This was the intention from the beginning. But when I mentioned this
> on IRC the other day, Ansgar said ftp-masters might not be too happy
> about accepting buildinfo files referencing artifacts that we might
> not be able to generated. Because there's no distinction between
> packages that are reproducible and ones that are not.
> 
>> This was one idea we had near the beginning of the R-B project. The
>> idea being developers could do this, then the buildds could try to
>> match what they built, as an extra check. Another advantage is that
>> the upload itself would be reduced in size.
> 
> Those are the reasons I gave to Ansgar, but I'd like his and
> ftp-masters input on this.
> 

OK, hopefully I can also argue in favour of it:

Indeed there's no distinction between packages that are reproducible, vs ones that are not. But this is the case even if we don't have source+buildonly-only uploads, and stick to binary or source-only uploads as in the current situation.

That is - if ftp-masters don't accept source+buildinfo-only uploads, and only allow source-only uploads, then there's *already* no indication whether the package is reproducible or not. People can certainly upload unreproducible packages and these would be accepted into the archive today.

What source+buildinfo-only .changes files do is, it allows the archive infrastructure to attempt a reproduction itself, since it has some knowledge of the result is "supposed" to be. If the reproduction fails, then it can either:

1. accept the upload (the current behaviour)
2. accept the upload but warn the uploader that it was unreproducible, and that the actually-built binaries are being used instead
3. reject the upload

I guess (3) would require much much more changes to the archive infrastructure and we're not even sure if we want that. However (2) would also be a significant improvement towards reproducibility, and is feasible to achieve - this wishlist feature being the first steps towards achieving that.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

Message #20 received at 846164-submitter@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: 846164-submitter@bugs.debian.org

Subject: Bug#846164 in package dpkg marked as pending

Date: Fri, 27 Jan 2017 04:12:40 +0000

Control: tag 846164 pending

Hi!

Bug #846164 in package dpkg reported by you has been fixed in
the dpkg/dpkg.git Git repository. You can see the changelog below, and
you can check the diff of the fix at:

    https://anonscm.debian.org/cgit/dpkg/dpkg.git/diff/?id=0701185

---
commit 0701185eae3629224a7f74a2ff1b8e1d08630180
Author: Guillem Jover <guillem@debian.org>
Date:   Fri Dec 23 00:25:38 2016 +0100

    dpkg-genchanges: Include .buildinfo files also for source-only uploads
    
    The .buildinfo file also makes sense on source-only uploads, because it is
    still a build. And more so when we have done a full build, but filtered the
    changes to only include the sources in the upload.
    
    In any case, this was the intended behavior from the beginning.
    
    Closes: #846164

diff --git a/debian/changelog b/debian/changelog
index ba9fae5..2cdf455 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -31,6 +31,8 @@ dpkg (1.18.19) UNRELEASED; urgency=medium
     architectures for artifacts we are not going to distribute, and do not
     unnecessarily recompute the checksums for artifacts like the sources.
   * Do not compute the architecture list twice in dpkg-genchanges.
+  * Include .buildinfo files also for source-only uploads in dpkg-genchanges.
+    Closes: #846164
   * Portability:
     - On GNU/Hurd try to use the new process executable name attribute from
       libps, to properly match on start-stop-daemon --exec.

Message #27 received at 846164-close@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: 846164-close@bugs.debian.org

Subject: Bug#846164: fixed in dpkg 1.18.19

Date: Fri, 27 Jan 2017 06:03:34 +0000

Source: dpkg
Source-Version: 1.18.19

We believe that the bug you reported is fixed in the latest version of
dpkg, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 846164@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Guillem Jover <guillem@debian.org> (supplier of updated dpkg package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Fri, 27 Jan 2017 05:43:36 +0100
Source: dpkg
Binary: dpkg libdpkg-dev dpkg-dev libdpkg-perl dselect
Architecture: source
Version: 1.18.19
Distribution: unstable
Urgency: medium
Maintainer: Dpkg Developers <debian-dpkg@lists.debian.org>
Changed-By: Guillem Jover <guillem@debian.org>
Description:
 dpkg       - Debian package management system
 dpkg-dev   - Debian package development tools
 dselect    - Debian package management front-end
 libdpkg-dev - Debian package management static library
 libdpkg-perl - Dpkg perl modules
Closes: 831524 843925 846164 847926 848705 849081 849913 851441 851889 851891
Changes:
 dpkg (1.18.19) unstable; urgency=medium
 .
   [ Guillem Jover ]
   * Stop emitting Built-For-Profiles from dpkg-gencontrol. The information
     is already provided in .buildinfo files, and including it in the binary
     packages makes them unreproducible even when the profile used would not
     alter its contents. Closes: #831524
   * Do not allow empty epochs and revisions in versions. When there's at
     least one colon or one dash, we should expect epoch and revision numbers.
   * Always set SOURCE_DATE_EPOCH in dpkg-buildpackage and dpkg-source. Use
     the current date if the changelog does not have one. Closes: #849081
   * Refactor update-alternatives pathname existence check into a new function.
   * Avoid useless repeated lstat()s in update-alternatives.
   * Only check for debian/tests/control file once in dpkg-source.
   * Generate Testsuite-Restrictions fields from the test restrictions in
     dpkg-source into .dsc files. Closes: #847926
     Based on a patch by Iain Lane <laney@debian.org>.
   * Improve the ELF ABI mismatch detector in dpkg-shlibdeps, by parsing the
     ELF header ourselves. While still not perfect (things like linux-i386 and
     hurd-i386 will still match), it will filter lots of previously matching
     objects that should have been ignored, and will work even when objdump
     does not know about the specific object details. Closes: #849913
   * Add initial support for DEB_BUILD_OPTIONS to dpkg-genbuildinfo. This will
     make it possible to enable or disable specific features that should be
     recorded in the .buildinfo file. For now only “all” and “path” are
     supported. Closes: #848705
   * Add again the architecture from the filename to .changes files for any
     artifact with one. This reverts the change introduced in dpkg 1.18.11.
   * Fold the filtering and checksumming of files to distribute in a .changes
     file in dpkg-genchanges into the initial loop. This way we do not include
     architectures for artifacts we are not going to distribute, and do not
     unnecessarily recompute the checksums for artifacts like the sources.
   * Do not compute the architecture list twice in dpkg-genchanges.
   * Include .buildinfo files also for source-only uploads in dpkg-genchanges.
     Closes: #846164
   * Fix check for expected number of binary artifacts in dpkg-genchanges, to
     only take into account the artifacts that we are distributing.
   * Fix parsing of Pre-Depends and Depends in dpkg-genbuildinfo, so that
     the code parses both and not just the first to appear in the stanza.
     Based on a patch by Johannes Schauer <josch@debian.org>.
   * Add support for signed .buildinfo files to dpkg-buildpackage. Add new
     -ui and --unsigned-buildinfo options. Closes: #843925
   * Portability:
     - On GNU/Hurd try to use the new process executable name attribute from
       libps, to properly match on start-stop-daemon --exec.
   * Perl modules:
     - Fix Debian architecture wildcard parsing so that matching four-tuple
       matchings work. Missed in dpkg 1.18.11.
       Reported by Julian Andres Klode <jak@debian.org>.
     - Add new import tags for Dpkg::Arch.
     - Abort on EOF in patch name prompt in Dpkg::Source::Package::V2,
       instead of getting into an infinite loop. Closes: #851441
     - Call anonymous subs via -> operator instead of casting with &, and fix
       bogus POD documentation to match the code.
     - Add new Auto-Built-Package field to Dpkg::Control::Fields.
     - Add a new debug() reporting function, and switch code to use it.
     - Add new Dpkg::BuildOption parse_features() method refactored from
       Dpkg::Vendor::Debian.
   * Documentation:
     - Cleanup software requirements in README.
     - Move control member file references from dpkg(1) to deb(5).
     - Fix typos in docs and code comments.
     - Document Auto-Built-Package field in deb-control(5).
   * Build system:
     - Disable disk pre-allocation by default, but let the builder re-enable
       it via a new configure option. This has been causing major performance
       issues on "modern" filesystems.
   * Packaging:
     - Add debsig-verify to dpkg Suggests. The code optionally supports this
       specific signed .deb verification program.
       Prompted by Stuart Prescott <stuart@debian.org>.
   * Test suite:
     - Generate and check all currently possible architecture wildcards.
     - Correctly iterate over all default and passed .dsc template substvars.
 .
   [ Updated programs translations ]
   * Dutch (Frans Spiesschaert). Closes: #851889
   * German (Sven Joachim).
 .
   [ Updated scripts translations ]
   * German (Helge Kreutzmann).
 .
   [ Updated man pages translations ]
   * Dutch (Frans Spiesschaer). Closes: #851891
   * German (Helge Kreutzmann).
Checksums-Sha1:
 b095dc40f8f1a76a1f0cafe3a4c33b9527cead67 2032 dpkg_1.18.19.dsc
 f8ec626d3503e0c8e6dfff5d11c95104811db9db 4516116 dpkg_1.18.19.tar.xz
 a24f616884b03619e07017518053202651875d5a 7301 dpkg_1.18.19_amd64.buildinfo
Checksums-Sha256:
 8b46dcac0a09b0c9ca9a462c1b23b2ece9ec5d5c5d9a4a1aa91406d83de7be78 2032 dpkg_1.18.19.dsc
 67c8b4d580497991892ecd6745267ed4be9f65d2cc842b75b758f999c6ee7bbb 4516116 dpkg_1.18.19.tar.xz
 683b0c34af65ea0ac7ded8e63395d937dd9494a97b7c317640def47a7d30c1e4 7301 dpkg_1.18.19_amd64.buildinfo
Files:
 b41ba9c5d6a34aba330ffec62a2f0cae 2032 admin required dpkg_1.18.19.dsc
 231a66f09747e1b77b236ff48cd71a9e 4516116 admin required dpkg_1.18.19.tar.xz
 d0fd205f0f98b27401700522514e1e37 7301 admin required dpkg_1.18.19_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEETz509DYFDBD1aWV0uXK/PqSuV6MFAliK348ACgkQuXK/PqSu
V6M6Dg//R5i0tqcZStNnQTKmwCfK10A49fS87qTHRydeuqpEZGC2jt8b41LEmmqm
p4dNInnpH3LJPpJ1+SWFWP91vX6AwrRbW1a+5KcSYrsfJQqPzi2TF77uofcPs5Dd
Uf9C1TQCU6TjV9Pb0JA4D1e3QLjIFX2QFyBVQbzqWb1XsLnyvXZ2l2Etj8f2EO97
OXVEDdTNy1s9biY//PYW2d+e+nX7IG+eNXwI5qfy7PXC05bvhUPcxEWA2LSMM+WB
0zjT0QkUFG+Nxf6Fm6FT62N4ms67xbCi4nvYg2PKMbXAGDmhwJYm+JAioFhQFmUv
0S1Y6X1v2RDYuhI/HJfTvJ3SreClvtgGKQR4H231axuv0+ZXNcXe17DXI0z5qe2l
vaUrmafPHcBPxtkl9pNzBJPYUvmev8jKRp2XOqxlwnDWoFkeupcMxjItyK3bTyfN
4alLG4H66BmHGVxyNSTOoSsqoz1Ew+HlhUX0TkacURqigKICLEThgwzVdmqbIm1d
BdI8UVB5trXAmG8nyEoMKkaGkh6nI5a5TKnKsl7iiXJdg6nhaqU8J/MXLeorNe8a
bzWFsp4SbfZuiQuG2OTXBevjmwGaLTa9VXllFtuf1CeiiVxn706lupN/eTieRtZH
VMec0ITJ8U6Hx2mxqmASORuSUfyTJ+AFrxThfkm7Cxc+/AsKmho=
=sLUv
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.