Debian Bug report logs - #1009797
apt: support "nodoc" build profile

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: submit@bugs.debian.org

Subject: apt: support "nodoc" build profile

Date: Sun, 17 Apr 2022 18:50:19 -0700

[Message part 1 (text/plain, inline)]

Source: apt
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: randomness
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org

There are some non-deterministic identifiers that doxygen introduces
into apt's documentation packages:

  https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffoscope-results/apt.html

The attached patch, adds support for the "nodoc" build profile, allowing
to build the other apt packages reproducibly by excluding the
documentation packages.

This also allows building functional apt packages with a smaller
dependency chain, so might help with bootstrapping efforts too!

I thought docbook* and xsltproc could also be excluded from the
Build-Depends, but that triggered some other build failures.


Of course, ideally building documentation reproducibly would be very
nice as well, so it would be good to eventually fix the underlying
issues in doxygen:

  https://tests.reproducible-builds.org/debian/issues/unstable/nondeterminstic_todo_identifiers_in_documentation_generated_by_doxygen_issue.html
  https://tests.reproducible-builds.org/debian/issues/unstable/nondeterministic_ordering_in_documentation_generated_by_doxygen_issue.html


Thanks for maintaining apt!


live well,
  vagrant

[0002-Add-support-for-nodoc-build-profile.patch (text/x-diff, inline)]

From d408a3b439dafa4378a1eec6c7c4dbefe4986897 Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Sun, 17 Apr 2022 22:11:12 +0000
Subject: [PATCH 2/2] Add support for "nodoc" build profile.

  https://wiki.debian.org/BuildProfileSpec#Registered_profile_names
---
 debian/control | 4 +++-
 debian/rules   | 8 +++++++-
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/debian/control b/debian/control
index 2f1d0515e..7f10eec3e 100644
--- a/debian/control
+++ b/debian/control
@@ -31,7 +31,7 @@ Build-Depends: cmake (>= 3.4),
                triehash,
                xsltproc,
                zlib1g-dev
-Build-Depends-Indep: doxygen, graphviz, w3m
+Build-Depends-Indep: doxygen <!nodoc>, graphviz <!nodoc>, w3m <!nodoc>
 Vcs-Git: https://salsa.debian.org/apt-team/apt.git
 Vcs-Browser: https://salsa.debian.org/apt-team/apt
 
@@ -101,6 +101,7 @@ Priority: optional
 Depends: ${misc:Depends}
 Section: doc
 Multi-Arch: foreign
+Build-Profiles: <!nodoc>
 Description: documentation for APT
  This package contains the user guide and offline guide for various
  APT tools which are provided in a html and a text-only version.
@@ -125,6 +126,7 @@ Priority: optional
 Depends: ${misc:Depends}
 Section: doc
 Multi-Arch: foreign
+Build-Profiles: <!nodoc>
 Description: documentation for APT development
  This package contains documentation for development of the APT
  Debian package manipulation program and its libraries.
diff --git a/debian/rules b/debian/rules
index ce9218968..38db7819a 100755
--- a/debian/rules
+++ b/debian/rules
@@ -17,6 +17,12 @@ else
 	configure_test_flags =
 endif
 
+ifeq ($(filter nodoc,$(DEB_BUILD_PROFILES)),)
+	WITH_DOC=-DWITH_DOC=ON
+else
+	WITH_DOC=-DWITH_DOC=OFF
+endif
+
 %:
 	dh $@ --buildsystem=cmake+ninja
 
@@ -45,6 +51,6 @@ override_dh_installsystemd:
 	dh_installsystemd --remaining-packages
 
 override_dh_auto_configure-arch: flags=-DWITH_DOC=OFF
-override_dh_auto_configure-indep: flags=-DWITH_DOC=ON
+override_dh_auto_configure-indep: flags=$(WITH_DOC)
 override_dh_auto_configure-arch override_dh_auto_configure-indep:
 	dh_auto_configure -- $(flags) $(configure_test_flags)
-- 
2.30.2

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 1009797@bugs.debian.org (full text, mbox, reply):

From: David Kalnischkies <david@kalnischkies.de>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>, 1009797@bugs.debian.org

Subject: Re: Bug#1009797: apt: support "nodoc" build profile

Date: Mon, 18 Apr 2022 23:16:40 +0200

[Message part 1 (text/plain, inline)]

Hi,

On Sun, Apr 17, 2022 at 06:50:19PM -0700, Vagrant Cascadian wrote:
> This also allows building functional apt packages with a smaller
> dependency chain, so might help with bootstrapping efforts too!

Bootstrap usually doesn't care about arch:all packages, so that argument
doesn't work that well here. I would even say it works against you:


> I thought docbook* and xsltproc could also be excluded from the
> Build-Depends, but that triggered some other build failures.

They (alongside po4a) are used to build the manpages which we ship in
our arch:any packages (we could go with apt-common, but while that
saves mirror space, it could waste system space as you now have manpages
installed for things you haven't installed… or we go with multiple
apt-common packages which increases complexity and overhead… so far we
haven't gone down this road as it seems not very beneficial in the end).

We certainly could improve support for nodoc (upon your patch) by not
building the manpages in this profile which could indeed help boot-
strapping (although they never asked so far, which I am somewhat
surprised now to be honest) – but it would also end up changing the
contents of every package and hence spoil src:apt reproducibility in
that it will be reproducible on paper, but nobody can actually use the
result.


> Of course, ideally building documentation reproducibly would be very
> nice as well, so it would be good to eventually fix the underlying
> issues in doxygen:
> 
>   https://tests.reproducible-builds.org/debian/issues/unstable/nondeterminstic_todo_identifiers_in_documentation_generated_by_doxygen_issue.html
>   https://tests.reproducible-builds.org/debian/issues/unstable/nondeterministic_ordering_in_documentation_generated_by_doxygen_issue.html

It seems like hard issue(s) to solve and I am certainly not up to work
on this, but there seem not too many effected, so perhaps its worthwhile
to go the route of a nodoxygen (or pkg.*.nodoxygen) profile instead as
it would mean less variation and e.g. a reproducible binary apt package
would at least mean something as everyone has that variant installed.

I would at least be happy to beat our build system into omitting just
the doxygen part rather than some (currently with patch) or all
(possible future) docs. Shouldn't be hard (= famous last words).


Best regards

David Kalnischkies

[signature.asc (application/pgp-signature, inline)]

Message #15 received at 1009797@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: David Kalnischkies <david@kalnischkies.de>, 1009797@bugs.debian.org

Subject: Re: Bug#1009797: apt: support "nodoc" build profile

Date: Mon, 18 Apr 2022 19:32:15 -0700

[Message part 1 (text/plain, inline)]

On 2022-04-18, David Kalnischkies wrote:
> On Sun, Apr 17, 2022 at 06:50:19PM -0700, Vagrant Cascadian wrote:
>> This also allows building functional apt packages with a smaller
>> dependency chain, so might help with bootstrapping efforts too!
>
> Bootstrap usually doesn't care about arch:all packages, so that argument
> doesn't work that well here.

Fair.

> I would even say it works against you:

*raised eyebrows* :)


>> I thought docbook* and xsltproc could also be excluded from the
>> Build-Depends, but that triggered some other build failures.
>
> They (alongside po4a) are used to build the manpages which we ship in
> our arch:any packages (we could go with apt-common, but while that
> saves mirror space, it could waste system space as you now have manpages
> installed for things you haven't installed… or we go with multiple
> apt-common packages which increases complexity and overhead… so far we
> haven't gone down this road as it seems not very beneficial in the end).

Thanks for the explanation, that makes sense!


> We certainly could improve support for nodoc (upon your patch) by not
> building the manpages in this profile which could indeed help boot-
> strapping (although they never asked so far, which I am somewhat
> surprised now to be honest) –

Heh. Either way, works for me.


> but it would also end up changing the contents of every package and
> hence spoil src:apt reproducibility in that it will be reproducible on
> paper, but nobody can actually use the result.

I'm fine with that for my purposes, as it arguably a build profile is an
input to the build process; we don't expect packages built with a
different build profile to come out identical, especially the nodoc
profile which explicitly allows for differences in packages.

If you do two builds with "nodoc" and they come out identical, that
works for my use-case, with or without the man pages. Though I guess
then it's not so much "nodoc" as "lessdoc" which is less compelling as a
generic build profile name. :)


>> Of course, ideally building documentation reproducibly would be very
>> nice as well, so it would be good to eventually fix the underlying
>> issues in doxygen:
>> 
>>   https://tests.reproducible-builds.org/debian/issues/unstable/nondeterminstic_todo_identifiers_in_documentation_generated_by_doxygen_issue.html
>>   https://tests.reproducible-builds.org/debian/issues/unstable/nondeterministic_ordering_in_documentation_generated_by_doxygen_issue.html
>
> It seems like hard issue(s) to solve and I am certainly not up to work
> on this,

Indeed, which is why I am exploring the "nodoc" route.


> but there seem not too many effected, so perhaps its worthwhile
> to go the route of a nodoxygen (or pkg.*.nodoxygen) profile instead as
> it would mean less variation and e.g. a reproducible binary apt package
> would at least mean something as everyone has that variant installed.

Thanks, that's an interesting angle, will chew on it a bit!

I wanted to explore what we could get out of the existing and somewhat
established and broader scope "nodoc" build profile, as there are a few
other documentation generation tools with similar reproducibility issues
(sphinx comes to mind).


> I would at least be happy to beat our build system into omitting just
> the doxygen part rather than some (currently with patch) or all
> (possible future) docs. Shouldn't be hard (= famous last words).

Thanks, if it intrigues and inspires you, go for it, though I'd hate to
send you too deep down that rabbit hole otherwise...

I was mostly just looking at smallish changes that would give some
nominal level of ability to programatically check for reproducibility in
apt (and a few other remaining essential/required/build-essential
packages) even if we can't reproducibly build the documentation at the
moment.

One can manually see that the arch:any packages for apt are generally
reproducible in bookworm already:

  https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffoscope-results/apt.html

Amoung other plans, I'd hope to have stats at the binary package level
rather than just source package level on tests.r-b.org someday... but
not in the immediate future.


Thanks for the quick response and good thoughts!


live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Message #18 received at 1009797@bugs.debian.org (full text, mbox, reply):

From: Johannes Schauer Marin Rodrigues <josch@debian.org>

To: 1009797@bugs.debian.org, David Kalnischkies <david@kalnischkies.de>, Vagrant Cascadian <vagrant@reproducible-builds.org>

Subject: Re: Bug#1009797: apt: support "nodoc" build profile

Date: Tue, 19 Apr 2022 07:48:55 +0200

[Message part 1 (text/plain, inline)]

Quoting David Kalnischkies (2022-04-18 23:16:40)
> On Sun, Apr 17, 2022 at 06:50:19PM -0700, Vagrant Cascadian wrote:
> > I thought docbook* and xsltproc could also be excluded from the
> > Build-Depends, but that triggered some other build failures.
> 
> They (alongside po4a) are used to build the manpages which we ship in
> our arch:any packages (we could go with apt-common, but while that
> saves mirror space, it could waste system space as you now have manpages
> installed for things you haven't installed… or we go with multiple
> apt-common packages which increases complexity and overhead… so far we
> haven't gone down this road as it seems not very beneficial in the end).
> 
> We certainly could improve support for nodoc (upon your patch) by not
> building the manpages in this profile which could indeed help boot-
> strapping (although they never asked so far, which I am somewhat
> surprised now to be honest)

if apt is a problem for bootstrapping, you'd probably hear from Helmut
immediately. :)

Right now, to rebootstrap a new architecture, apt is cross compiled. This means
that build dependencies like xsltproc, docbook-xml and docbook-xsl can come
from an existing architecture because both packages are Multi-Arch:foreign.
This is why those build dependencies do not present a problem for
bootstrapping. Other big dependencies like doxygen, graphviz and w3m are in
Build-Depends-Indep so they are also not interesting for bootstrapping as they
are only used to create Architecture:all packages.

Thanks!

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #23 received at 1009797@bugs.debian.org (full text, mbox, reply):

From: Paul Gevers <elbrus@debian.org>

To: 1009797@bugs.debian.org

Subject: Re: Bug#1009797: apt: support "nodoc" build profile

Date: Sat, 7 May 2022 15:35:52 +0200

[Message part 1 (text/plain, inline)]

Hi,

On Tue, 19 Apr 2022 07:48:55 +0200 Johannes Schauer Marin Rodrigues 
<josch@debian.org> wrote:
> 
> This means
> that build dependencies like xsltproc, docbook-xml and docbook-xsl can come
> from an existing architecture because both packages are Multi-Arch:foreign.
> This is why those build dependencies do not present a problem for
> bootstrapping. Other big dependencies like doxygen, graphviz and w3m are in
> Build-Depends-Indep so they are also not interesting for bootstrapping as they
> are only used to create Architecture:all packages.

Just to add a rather unrelated argument for the nodoc support in apt. 
apt is (understandably) a key package. The Release Team (of which I'm a 
member) is currently exploring improvements to our tools to get a view 
RC buggy key packages. One of the facets we using is the nodoc (and 
nocheck) build annotations. So, we are really in favor of adding these 
nodoc annotations and the support of them in the build process.

Paul

[OpenPGP_signature (application/pgp-signature, attachment)]

Message #28 received at 1009797@bugs.debian.org (full text, mbox, reply):

From: David Kalnischkies <david@kalnischkies.de>

To: 1009797@bugs.debian.org

Cc: Vagrant Cascadian <vagrant@reproducible-builds.org>, Paul Gevers <elbrus@debian.org>

Subject: Re: Bug#1009797: apt: support "nodoc" build profile

Date: Sat, 7 May 2022 23:24:14 +0200

[Message part 1 (text/plain, inline)]

> Just to add a rather unrelated argument for the nodoc support in apt. apt is

For the record: I wrote nodoc and pkg.apt.nodoxygen support earlier last
month and have it finally proposed earlier today in a MR request…
https://salsa.debian.org/apt-team/apt/-/merge_requests/238

Another argument is actually that it helps our own CI tests: We do not
build the packages so far, but we build i386 and amd64 to run tests in
different setups (as root and as non-root) and building the docs twice
is kinda pointless, so now we don't do this anymore while also using
apts support for build-profiles to make cuts on setup costs as well.

So…

> Thanks, if it intrigues and inspires you, go for it, though I'd hate to
> send you too deep down that rabbit hole otherwise...

don't worry, I got a bit sidetracked and there could be done a lot more
(as always) but sometimes I enjoy digging myself into a hole even if
nobody really needs it… with three-ish (potential) "wouldn't it be nice"
users this feature set might even be in the upper bracket of usefulness
for things I did in these rabbit hole endeavours… ;P


Best regards

David Kalnischkies

[signature.asc (application/pgp-signature, inline)]

Message #33 received at 1009797-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 1009797-close@bugs.debian.org

Subject: Bug#1009797: fixed in apt 2.5.0

Date: Wed, 18 May 2022 08:22:18 +0000

Source: apt
Source-Version: 2.5.0
Done: Julian Andres Klode <jak@debian.org>

We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1009797@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Julian Andres Klode <jak@debian.org> (supplier of updated apt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Tue, 17 May 2022 18:50:45 +0200
Source: apt
Built-For-Profiles: noudeb
Architecture: source
Version: 2.5.0
Distribution: unstable
Urgency: medium
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Julian Andres Klode <jak@debian.org>
Closes: 1008036 1009796 1009797 1010029 1010030
Changes:
 apt (2.5.0) unstable; urgency=medium
 .
   [ Helmut Grohne ]
   * Avoid use of deprecated std::iterator (twice) (Closes: #1008036)
 .
   [ David Kalnischkies ]
   * Document tagfile-keys.h as internal to apt
   * Drop support for long obsoleted Suggests alias: Optional
   * Do not order long obsoleted fields anymore
   * Stop parsing undocumented unknown EDSP field APT-Hash
   * Avoid .c_str() on strings feed into pkgTagSection::FindS
   * Use pkgTagSection::Key in more places in src:apt
   * Parse Checksum fields via pkgTagSection::Key, too
   * Avoid building all docs in nodoc build profile
   * Add pkg.apt.nodoxygen build profile
   * Use build-dep & satisfy for our CI dependency resolution
   * Mark pkg-config-test autopkgtest as superficial
   * Avoid building inside the source dir in autopkgtest
   * Link interactive helpers against system libapt for autopkgtest
   * Remove unused public zlib include from libapt fileutl.h
   * Run ./prepare-release post-build checks in Gitlab CI
   * Do not build documentation twice in CI
   * Ignore stty failures in testcases
   * Include our config.h in all C++ files to avoid ODR violations
   * Ship our README.md in apt package
 .
   [ Frans Spiesschaert ]
   * Dutch program translation update (Closes: #1010029)
   * Dutch manpages translation update (Closes: #1010030)
 .
   [ Troy Varney ]
   * Fix mirror method dequeuing incorrect items
 .
   [ Julian Andres Klode ]
   * Do not accept arguments for apt-cache dotty, xvcg.
     These commands do not actually interpret the same
     arguments as depends, or any own ones for that matter.
   * tests: Add g++ dependency to pkg-config test
 .
   [ Vagrant Cascadian ]
   * Pass -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON via dh_auto_configure override
     (Closes: #1009796)
   * Add support for "nodoc" build profile (Closes: #1009797)
Checksums-Sha1:
 5c18d9d73517671760d3a3b4e6d6e87ad6a0da11 2933 apt_2.5.0.dsc
 88e92d17df9b3681f711d6337d68ebfef47bb344 2218792 apt_2.5.0.tar.xz
 673970e1da89025782a9a78db2a6bee5c0bfd071 10283 apt_2.5.0_source.buildinfo
Checksums-Sha256:
 2546e510fc11d17b29bc63dc0460ac58b3af2b12e953350468ab9644be653640 2933 apt_2.5.0.dsc
 61cd84d04e2a7858f70dca21160977479c74150853cdeef7c843d59e72488c1f 2218792 apt_2.5.0.tar.xz
 b2cee4c52ca4699aff3bb7fc43c317a47728ca7c2f5ec930e2af9929659008ed 10283 apt_2.5.0_source.buildinfo
Files:
 b2699c1572cf5bd85265d9f0ccc60cb2 2933 admin important apt_2.5.0.dsc
 4d0beed7d4732145269ab64fd6eaa811 2218792 admin important apt_2.5.0.tar.xz
 4b23e76db907741633655f0d2a27d826 10283 admin important apt_2.5.0_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQJDBAEBCgAtFiEET7WIqEwt3nmnTHeHb6RY3R2wP3EFAmKEo8APHGpha0BkZWJp
YW4ub3JnAAoJEG+kWN0dsD9xnMoQAItMDCJQ8j5pKzqH9xcgB0117mDE1G4Zrhoq
EBWHQZJa3ITyCWYBGAekcUIXVjvtV2nxWeTlkVz7HoT3D/Cft8yfxGZkeo7+rnTE
4gC6VTJZeZ9zG07km1H4/OwHv8aYQEJZWBCbtkuduUijOJCPac7SCVbHtvFaWdXx
9KcuIg+i6WEq4b3MRCwJ/UCruj6D37KcvPUkFNP29mz34sawsPCNFqk2e67ab4Uv
3sySJzD6rP5EriWVPruno4bAriOdCDsmVpOvYXP+jOVCXips7/R67/SYVKAh/EAb
gESIevQ8+M5yUt827Jw7SEqOvB6Ca1BGMH3yZ8QF2XsHLUYWYft/zx6SRef/CVOM
tbpKfZrw9589u9b+YuzLa2Bf1p6iCX9AEmpsdiXwJQ1Zax8U7gGcepAOQ9KmDvaD
2LhSMZGBguz9W0wUUMt9KLEOipuihY8phsurVhDmxrHNIgRKXXUMEFkTyMJ47DOw
xgbry97BsjYaTNPT3++BVCuAPOzjAlQ/0WhNk9UTxb7ubIxOt1WL8r5mNr+HuZjO
I13azE6ZgJqT0eC/cvLboP14e+or3RfKwzBBPWHSaoohoZvcoFeTL+v+whCogWBW
mzUHilnD1yQ3HiF1aVJ3LFnfYTfDRUxo/FxyNeldaPCWtTNQ3ftFC2RIbhjiY2BO
4qWFMAvt
=rUGa
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.