Debian Bug report logs - #1006471
ruby3.0: reproducible builds: embeds path to various binaries

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: submit@bugs.debian.org

Subject: ruby3.0: reproducible builds: embeds path to various binaries

Date: Fri, 25 Feb 2022 15:26:51 -0800

[Message part 1 (text/plain, inline)]

Source: ruby3.0
Severity: normal
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: usrmerge shell
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org

The paths to various binaries, which differs on a usrmerge
vs. non-usrmerge system, are embedded in rbconfig.rb:

  https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffoscope-results/ruby3.0.html

  /usr/lib/x86_64-linux-gnu/ruby/3.0.0/rbconfig.rb

  CONFIG["EGREP"]·=·"/bin/grep·-E"
  vs.
  CONFIG["EGREP"]·=·"/usr/bin/grep·-E"

Patch attached which passes variables to configure to use the
non-usrmerge locations, as usrmerge installations typically have
compatibility symlinks, but not vice-versa. The patch also sets
variables to ensure consistent values for bash, which can be triggered
when /bin/sh points to bash.

This patch alone does not fix all reproducibility issues (e.g. build
paths on unstable and experimental), but should build reproducibly once
it migrates to bookworm/testing!


Thanks for maintaining ruby3.0!


live well,
  vagrant

[0001-debian-rules-Pass-variables-to-configure-to-make-the.patch (text/x-diff, inline)]

From d0a1365cba685b8ab22be92463d28fb79d55a03b Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Fri, 25 Feb 2022 23:17:08 +0000
Subject: [PATCH] debian/rules: Pass variables to configure to make the package
 build reproducibly regardless of usrmerge.

The variables EGREP, GREP, MAKEDIRS, MKDIR_P and SHELL should all
point to their non-usrmerge locations.

https://tests.reproducible-builds.org/debian/issues/paths_vary_due_to_usrmerge_issue.html
---
 debian/rules | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/debian/rules b/debian/rules
index 76fa0b6..2d2c086 100755
--- a/debian/rules
+++ b/debian/rules
@@ -64,6 +64,14 @@ configure_options += --with-compress-debug-sections=no
 export DEB_BUILD_MAINT_OPTIONS = hardening=+bindnow optimize=-lto
 configure_options += $(shell dpkg-buildflags --export=configure)
 
+# Pass variables to ensure consistent values when built on a usrmerge
+# or non-usrmerge system.
+configure_options += EGREP='/bin/grep -E'
+configure_options += GREP='/bin/grep'
+configure_options += MAKEDIRS='/bin/mkdir -p'
+configure_options += MKDIR_P='/bin/mkdir -p'
+configure_options += SHELL='/bin/sh'
+
 # For more info see #999351
 ifneq (,$(filter $(DEB_HOST_ARCH),alpha))
 export DEB_CFLAGS_MAINT_APPEND += -O1
-- 
2.30.2

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 1006471@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>, 1006471@bugs.debian.org

Subject: Re: Bug#1006471: ruby3.0: reproducible builds: embeds path to various binaries

Date: Sun, 17 Jul 2022 12:04:45 +0100

Control: severity -1 serious

On Fri, 25 Feb 2022 at 15:26:51 -0800, Vagrant Cascadian wrote:
> The paths to various binaries, which differs on a usrmerge
> vs. non-usrmerge system, are embedded in rbconfig.rb:
> 
>   https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffoscope-results/ruby3.0.html
> 
>   /usr/lib/x86_64-linux-gnu/ruby/3.0.0/rbconfig.rb
> 
>   CONFIG["EGREP"]·=·"/bin/grep·-E"
>   vs.
>   CONFIG["EGREP"]·=·"/usr/bin/grep·-E"

If these CONFIG variables are used for something at runtime, then this
will become a practical problem as soon as Debian starts using merged-/usr
buildds. The problem scenario is:

- ruby3.0 is built on a merged-/usr buildd
- /usr/bin/grep is recorded in rbconfig.rb
- this build of ruby3.0 is installed on a non-merged-/usr system during
  the upgrade from Debian 11 to Debian 12
- whatever feature uses CONFIG["EGREP"] will not work, because
  non-merged-/usr systems only have /bin/grep

Technical Committee resolution
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#110 recommends
that this class of bug is treated as release-critical, so I'm raising the
severity of this bug report.

If none of the affected CONFIG variables are actually used for anything
on installed systems, then the severity of this bug can be downgraded
to non-RC (but it would be better to fix it anyway, because reproducible
builds are a useful goal for other reasons).

> Patch attached which passes variables to configure to use the
> non-usrmerge locations, as usrmerge installations typically have
> compatibility symlinks, but not vice-versa.

To clarify: in Debian, merged-/usr installations are *guaranteed* to
have these compatibility symlinks. The patch looks appropriate to me,
although I have not tested it.

    smcv

Message #17 received at 1006471@bugs.debian.org (full text, mbox, reply):

From: Antonio Terceiro <terceiro@debian.org>

To: Simon McVittie <smcv@debian.org>, 1006471@bugs.debian.org

Cc: Vagrant Cascadian <vagrant@reproducible-builds.org>

Subject: Re: Bug#1006471: ruby3.0: reproducible builds: embeds path to various binaries

Date: Wed, 27 Jul 2022 08:49:27 -0300

[Message part 1 (text/plain, inline)]

Control: clone -1 -2
Control: reassign -2 src:ruby3.1
Control: retitle -2 ruby3.1: reproducible builds: embeds path to various binaries

Hi,

On Sun, Jul 17, 2022 at 12:04:45PM +0100, Simon McVittie wrote:
> Control: severity -1 serious
> 
> On Fri, 25 Feb 2022 at 15:26:51 -0800, Vagrant Cascadian wrote:
> > The paths to various binaries, which differs on a usrmerge
> > vs. non-usrmerge system, are embedded in rbconfig.rb:
> > 
> >   https://tests.reproducible-builds.org/debian/rb-pkg/bookworm/amd64/diffoscope-results/ruby3.0.html
> > 
> >   /usr/lib/x86_64-linux-gnu/ruby/3.0.0/rbconfig.rb
> > 
> >   CONFIG["EGREP"]·=·"/bin/grep·-E"
> >   vs.
> >   CONFIG["EGREP"]·=·"/usr/bin/grep·-E"
> 
> If these CONFIG variables are used for something at runtime, then this
> will become a practical problem as soon as Debian starts using merged-/usr
> buildds. The problem scenario is:
> 
> - ruby3.0 is built on a merged-/usr buildd
> - /usr/bin/grep is recorded in rbconfig.rb
> - this build of ruby3.0 is installed on a non-merged-/usr system during
>   the upgrade from Debian 11 to Debian 12
> - whatever feature uses CONFIG["EGREP"] will not work, because
>   non-merged-/usr systems only have /bin/grep
> 
> Technical Committee resolution
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994388#110 recommends
> that this class of bug is treated as release-critical, so I'm raising the
> severity of this bug report.
> 
> If none of the affected CONFIG variables are actually used for anything
> on installed systems, then the severity of this bug can be downgraded
> to non-RC (but it would be better to fix it anyway, because reproducible
> builds are a useful goal for other reasons).

Those variables are read from config.status during the builds. Maybe
this should be fixed centrally in autoconf instead?

> > Patch attached which passes variables to configure to use the
> > non-usrmerge locations, as usrmerge installations typically have
> > compatibility symlinks, but not vice-versa.
> 
> To clarify: in Debian, merged-/usr installations are *guaranteed* to
> have these compatibility symlinks. The patch looks appropriate to me,
> although I have not tested it.

Sure.

[signature.asc (application/pgp-signature, inline)]

Message #24 received at 1006471@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Antonio Terceiro <terceiro@debian.org>

Cc: 1006471@bugs.debian.org, Vagrant Cascadian <vagrant@reproducible-builds.org>

Subject: Re: Bug#1006471: ruby3.0: reproducible builds: embeds path to various binaries

Date: Wed, 27 Jul 2022 14:52:47 +0100

On Wed, 27 Jul 2022 at 08:49:27 -0300, Antonio Terceiro wrote:
> > On Fri, 25 Feb 2022 at 15:26:51 -0800, Vagrant Cascadian wrote:
> > > The paths to various binaries, which differs on a usrmerge
> > > vs. non-usrmerge system, are embedded in rbconfig.rb:
> > > 
> > >   /usr/lib/x86_64-linux-gnu/ruby/3.0.0/rbconfig.rb
> > > 
> > >   CONFIG["EGREP"]·=·"/bin/grep·-E"
> > >   vs.
> > >   CONFIG["EGREP"]·=·"/usr/bin/grep·-E"
> > 
> > If these CONFIG variables are used for something at runtime, then this
> > will become a practical problem as soon as Debian starts using merged-/usr
> > buildds.
> 
> Those variables are read from config.status during the builds. Maybe
> this should be fixed centrally in autoconf instead?

autoconf is designed to support arbitrarily bad host OSs, including those
that are non-POSIX or otherwise defective, where the only fully-functional
grep might be /opt/sw/addons/misc/gnu/grep or something; so it has a
tendency to discover a known-good absolute path and save that.

This is great if you're building Ruby on an awful 1990s Unix machine
and the result of AC_PROG_EGREP will only be used during build, or if
it is used at runtime but you only plan to run the resulting Ruby binaries
on that same machine, but it goes wrong when facts about the build system
start to diverge from facts about the host system.
In this case, the fact that is different is the merged-/usr status
of the build system and the host system, but it could be almost anything.

The macros that Ruby uses to find these commands are probably AC_PROG_GREP,
AC_PROG_EGREP, etc., which are not explicitly documented to output an
absolute path (the documentation just says "whatever is chosen"), but
looking at their implementation, it seems they do: they are like
AC_PATH_PROG rather than AC_CHECK_PROG.

It's entirely possible that Ruby is not doing this deliberately, those
macros might well be a dependency for something else.

Is the Ruby build intentionally putting EGREP into rbconfig.rb for use by
some other component, or is it just populating that file with everything
that Autoconf happens to have discovered, on the off chance that it
might become necessary at some point? If the latter, then that seems
like it will cause unpredictable action-at-a-distance if Autoconf stops
needing to discover some particular thing (for instance if Autoconf's
maintainers decide that they are only going to support systems where
the first grep in PATH is POSIX.1-2001 compliant, and stop checking for a
possibly-better-quality grep elsewhere).

    smcv

Message #29 received at 1006471@bugs.debian.org (full text, mbox, reply):

From: Antonio Terceiro <terceiro@debian.org>

To: Simon McVittie <smcv@debian.org>

Cc: 1006471@bugs.debian.org, Vagrant Cascadian <vagrant@reproducible-builds.org>

Subject: Re: Bug#1006471: ruby3.0: reproducible builds: embeds path to various binaries

Date: Fri, 29 Jul 2022 08:07:52 -0300

[Message part 1 (text/plain, inline)]

Control: tag -1 + pending

On Wed, Jul 27, 2022 at 02:52:47PM +0100, Simon McVittie wrote:
> Is the Ruby build intentionally putting EGREP into rbconfig.rb for use by
> some other component, or is it just populating that file with everything
> that Autoconf happens to have discovered, on the off chance that it
> might become necessary at some point? If the latter, then that seems
> like it will cause unpredictable action-at-a-distance if Autoconf stops
> needing to discover some particular thing (for instance if Autoconf's
> maintainers decide that they are only going to support systems where
> the first grep in PATH is POSIX.1-2001 compliant, and stop checking for a
> possibly-better-quality grep elsewhere).

The later. Thanks for the explanation. This has already been fixed for
ruby3.1, and is fixed in git for ruby3.0.

[signature.asc (application/pgp-signature, inline)]

Message #34 received at 1006471-submitter@bugs.debian.org (full text, mbox, reply):

From: Antonio Terceiro <noreply@salsa.debian.org>

To: 1006471-submitter@bugs.debian.org

Subject: Bug#1006471 marked as pending in ruby

Date: Fri, 29 Jul 2022 11:06:37 +0000

Control: tag -1 pending

Hello,

Bug #1006471 in ruby reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:

https://salsa.debian.org/ruby-team/ruby/-/commit/5dbe7a8f795d2427379cb76c6fd64de69f963cfb

------------------------------------------------------------------------
debian/rules: ensure rbconfig.rb is reproducible regardless of usr-merge

Closes: #1006471
Signed-off-by: Vagrant Cascadian <vagrant@reproducible-builds.org>
Signed-off-by: Antonio Terceiro <terceiro@debian.org>
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/1006471

Message #39 received at 1006471-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 1006471-close@bugs.debian.org

Subject: Bug#1006471: fixed in ruby3.0 3.0.4-8

Date: Sun, 11 Sep 2022 02:42:28 +0000

Source: ruby3.0
Source-Version: 3.0.4-8
Done: Antonio Terceiro <terceiro@debian.org>

We believe that the bug you reported is fixed in the latest version of
ruby3.0, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1006471@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Antonio Terceiro <terceiro@debian.org> (supplier of updated ruby3.0 package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Sat, 10 Sep 2022 23:02:54 -0300
Source: ruby3.0
Architecture: source
Version: 3.0.4-8
Distribution: unstable
Urgency: medium
Maintainer: Debian Ruby Team <pkg-ruby-extras-maintainers@lists.alioth.debian.org>
Changed-By: Antonio Terceiro <terceiro@debian.org>
Closes: 1006471 1006476 1018230
Changes:
 ruby3.0 (3.0.4-8) unstable; urgency=medium
 .
   [ Vagrant Cascadian ]
   * debian/rules: ensure rbconfig.rb is reproducible regardless of usr-merge
     (Closes: #1006471)
   * debian/rules: Strip the build path from rbconfig.rb (Closes: #1006476)
 .
   [ Antonio Terceiro ]
   * libruby3.0: depend on packages that used to be provided by ruby2.7.
     This allows *ruby2.7 to be removed after upgrades from bullseye where
     the user has installed packages that depend on either ruby-webrick or
     ruby-sdbm.
   * rbconfig, mkmf: call foreign pkg-config when cross compiling
     (Closes: #1018230)
Checksums-Sha1:
 353a7e98bb89ee0c7747eb159e04b35dc9033b4d 2509 ruby3.0_3.0.4-8.dsc
 a022126967ae8a38e2ecf1b6cc383a5055907b69 162128 ruby3.0_3.0.4-8.debian.tar.xz
 006f167e71c9e4451380bfb839774a45de621481 7872 ruby3.0_3.0.4-8_source.buildinfo
Checksums-Sha256:
 67c72c380157a8df024db7a83b9998f839eb08b58f0dfb0bea36c6477c1be941 2509 ruby3.0_3.0.4-8.dsc
 a4dc02a5ebee13a920232cc950a66169e1e7c5924db74841319aa433816e10d2 162128 ruby3.0_3.0.4-8.debian.tar.xz
 be4a7749ad48258ec61ab4294d3a856a711be8814f3e9cc4ce81f8018b23f0da 7872 ruby3.0_3.0.4-8_source.buildinfo
Files:
 573b7fbeb7346ec36bcc971d8621e478 2509 ruby optional ruby3.0_3.0.4-8.dsc
 33895a89f751e39ca13f6cb56c970706 162128 ruby optional ruby3.0_3.0.4-8.debian.tar.xz
 21fbec4fbfd8ea12f1b4c4a591c3ac37 7872 ruby optional ruby3.0_3.0.4-8_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEst7mYDbECCn80PEM/A2xu81GC94FAmMdRz0ACgkQ/A2xu81G
C96ICw//f3yq6vBlcA3PfilDdXJbuTPeU2cFtLiMiE59S7o1Xl5WJmXPOexaeRHf
F+WvrKjTGB3alEe24ywVkwjm0Qrv6I2sQEI2fVUJOz/sVq7YgceGQT4+bXF9x46b
5NqJvjBGfXbzQrBxUiNH4rB3KdOtUbv27xbYAqM/ejNQXioEp8GmsmWFZLjBeTaW
lb1sPSb3jvj76KYtMhIHEpxuIQlaJ+aGNbd1s80baFU0eeeW4qbaLTPh+MSWSD1R
HJMg5HwV3BQAEEVVfNbeMTaEhJkCa4BONIl1zYlFN+j6iuiBYC49FkIcbuHwqD9P
3AjOPDnv74zZEM0VHNB9Mhv+LEgqMj4pvXUPyHqZ+rXXd3SghgKOrD8UdGABje2u
9JmLQAbbBi0Pptu6qruca5l9v4H8o7ic4QebXsyRS3OkJARgzHaz/BK8wD6JY8+x
ISxaseEO1BSIVfkl8CrUuLvoQ3ISRHi49A4JgVdh8RtZaMvEzdzk2nAEpXYOEwW2
qSnG/kPg4H1k47lpvy8Ar3bzx8DrYxgQYQU963W1/CskfRFbQkFbvrPL+254sdDS
hhA9L4k1k8oTvgeZK+ZWsxRKYl+0QfbS4w3QuqeccQ7JeC4I2SVHNl0SnPk738Wo
sUhGbkWQlXo98U+iXDJzAJZNH0EJcDWvw2bsHLPfsJ65cAJ3Anc=
=1IA2
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.