Debian Bug report logs - #806331
xz-utils: make the selected POSIX shell stable accross build environments

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Jérémy Bobbio <lunar@debian.org>

To: submit@bugs.debian.org

Subject: xz-utils: make the selected POSIX shell stable accross build environments

Date: Thu, 26 Nov 2015 17:35:26 +0100

[Message part 1 (text/plain, inline)]

Source: xz-utils
Version: 5.1.1alpha+20120614-2.1
Severity: normal
User: reproducible-builds@lists.alioth.debian.org
Usertags: environment
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org, sanvila@debian.org

Hi!

While working on the “reproducible builds” effort [1], we have noticed
that xz-utils could not be built reproducibly.

When dash is the default shell, the configure script
m4/posix-shell.m4 will select /bin/bash as the “conforming POSIX
shell”. When bash is the default shell, /bin/sh will be selected.

The binary package currently in sid on amd64 uses /bin/bash. As bash
is currently Essential:yes (#103284), this is probably not a problem.
But I wonder if they would not be troubles if the package was built on
an environment were bash is the default shell, but later installed on a
system where /bin/dash is /bin/sh.

So for reproducibility and safety reason, it would be best if the
selected path to the shell would not depend on the build environment.

Thanks!

 [1]: https://wiki.debian.org/ReproducibleBuilds

-- 
Lunar                                .''`. 
lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
                                    `. `'` 
                                      `-   

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org>

To: 806331@bugs.debian.org, xz-devel@tukaani.org

Subject: Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Tue, 7 Jun 2016 13:33:25 +0200

[Message part 1 (text/plain, inline)]

Control: tags -1 + patch upstream

I've attached a patch that makes m4/posix-shell.m4 try constant paths first. This should fix the issue.

Upstream should also apply it - see more-stable-shell.patch.

This *should* be enough to make xz-utils completely reproduce (#806328 is now fixed in another way), but I haven't yet tested this.

X

On Thu, 26 Nov 2015 17:35:26 +0100 =?iso-8859-1?B?Suly6W15?= Bobbio <lunar@debian.org> wrote:
> Source: xz-utils
> Version: 5.1.1alpha+20120614-2.1
> Severity: normal
> User: reproducible-builds@lists.alioth.debian.org
> Usertags: environment
> X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org, sanvila@debian.org
> 
> Hi!
> 
> While working on the “reproducible builds” effort [1], we have noticed
> that xz-utils could not be built reproducibly.
> 
> When dash is the default shell, the configure script
> m4/posix-shell.m4 will select /bin/bash as the “conforming POSIX
> shell”. When bash is the default shell, /bin/sh will be selected.
> 
> The binary package currently in sid on amd64 uses /bin/bash. As bash
> is currently Essential:yes (#103284), this is probably not a problem.
> But I wonder if they would not be troubles if the package was built on
> an environment were bash is the default shell, but later installed on a
> system where /bin/dash is /bin/sh.
> 
> So for reproducibility and safety reason, it would be best if the
> selected path to the shell would not depend on the build environment.
> 
> Thanks!
> 
>  [1]: https://wiki.debian.org/ReproducibleBuilds
> 
> -- 
> Lunar                                .''`. 
> lunar@debian.org                    : :Ⓐ  :  # apt-get install anarchism
>                                     `. `'` 
>                                       `-   

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

[more-stable-shell (text/plain, attachment)]

[xz-utils_5.1.1alpha+20120614-2.1_5.1.1alpha+20120614-2.2~reproducible1.debdiff (text/plain, attachment)]

Message #17 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Lasse Collin <lasse.collin@tukaani.org>

To: xz-devel@tukaani.org

Cc: 806331@bugs.debian.org

Subject: Re: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 15 Jun 2016 22:04:52 +0300

On 2016-06-07 Ximin Luo wrote:
> I've attached a patch that makes m4/posix-shell.m4 try constant paths
> first. This should fix the issue.
> 
> Upstream should also apply it - see more-stable-shell.patch.

Thanks!

posix-shell.m4 comes from gnulib so it would be nice if you could send
the patch there and discuss the issue with gnulib developers. Perhaps
there is a reason why the shells are tested in that order, although the
gnulib commit messages don't have any reasoning. A guess is that
someone might prefer if the same shell is used for running
configure and the test suite scripts. Anyway, getting it fixed in gnulib
would get it fixed in projects other than XZ Utils too.

One can force the POSIX shell to a specific value on the configure
command line by passing, for example, "gl_cv_posix_shell=/bin/sh" as an
argument. It's not documented in the --help output but it's mentioned
in INSTALL section 3.1. That is an alternative to patching to get
reproducible builds.

-- 
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode

Message #22 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org>

To: Lasse Collin <lasse.collin@tukaani.org>, xz-devel@tukaani.org, bug-gnulib@gnu.org, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Cc: 806331@bugs.debian.org

Subject: Re: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 15 Jun 2016 21:31:55 +0200

[Message part 1 (text/plain, inline)]

+bugs-gnulib, reproducible-builds

Lasse Collin:
> On 2016-06-07 Ximin Luo wrote:
>> I've attached a patch that makes m4/posix-shell.m4 try constant paths
>> first. This should fix the issue.
>>
>> Upstream should also apply it - see more-stable-shell.patch.
> 
> Thanks!
> 
> posix-shell.m4 comes from gnulib so it would be nice if you could send
> the patch there and discuss the issue with gnulib developers. Perhaps
> there is a reason why the shells are tested in that order, although the
> gnulib commit messages don't have any reasoning. A guess is that
> someone might prefer if the same shell is used for running
> configure and the test suite scripts. Anyway, getting it fixed in gnulib
> would get it fixed in projects other than XZ Utils too.
> 

Thanks, yes this wasn't clear to me. We'd still need to contact projects that have already copied posix-shell.m4 into their source tree, but I suppose at least future projects will benefit.

bugs-gnulib, do you see any issue with this patch? The context is that some projects embed POSIX_SHELL into build products, so for build reproducibility it is better to have this detection script first try constant paths.

(Yes, if /bin/sh is not POSIX for one of the build machines, then we still have unreproducibility. But that's much less likely to happen. This is an "effort vs correctness" trade-off that I'm making.)

Regarding "someone might prefer if the same shell" as Lasse theorised, I'm not sure this will ever be an issue: if they prefer "the same shell", they would just use $SHELL or $CONFIG_SHELL instead of $POSIX_SHELL, so the only useful scenario is if somehow they wanted a shell that was POSIX-compatible but for some reason (I can't imagine why) needed this selection to favour $SHELL/$CONFIG_SHELL but still for it to be OK to fall back to the other constant values.

> One can force the POSIX shell to a specific value on the configure
> command line by passing, for example, "gl_cv_posix_shell=/bin/sh" as an
> argument. It's not documented in the --help output but it's mentioned
> in INSTALL section 3.1. That is an alternative to patching to get
> reproducible builds.
> 

Yeah, I saw that too, but thought this approach was a bit cleaner.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

[more-stable-shell (text/plain, attachment)]

Message #27 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Ximin Luo <infinity0@debian.org>

Cc: Lasse Collin <lasse.collin@tukaani.org>, xz-devel@tukaani.org, bug-gnulib@gnu.org, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>, 806331@bugs.debian.org

Subject: Re: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 15 Jun 2016 19:46:29 +0000 (UTC)

Ximin Luo dixit:

>bugs-gnulib, do you see any issue with this patch? The context is that

I’m not bugs-gnulib, but I’ve ported many a code using autotools to new
platforms (MirBSD, MidnightBSD, Interix, Debian GNU/kFreeBSD), and also
hacked autotools some. For added credibility, I’m the developer of mksh
(a contemporary kind of Korn shell).

I think this patch is not only harmful but also actually hinders your
higher goal of reproducibility.

$CONFIG_SHELL should stay the ultimate instance. Users absolutely MUST
be able to override not-working-but-recognised-as-good-enough system
shells like /bin/sh with this variable, to get things working even in
the presence of known bugs.

Au contraire, I suggest you to export CONFIG_SHELL in debian/rules,
possibly buildflags.mk even (ask Guillem if he’d mind adding it).

bye,
//mirabilos
-- 
FWIW, I'm quite impressed with mksh interactively. I thought it was much
*much* more bare bones. But it turns out it beats the living hell out of
ksh93 in that respect. I'd even consider it for my daily use if I hadn't
wasted half my life on my zsh setup. :-) -- Frank Terbeck in #!/bin/mksh

Message #32 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org>

To: bug-gnulib@gnu.org

Cc: Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>, xz-devel@tukaani.org, 806331@bugs.debian.org

Subject: Re: [Reproducible-builds] [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 15 Jun 2016 22:05:34 +0200

Thorsten Glaser:
> Ximin Luo dixit:
> 
>> bugs-gnulib, do you see any issue with this patch? The context is that
> 
> [..]
> 
> $CONFIG_SHELL should stay the ultimate instance. Users absolutely MUST
> be able to override not-working-but-recognised-as-good-enough system
> shells like /bin/sh with this variable, to get things working even in
> the presence of known bugs.
> 

Hi,

I'm not sure if you understood what was being discussed. How does my proposed patch affect your scenario? This is not about CONFIG_SHELL, but about POSIX_SHELL. Could you give a concrete example or situation, which my patch will cause to fail?

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Message #37 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Ximin Luo <infinity0@debian.org>, 806331@bugs.debian.org

Cc: bug-gnulib@gnu.org, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>, xz-devel@tukaani.org, debian-bugs-dist@lists.debian.org, Jonathan Nieder <jrnieder@gmail.com>

Subject: Re: Bug#806331: [Reproducible-builds] [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 15 Jun 2016 20:24:23 +0000 (UTC)

Ximin Luo dixit:

>I'm not sure if you understood what was being discussed.

I understand it extremely well.

>proposed patch affect your scenario? This is not about CONFIG_SHELL,

It is. Straight from your diff:

>      for gl_cv_posix_shell in \
>-        "$CONFIG_SHELL" "$SHELL" /bin/sh /bin/bash /bin/ksh /bin/sh5 no; do
>+        /bin/sh /bin/bash /bin/ksh /bin/sh5 "$CONFIG_SHELL" "$SHELL" no; do

This makes $CONFIG_SHELL no longer the preferred POSIX shell.

This breaks when a system has a /bin/sh that passes all gnulib
tests (and thus is used as POSIX_SHELL) but has other bugs that
make it unusable, for example. The user can then no longer over‐
ride this bad choice of the configury.

bye,
//mirabilos
-- 
„Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund,
mksh auf jedem System zu installieren.“
	-- XTaran auf der OpenRheinRuhr, ganz begeistert
(EN: “[…]uhr.gz is a reason to install mksh on every system.”)

Message #42 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org>

To: 806331@bugs.debian.org, bug-gnulib@gnu.org

Cc: Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>, xz-devel@tukaani.org

Subject: Re: Bug#806331: [Reproducible-builds] [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 15 Jun 2016 22:44:36 +0200

(Did you mean to add debian-bugs-dist and Jonathan Nieder on purpose or by accident? I removed them, but feel free to add them back in.)

Thorsten Glaser:
>> proposed patch affect your scenario? This is not about CONFIG_SHELL,
> 
> It is. Straight from your diff:
> 
>>      for gl_cv_posix_shell in \
>> -        "$CONFIG_SHELL" "$SHELL" /bin/sh /bin/bash /bin/ksh /bin/sh5 no; do
>> +        /bin/sh /bin/bash /bin/ksh /bin/sh5 "$CONFIG_SHELL" "$SHELL" no; do
> 
> This makes $CONFIG_SHELL no longer the preferred POSIX shell.
> 
> This breaks when a system has a /bin/sh that passes all gnulib
> tests (and thus is used as POSIX_SHELL) but has other bugs that
> make it unusable, for example. The user can then no longer over‐
> ride this bad choice of the configury.
> 

In such a case, it is a bug to be using $POSIX_SHELL - which only tests for conformance with POSIX and not these "other bugs that make it unusable".

In other words: in your scenario, one would not use $POSIX_SHELL but some other specific test for those "other bugs".

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Message #47 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Paul Eggert <eggert@cs.ucla.edu>

To: Ximin Luo <infinity0@debian.org>, 806331@bugs.debian.org, bug-gnulib@gnu.org

Cc: xz-devel@tukaani.org, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: Re: Bug#806331: [Reproducible-builds] [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 15 Jun 2016 14:02:58 -0700

On 06/15/2016 01:44 PM, Ximin Luo wrote:
> In such a case, it is a bug to be using $POSIX_SHELL - which only tests for conformance with POSIX and not these "other bugs that make it unusable".
Gnulib can't test for all POSIX violations, only for the ones it knows 
about. CONFIG_SHELL lets the user override Gnulib's guess in 
environments where the guess is wrong. This sort of thing has been in 
Gnulib (and Autoconf) for ages, I expect many people have grown used to 
it, and I'm leery of changing this just for the purpose of reproducible 
builds. For reproducible builds, I suggest configuring with 
CONFIG_SHELL=/bin/sh as that should make the build reproducible without 
having to change Autoconf or Gnulib.

More generally, 'configure' and reproducible builds are competing 
objectives. 'configure' aims to guess characteristics of the target 
environment by depending on details of the build environment; in 
contrast, reproducible builds want to suppress details of the build 
environment whenever possible. Probably the best way to marry these two 
is for the reproducible build to start with a reproducible environment, 
and setting CONFIG_SHELL to a known value is one step in that direction.

Message #52 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org>

To: Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>, 806331@bugs.debian.org, bug-gnulib@gnu.org

Cc: xz-devel@tukaani.org

Subject: Re: [Reproducible-builds] Bug#806331: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 15 Jun 2016 23:22:37 +0200

Paul Eggert:
> On 06/15/2016 01:44 PM, Ximin Luo wrote:
>> In such a case, it is a bug to be using $POSIX_SHELL - which only tests for conformance with POSIX and not these "other bugs that make it unusable".
> Gnulib can't test for all POSIX violations, only for the ones it knows about. CONFIG_SHELL lets the user override Gnulib's guess in environments where the guess is wrong. This sort of thing has been in Gnulib (and Autoconf) for ages, I expect many people have grown used to it, and I'm leery of changing this just for the purpose of reproducible builds. For reproducible builds, I suggest configuring with CONFIG_SHELL=/bin/sh as that should make the build reproducible without having to change Autoconf or Gnulib.
> 

OK, thanks for the explanation. I was under the impression that, since POSIX is a finite set of specified behaviours, posix-shell would be or could be a complete test. But I guess it's harder than I thought, to write that sort of test.

> More generally, 'configure' and reproducible builds are competing objectives. 'configure' aims to guess characteristics of the target environment by depending on details of the build environment; in contrast, reproducible builds want to suppress details of the build environment whenever possible. Probably the best way to marry these two is for the reproducible build to start with a reproducible environment, and setting CONFIG_SHELL to a known value is one step in that direction.

I wouldn't say they're necessarily competing, just that the build process (e.g. 'configure' in this case) needs to more clearly distinguish between the build and the host environment - like how compilers do. So for example, here the "most correct" solution would be to add a HOST_POSIX_SHELL and default this to POSIX_SHELL (or something). But this is also more complex, so I can understand if you don't want to do this.

Thanks for the tip, we'll try the other options.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Message #57 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Ximin Luo <infinity0@debian.org>

Cc: 806331@bugs.debian.org, bug-gnulib@gnu.org, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>, xz-devel@tukaani.org

Subject: Re: Bug#806331: [Reproducible-builds] [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 15 Jun 2016 23:45:57 +0000 (UTC)

Ximin Luo dixit:

>(Did you mean to add debian-bugs-dist and Jonathan Nieder on purpose
>or by accident? I removed them, but feel free to add them back in.)

I didn’t, maybe debbugs did.

>In other words: in your scenario, one would not use $POSIX_SHELL but
>some other specific test for those "other bugs".

No, you don’t understand what I tried to write. The thing is, this
is about the *user* (the person calling the configure script) being
able to *override* the automatic mechanism to determine POSIX_SHELL
by setting CONFIG_SHELL to some value, forcing it to use it (if it
passes the tests).

Removing this ability will be A Bad Thing™.

bye,
//mirabilos
-- 
18:47⎜<mirabilos:#!/bin/mksh> well channels… you see, I see everything in the
same window anyway      18:48⎜<xpt:#!/bin/mksh> i know, you have some kind of
telnet with automatic pong         18:48⎜<mirabilos:#!/bin/mksh> haha, yes :D
18:49⎜<mirabilos:#!/bin/mksh> though that's more tinyirc – sirc is more comfy

Message #62 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Ximin Luo <infinity0@debian.org>

Cc: Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>, 806331@bugs.debian.org, bug-gnulib@gnu.org, xz-devel@tukaani.org

Subject: Re: [Reproducible-builds] Bug#806331: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 15 Jun 2016 23:51:41 +0000 (UTC)

Ximin Luo dixit:

>needs to more clearly distinguish between the build and the host
>environment - like how compilers do. So for example, here the "most
>correct" solution would be to add a HOST_POSIX_SHELL and default this

No, this is outside of the scope of autotools and a common misuse
of them actually.

The real bug here is that the configure.ac script of the package
hardcodes POSIX_SHELL in the output while your reproducible builds
effort treats this as a(n unsupported, at least in this scenario)
cross compilation.

The same issue would happen when you set CONFIG_SHELL=/bin/bash
and then cross-compile xz-utils from GNU/Linux to, say, MirBSD,
where There Is No Such Thing As /bin/bash.

So the root of the problem is that xz-utils is not cross-compilable
(well, not completely, and enough for Helmut’s bootstrapping effort).

bye,
//mirabilos
-- 
Yay for having to rewrite other people's Bash scripts because bash
suddenly stopped supporting the bash extensions they make use of
	-- Tonnerre Lombard in #nosec

Message #67 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Paul Eggert <eggert@cs.ucla.edu>, 806331@bugs.debian.org

Cc: Ximin Luo <infinity0@debian.org>, bug-gnulib@gnu.org, xz-devel@tukaani.org, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>, Jonathan Nieder <jrnieder@gmail.com>

Subject: Re: Bug#806331: [Reproducible-builds] [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 15 Jun 2016 23:48:00 +0000 (UTC)

Paul Eggert dixit:

> CONFIG_SHELL lets the user override Gnulib's guess in environments where the
> guess is wrong. This sort of thing has been in Gnulib (and Autoconf) for ages,
> I expect many people have grown used to it, and I'm leery of changing this just
> for the purpose of reproducible builds. For reproducible builds, I suggest
> configuring with CONFIG_SHELL=/bin/sh as that should make the build
> reproducible without having to change Autoconf or Gnulib.

Right. (If /bin/sh is usable and constant, that is. Some packages in
Debian may cause this to become unreproducible, in which case setting
CONFIG_SHELL=/bin/bash is, unfortunately, the correct thing to do.)

> More generally, 'configure' and reproducible builds are competing objectives.

Yes…

> Probably the best way to marry these two is for the reproducible
> build to start with a reproducible environment, and setting
> CONFIG_SHELL to a known value is one step in that direction.

This is something I hinted at earlier, and fully agree with!
Thanks!

bye,
//mirabilos
-- 
“It is inappropriate to require that a time represented as
 seconds since the Epoch precisely represent the number of
 seconds between the referenced time and the Epoch.”
	-- IEEE Std 1003.1b-1993 (POSIX) Section B.2.2.2

Message #72 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org>

To: Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Cc: bug-gnulib@gnu.org, xz-devel@tukaani.org, 806331@bugs.debian.org

Subject: Re: [Reproducible-builds] Bug#806331: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Thu, 16 Jun 2016 03:20:11 +0200

Thorsten Glaser:
> Ximin Luo dixit:
> 
>> needs to more clearly distinguish between the build and the host
>> environment - like how compilers do. So for example, here the "most
>> correct" solution would be to add a HOST_POSIX_SHELL and default this
> 
> No, this is outside of the scope of autotools and a common misuse
> of them actually.
> 
> The real bug here is that the configure.ac script of the package
> hardcodes POSIX_SHELL in the output while your reproducible builds
> effort treats this as a(n unsupported, at least in this scenario)
> cross compilation.
> 
> The same issue would happen when you set CONFIG_SHELL=/bin/bash
> and then cross-compile xz-utils from GNU/Linux to, say, MirBSD,
> where There Is No Such Thing As /bin/bash.
> 
> So the root of the problem is that xz-utils is not cross-compilable
> (well, not completely, and enough for Helmut’s bootstrapping effort).
> 

I agree there is an issue with configure.ac, but it can only be fully solved via HOST_POSIX_SHELL. 

In other words, the reason that xz-utils is not completely cross-compilable is *because* there is no such thing as HOST_POSIX_SHELL. xz-utils (using gnulib macros) is *unable* to distinguish between these two cases:

1. build-time scripts that require a POSIX shell
2. places where it needs to embed POSIX_SHELL into a shebang line, for scripts installed into the host architecture

xz-utils is forced to use POSIX_SHELL for both these cases, and that is where the "bug" comes in. It has nothing to do with reproducible builds, although it does manifest in a more obvious way if you are trying to reproducible a build across different machines.

If HOST_POSIX_SHELL was available, then xz-utils could use this for case (2). Then, if we wanted to cross-compile xz-utils from GNU/Linux to MirBSD, we would set HOST_POSIX_SHELL to whatever shell works on MirBSD.

Distinguishing cases (1) and (2) is not a "misuse" nor "outside the scope" of autotools, but a vital part of any well-engineered build tool.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Message #77 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org>

To: xz-devel@tukaani.org

Cc: 806331@bugs.debian.org

Subject: Re: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Tue, 5 Jul 2016 23:38:10 +0200

[Message part 1 (text/plain, inline)]

Lasse Collin:
> posix-shell.m4 comes from gnulib
> 
> [..]
> 
> One can force the POSIX shell to a specific value on the configure
> command line by passing, for example, "gl_cv_posix_shell=/bin/sh" as an
> argument. It's not documented in the --help output but it's mentioned
> in INSTALL section 3.1. That is an alternative to patching to get
> reproducible builds.
> 

Hi Lasse,

Looks like the gnulib guys weren't willing to accept my patch so I'll go the gl_cv_posix_shell route. This is Debian-specific, so xz-utils won't need to make any changes.

Thanks for your time and your pointers!

@Debian maintainer, please see the attached patch.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

[xz-utils_5.1.1alpha+20120614-2.1_5.1.1alpha+20120614-2.2.debdiff (text/plain, attachment)]

Message #82 received at 806331@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org>

Cc: 806331@bugs.debian.org, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: Re: [xz-devel] Re: xz-utils: make the selected POSIX shell stable accross build environments

Date: Wed, 28 Jun 2017 17:10:00 +0000

[Message part 1 (text/plain, inline)]

Ximin Luo:
> Lasse Collin:
>> posix-shell.m4 comes from gnulib
>>
>> [..]
>>
>> One can force the POSIX shell to a specific value on the configure
>> command line by passing, for example, "gl_cv_posix_shell=/bin/sh" as an
>> argument. It's not documented in the --help output but it's mentioned
>> in INSTALL section 3.1. That is an alternative to patching to get
>> reproducible builds.
>>
> 
> Hi Lasse,
> 
> Looks like the gnulib guys weren't willing to accept my patch so I'll go the gl_cv_posix_shell route. This is Debian-specific, so xz-utils won't need to make any changes.
> 
> Thanks for your time and your pointers!
> 
> @Debian maintainer, please see the attached patch.
> 

I've uploaded this to DELAYED/14. The debdiff is attached, it's the same patch as before just with a proper changelog entry.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

[xz-utils_5.2.2-1.3.debdiff (text/plain, attachment)]

Message #87 received at 806331-close@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org>

To: 806331-close@bugs.debian.org

Subject: Bug#806331: fixed in xz-utils 5.2.2-1.3

Date: Wed, 12 Jul 2017 18:07:03 +0000

Source: xz-utils
Source-Version: 5.2.2-1.3

We believe that the bug you reported is fixed in the latest version of
xz-utils, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 806331@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Ximin Luo <infinity0@debian.org> (supplier of updated xz-utils package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Wed, 28 Jun 2017 18:39:19 +0200
Source: xz-utils
Binary: liblzma5 xz-utils xzdec liblzma-dev liblzma-doc
Architecture: source
Version: 5.2.2-1.3
Distribution: unstable
Urgency: medium
Maintainer: Jonathan Nieder <jrnieder@gmail.com>
Changed-By: Ximin Luo <infinity0@debian.org>
Description:
 liblzma-dev - XZ-format compression library - development files
 liblzma-doc - XZ-format compression library - API documentation
 liblzma5   - XZ-format compression library
 xz-utils   - XZ-format compression utilities
 xzdec      - XZ-format compression utilities - tiny decompressors
Closes: 806331
Changes:
 xz-utils (5.2.2-1.3) unstable; urgency=medium
 .
   * Non-maintainer upload.
   * Force a constant /bin/sh for installed scripts. This helps the build
     be reproducible; /bin/sh on Debian is always POSIX. (Closes: #806331)
Checksums-Sha1:
 49fcb01276585fae842516264eb771659756ac73 2575 xz-utils_5.2.2-1.3.dsc
 7913d19b1280c3f5a1aead93cbc64a8972d8c1b2 108680 xz-utils_5.2.2-1.3.debian.tar.xz
 293b1ac01a7cb2b917e14db3ce392d1e0d9f24df 5686 xz-utils_5.2.2-1.3_source.buildinfo
Checksums-Sha256:
 3ea4e6a32f6265b152f89ceafe78c8839e5f4bb1cad137b159fe2013817f9342 2575 xz-utils_5.2.2-1.3.dsc
 d76c3acf39d0999c14384695394970e8f98853fd6736ba91972d3e67106bc6f6 108680 xz-utils_5.2.2-1.3.debian.tar.xz
 3df2402250af3707c7a062a30c44e855b77258b899316c71e215e8bc5822ccd3 5686 xz-utils_5.2.2-1.3_source.buildinfo
Files:
 8b0c93ec2f10c01b2b7e7b8999971de7 2575 utils required xz-utils_5.2.2-1.3.dsc
 d42bf32fb1d1623ab810b6ce882eb885 108680 utils required xz-utils_5.2.2-1.3.debian.tar.xz
 0f71895b8ef90fcf67fcabc139f61a60 5686 utils required xz-utils_5.2.2-1.3_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQJJBAEBCgAzFiEENmdIajJtsnZtJVVGhg3vO49lC3kFAllT28sVHGluZmluaXR5
MEBkZWJpYW4ub3JnAAoJEIYN7zuPZQt5bmUQAIEDEmBZzZoR3mGyGhETiJX4yu68
5nYzrEd8VbR+sogRVPK+uCLbplEFrapOuAKOab5Ro+C05L6gqdkW9og6i6LjtMWh
LpU0u8eWDYd0Xqv0M7NufU3/WmdlES7eU5H3tCVnGJz+IgYpA+4zaPIxLfqX0gHl
XhgdgYlLww1VHiJElEFVjJ0aBiGp2NsPufqVwXR5WoNw5nMiC7kEvyN/dHYrOpRY
cD+1p6k/XH7zrp1F/5ypKcIfB+d2F5DXHr9yVnmI7fZS9Y4GpIDbvz+VDKbQUZ1J
Rqb+B2PFonDQUnUL+I2Qh+A/9SKaJiw3E+mIMZ6JufRq/bj4uJ5z6sMUNTArTKnO
AZEph52nyjXho8kk/PjEL8uvsyA0lgDhJJJ4ELUr1U4kZzxjddd5OXDoFvgLOtKV
hT/9SVYCMtavmkp8/9e9fb8oHEnpaTUgEKb6uWTYpjA+feLXLw8telPKEneYNFS0
Tc5RcdhfQdLomQs0QEjxMgrEXhSgTr9BCcoEphfpIPR1yNFLB+mZuLiofN8Lj4CH
h3hvu5hgPaH03ij6WH3qOvz6OuMOSSHkyfEfsUYpQ5nVog5OAfaiYCLSr1RIVeKj
TF5fEfA/rBtlb93YjCwpS+i/ib8c6YcOz0hKTJl31v23xGqA1Bg+vKw7zopKlxMB
sE4hnB/YigoohQ9D
=tI7L
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.