Debian Bug report logs - #1009196
texlive-binaries: Reproducible content of .fmt files

version graph

Package: texlive-binaries; Maintainer for texlive-binaries is Debian TeX Task Force <debian-tex-maint@lists.debian.org>; Source for texlive-binaries is src:texlive-bin (PTS, buildd, popcon).

Reported by: Roland Clobus <rclobus@rclobus.nl>

Date: Fri, 8 Apr 2022 17:15:02 UTC

Severity: wishlist

Tags: patch

Found in version texlive-bin/2021.20210626.59705-1

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, reproducible-bugs@lists.alioth.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Fri, 08 Apr 2022 17:15:04 GMT) (full text, mbox, link).


Acknowledgement sent to Roland Clobus <rclobus@rclobus.nl>:
New Bug report received and forwarded. Copy sent to reproducible-bugs@lists.alioth.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Fri, 08 Apr 2022 17:15:04 GMT) (full text, mbox, link).


Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Roland Clobus <rclobus@rclobus.nl>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: texlive-binaries: Reproducible content of .fmt files
Date: Fri, 08 Apr 2022 18:57:56 +0200
[Message part 1 (text/plain, inline)]
Package: texlive-binaries
Version: 2021.20210626.59705-1
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: randomness
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org

Hello maintainers of texlive-binaries,

While working on the “reproducible builds” effort [1], I have noticed that the
live image for Cinnamon in bookworm is no longer reproducible [2].

The attached patch ensures that the output of the function 'exception_strings'
always uses the same order of the hyphenation exceptions.
I've written the solution in C, perhaps someone more versed in lua could
rewrite it more elegantly.
(The lua manual says for the 'next' function: 'The order in which the indices
are enumerated is not specified' [3])

With the attached patch applied, I'm able (with the help of FORCE_SOURCE_DATE=1
and SOURCE_DATE_EPOCH) to reproducibly rebuild the .fmt files, as created by
'fmtutil --sys --all'.

Small test case to reproduce:
export FORCE_SOURCE_DATE=1
export SOURCE_DATE_EPOCH=$(date +%s)
for i in `seq 1 10`; do luahbtex -ini -jobname=luahbtex -progname=luabhtex
luatex.ini > /dev/null; md5sum luahbtex.*; done

With kind regards,
Roland Clobus

 [1]: https://wiki.debian.org/ReproducibleBuilds
 [2]:
https://jenkins.debian.net/view/live/job/reproducible_debian_live_build_cinnamon_bookworm/
 [3]: http://www.lua.org/manual/5.4/manual.html#pdf-next


-- System Information:
Debian Release: bookworm/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'testing-debug'), (50, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.16.0-5-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages texlive-binaries depends on:
ii  dpkg            1.21.7
ii  install-info    6.8-4+b1
ii  libc6           2.33-7
ii  libcairo2       1.16.0-5
ii  libfontconfig1  2.13.1-4.4
ii  libfreetype6    2.11.1+dfsg-1
ii  libgcc-s1       12-20220319-1
ii  libgraphite2-3  1.3.14-1
ii  libharfbuzz0b   2.7.4-1
ii  libicu67        67.1-7
ii  libkpathsea6    2021.20210626.59705-1
ii  libmpfr6        4.1.0-3
ii  libpaper1       1.1.28+b1
ii  libpixman-1-0   0.40.0-1
ii  libpng16-16     1.6.37-3
ii  libptexenc1     2021.20210626.59705-1
ii  libstdc++6      12-20220319-1
ii  libsynctex2     2021.20210626.59705-1
ii  libteckit0      2.5.11+ds1-1
ii  libtexlua53     2021.20210626.59705-1
ii  libtexluajit2   2021.20210626.59705-1
ii  libx11-6        2:1.7.2-2+b1
ii  libxaw7         2:1.0.14-1
ii  libxi6          2:1.8-1
ii  libxmu6         2:1.1.3-3
ii  libxpm4         1:3.5.12-1
ii  libxt6          1:1.2.1-1
ii  libzzip-0-13    0.13.72+dfsg.1-1.1
ii  perl            5.34.0-3
ii  t1utils         1.41-4
ii  tex-common      6.17
ii  zlib1g          1:1.2.11.dfsg-4

Versions of packages texlive-binaries recommends:
ii  dvisvgm       2.13.3-1
ii  texlive-base  2021.20220204-1

texlive-binaries suggests no packages.

-- no debconf information
[reproducible_exception_strings.patch (text/plain, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Mon, 11 Apr 2022 05:33:03 GMT) (full text, mbox, link).


Acknowledgement sent to Norbert Preining <norbert@preining.info>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Mon, 11 Apr 2022 05:33:03 GMT) (full text, mbox, link).


Message #10 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: Norbert Preining <norbert@preining.info>
To: dev-luatex@ntg.nl
Cc: Roland Clobus <rclobus@rclobus.nl>, 1009196@bugs.debian.org
Subject: Re: Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Mon, 11 Apr 2022 13:56:42 +0900
[Message part 1 (text/plain, inline)]
Hi Luigi, hi all luatex devs,

here at Debian we got a bug report about reproducability of luatex
format dumps. It contains a patch to make the hyphenation exception list
sorted. (I attach the patch)

Could you please take a look whether this is still relevant for the
latest release of luatex.

Thanks

Norbert

On Fri, 08 Apr 2022, Roland Clobus wrote:
> Hello maintainers of texlive-binaries,
> 
> While working on the “reproducible builds” effort [1], I have noticed that the
> live image for Cinnamon in bookworm is no longer reproducible [2].
> 
> The attached patch ensures that the output of the function 'exception_strings'
> always uses the same order of the hyphenation exceptions.
> I've written the solution in C, perhaps someone more versed in lua could
> rewrite it more elegantly.
> (The lua manual says for the 'next' function: 'The order in which the indices
> are enumerated is not specified' [3])
> 
> With the attached patch applied, I'm able (with the help of FORCE_SOURCE_DATE=1
> and SOURCE_DATE_EPOCH) to reproducibly rebuild the .fmt files, as created by
> 'fmtutil --sys --all'.
> 
> Small test case to reproduce:
> export FORCE_SOURCE_DATE=1
> export SOURCE_DATE_EPOCH=$(date +%s)
> for i in `seq 1 10`; do luahbtex -ini -jobname=luahbtex -progname=luabhtex
> luatex.ini > /dev/null; md5sum luahbtex.*; done
> 
> With kind regards,
> Roland Clobus
> 
>  [1]: https://wiki.debian.org/ReproducibleBuilds
>  [2]:
> https://jenkins.debian.net/view/live/job/reproducible_debian_live_build_cinnamon_bookworm/
>  [3]: http://www.lua.org/manual/5.4/manual.html#pdf-next
> 

--
PREINING Norbert                              https://www.preining.info
Mercari Inc.     +     IFMGA Guide     +     TU Wien     +     TeX Live
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
[reproducible_exception_strings.patch (text/plain, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Mon, 11 Apr 2022 07:15:10 GMT) (full text, mbox, link).


Acknowledgement sent to Hans Hagen <j.hagen@xs4all.nl>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Mon, 11 Apr 2022 07:15:10 GMT) (full text, mbox, link).


Message #15 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: Hans Hagen <j.hagen@xs4all.nl>
To: Norbert Preining <norbert@preining.info>, dev-luatex@ntg.nl
Cc: Roland Clobus <rclobus@rclobus.nl>, 1009196@bugs.debian.org
Subject: Re: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Mon, 11 Apr 2022 09:00:19 +0200
On 4/11/2022 6:56 AM, Norbert Preining wrote:
> Hi Luigi, hi all luatex devs,
> 
> here at Debian we got a bug report about reproducability of luatex
> format dumps. It contains a patch to make the hyphenation exception list
> sorted. (I attach the patch)
> 
> Could you please take a look whether this is still relevant for the
> latest release of luatex.
it actually defeats one of the security properties of lua (which was 
explicitly introduced at some point: make sure that hashes have random 
order each run so that it's harder to retrieve sensitive data from mem)

that said, it means that as soon as something gets stored in the format 
otherwise (than exceptions) one can face the same issue (although one 
can work around that by sorting etc)

if you want reproducibility for some testing, mess with this instead:

#if !defined(luai_makeseed)
#include <time.h>
#define luai_makeseed()		cast(unsigned int, time(NULL))
#endif

anyway, formats with embedded lua data (serialized or bytecode is never 
guaranteed the same unless one does soem effort)

fwiw: the easiest solution is to not store patterns and exceptions in 
the format and just load them runtime which is just as fast (in 
retrospect not a good idea to store it but it was needed for some plain 
compatibility testing)

Hans

(who in the past has been bitten by this 'random feature' when we made 
the switch to 5.3, or maybe it was even 5.2; it used to be 'random per 
binary' and became 'random per run' but we decided to stick with 
official lua)

-----------------------------------------------------------------
                                          Hans Hagen | PRAGMA ADE
              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
       tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------



Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Mon, 11 Apr 2022 11:03:02 GMT) (full text, mbox, link).


Acknowledgement sent to Norbert Preining <norbert@preining.info>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Mon, 11 Apr 2022 11:03:02 GMT) (full text, mbox, link).


Message #20 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: Norbert Preining <norbert@preining.info>
To: Hans Hagen <j.hagen@xs4all.nl>
Cc: dev-luatex@ntg.nl, Roland Clobus <rclobus@rclobus.nl>, 1009196@bugs.debian.org
Subject: Re: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Mon, 11 Apr 2022 20:01:32 +0900
Hi Hans, hi Roland,

thanks for your answer.

> it actually defeats one of the security properties of lua (which was
> explicitly introduced at some point: make sure that hashes have random order
> each run so that it's harder to retrieve sensitive data from mem)

Well, that is a good point to *not* implement the change.

Roland, do you have any comments? I guess the reproducability strive is
not as important as security.

So if something in this way should be done, it would need to
changes sort order if and only if FORCE_SOURCE_DATE=1 in the env
(this is what has required for tex engines to obey SOURCE_DATE_EPOCH
settings).

Roland, if you have time, please adjust the patch to work within the
above constraints.

Best regards

Norbert

--
PREINING Norbert                              https://www.preining.info
Mercari Inc.     +     IFMGA Guide     +     TU Wien     +     TeX Live
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13



Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Mon, 11 Apr 2022 11:51:05 GMT) (full text, mbox, link).


Acknowledgement sent to luigi scarso <luigi.scarso@gmail.com>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Mon, 11 Apr 2022 11:51:05 GMT) (full text, mbox, link).


Message #25 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: luigi scarso <luigi.scarso@gmail.com>
To: Norbert Preining <norbert@preining.info>
Cc: Hans Hagen <j.hagen@xs4all.nl>, Roland Clobus <rclobus@rclobus.nl>, luatex development list <dev-luatex@ntg.nl>, 1009196@bugs.debian.org
Subject: Re: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Mon, 11 Apr 2022 13:48:44 +0200
[Message part 1 (text/plain, inline)]
On Mon, Apr 11, 2022 at 1:01 PM Norbert Preining <norbert@preining.info>
wrote:

> Hi Hans, hi Roland,
>
> thanks for your answer.
>
> > it actually defeats one of the security properties of lua (which was
> > explicitly introduced at some point: make sure that hashes have random
> order
> > each run so that it's harder to retrieve sensitive data from mem)
>
> Well, that is a good point to *not* implement the change.
>
> Roland, do you have any comments? I guess the reproducability strive is
> not as important as security.
>
> So if something in this way should be done, it would need to
> changes sort order if and only if FORCE_SOURCE_DATE=1 in the env
> (this is what has required for tex engines to obey SOURCE_DATE_EPOCH
> settings).
>

not only fmt, every output  could suffer from the same problem if it
depends on a lua table that is not an array --   temp data, log and pdf .
The format should  serialize only array, or use a metatable
(e.g.
https://stackoverflow.com/questions/30970034/lua-in-pairs-with-same-order-as-its-written
)
Even if we hard code  in some way an ordered table data structure,  it's
still the responsibility of the format to use it -- but then  metatables
are more flexible.


-- 
luigi
[Message part 2 (text/html, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Mon, 11 Apr 2022 13:27:03 GMT) (full text, mbox, link).


Acknowledgement sent to Norbert Preining <norbert@preining.info>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Mon, 11 Apr 2022 13:27:03 GMT) (full text, mbox, link).


Message #30 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: Norbert Preining <norbert@preining.info>
To: luigi scarso <luigi.scarso@gmail.com>, 1009196@bugs.debian.org
Cc: Hans Hagen <j.hagen@xs4all.nl>, Roland Clobus <rclobus@rclobus.nl>, luatex development list <dev-luatex@ntg.nl>
Subject: Re: Bug#1009196: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Mon, 11 Apr 2022 22:26:04 +0900
> not only fmt, every output  could suffer from the same problem if it

If the final output (pdf) has traces of that, it might be of concern.
But for now the discussion is about the fmt dump, which is independent
of these items.

Best regards

Norbert

--
PREINING Norbert                              https://www.preining.info
Mercari Inc.     +     IFMGA Guide     +     TU Wien     +     TeX Live
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13



Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Mon, 11 Apr 2022 14:42:03 GMT) (full text, mbox, link).


Acknowledgement sent to Roland Clobus <rclobus@rclobus.nl>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Mon, 11 Apr 2022 14:42:03 GMT) (full text, mbox, link).


Message #35 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: Roland Clobus <rclobus@rclobus.nl>
To: Norbert Preining <norbert@preining.info>, Hans Hagen <j.hagen@xs4all.nl>
Cc: dev-luatex@ntg.nl, 1009196@bugs.debian.org
Subject: Re: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Mon, 11 Apr 2022 16:34:00 +0200
[Message part 1 (text/plain, inline)]
Hello Hans, Norbert,

Thanks for your answers.

On 11/04/2022 13:01, Norbert Preining wrote:
>> it actually defeats one of the security properties of lua (which was
>> explicitly introduced at some point: make sure that hashes have random order
>> each run so that it's harder to retrieve sensitive data from mem)
> 
> Well, that is a good point to *not* implement the change.
> 
> Roland, do you have any comments? I guess the reproducability strive is
> not as important as security.

Well, reproducibility is *another* aspect of security; this time not for 
the regular environments that users will use, but for build environments.

Reproducibility (as enforced by SOURCE_DATE_EPOCH) is typically enabled 
in an environment that generates binaries from source code for 
redistribution. It will guarantee that the build environment has not 
been tampered with, because you can (if you have made a similar build 
environment yourself) generate the binary files bit-for-bit identical. 
For a regular, production environment you should not have 
SOURCE_DATE_EPOCH set.

Other programming languages also have solved the security risks 
associated with the randomness of the hashes and reproducibility, see 
[1]. For Perl, the hashes can be de-randomized with PERL_HASH_SEED. 
Python uses PYTHONHASHSEED.
For Lua an environment variable LUA_HASH_SEED could be introduced, or 
per default the value of SOURCE_DATE_EPOCH (if set) instead of 
time(NULL) could be used to seed the hashes.

The texlive-binaries in Debian contain an embedded copy of Lua 5.3. The 
Lua 5.4 version of luai_makeseed is more complex, see [2]. I'll write a 
feature request for Lua later, that is out-of-scope for this scenario.

> So if something in this way should be done, it would need to
> changes sort order if and only if FORCE_SOURCE_DATE=1 in the env
> (this is what has required for tex engines to obey SOURCE_DATE_EPOCH
> settings).
> 
> Roland, if you have time, please adjust the patch to work within the
> above constraints.

Ack. Thanks for the pointer to luai_makeseed, that was some missing 
information that I needed. I'll post an updated patch soon (most 
probably much smaller and more elegant). As written above, the hash seed 
will be de-randomized only when both FORCE_SOURCE_DATE=1 and 
SOURCE_DATE_EPOCH are set.

With kind regards,
Roland Clobus

[1] https://reproducible-builds.org/docs/stable-outputs/
[2] https://sources.debian.org/src/lua5.4/5.4.4-1/src/lstate.c/?hl=73#L73
[OpenPGP_signature (application/pgp-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Mon, 11 Apr 2022 15:33:03 GMT) (full text, mbox, link).


Acknowledgement sent to Hans Hagen <j.hagen@xs4all.nl>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Mon, 11 Apr 2022 15:33:03 GMT) (full text, mbox, link).


Message #40 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: Hans Hagen <j.hagen@xs4all.nl>
To: Roland Clobus <rclobus@rclobus.nl>, Norbert Preining <norbert@preining.info>
Cc: dev-luatex@ntg.nl, 1009196@bugs.debian.org
Subject: Re: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Mon, 11 Apr 2022 17:29:11 +0200
On 4/11/2022 4:34 PM, Roland Clobus wrote:

> The texlive-binaries in Debian contain an embedded copy of Lua 5.3. The 
> Lua 5.4 version of luai_makeseed is more complex, see [2]. I'll write a 
> feature request for Lua later, that is out-of-scope for this scenario.
fyi: it is unlikely that luatex will move to 5.4 because it might break 
exisiting code and/or introduce incompatibilties (so we assume 5.3 for now)

Hans


-----------------------------------------------------------------
                                          Hans Hagen | PRAGMA ADE
              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
       tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------



Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Tue, 19 Apr 2022 07:30:02 GMT) (full text, mbox, link).


Acknowledgement sent to Roland Clobus <rclobus@rclobus.nl>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Tue, 19 Apr 2022 07:30:02 GMT) (full text, mbox, link).


Message #45 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: Roland Clobus <rclobus@rclobus.nl>
To: luatex development list <dev-luatex@ntg.nl>
Cc: 1009196@bugs.debian.org, Hans Hagen <j.hagen@xs4all.nl>, luigi scarso <luigi.scarso@gmail.com>
Subject: Re: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Tue, 19 Apr 2022 09:16:50 +0200
[Message part 1 (text/plain, inline)]
Hello list,

On 12/04/2022 08:44, Roland Clobus wrote:
> I'll follow-up soon with an updated patch.

As discussed, I've updated the patch.

For Lua-based TeX binaries, only when FORCE_SOURCE_DATE=1 and 
SOURCE_DATE_EPOCH are set, this will initialise the Lua seed to the 
value of SOURCE_DATE_EPOCH instead of a random value.
With this patch, the .fmt files can be generated bit-for-bit identical.

Regarding the patch:
* This patch is intended only for Lua 5.3 that is embedded in 
texlive-binaries
* A re-definition of `luai_makeseed` is unfortunately not sufficient for 
Lua 5.3, for 5.4.4 and later it would be. [1]
* I've added no validation for the content of SOURCE_DATE_EPOCH:
** 1) That happens in other code locations already
** 2) Even if the value would be incorrect, the Lua seed will still be 
de-randomized
* Do you want some comment lines?
* The sorting from by previous patch is no longer required. Only 
lstate.c needs to be modified.

With kind regards,
Roland Clobus


PS: If you later intend to upgrade to another version of Lua, the fixed 
seed value can help you in automated tests to see different behaviour 
due to the upgrade.

[1] 
https://github.com/lua/lua/commit/97e394ba1805fbe394a5704de660403901559e54
[lua_fixed_hash.patch (text/x-patch, attachment)]
[OpenPGP_signature (application/pgp-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Tue, 19 Apr 2022 07:57:03 GMT) (full text, mbox, link).


Acknowledgement sent to luigi scarso <luigi.scarso@gmail.com>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Tue, 19 Apr 2022 07:57:03 GMT) (full text, mbox, link).


Message #50 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: luigi scarso <luigi.scarso@gmail.com>
To: Roland Clobus <rclobus@rclobus.nl>
Cc: luatex development list <dev-luatex@ntg.nl>, 1009196@bugs.debian.org, Hans Hagen <j.hagen@xs4all.nl>
Subject: Re: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Tue, 19 Apr 2022 09:52:46 +0200
[Message part 1 (text/plain, inline)]
On Tue, Apr 19, 2022 at 9:16 AM Roland Clobus <rclobus@rclobus.nl> wrote:

> Hello list,
>
> On 12/04/2022 08:44, Roland Clobus wrote:
> > I'll follow-up soon with an updated patch.
>
> As discussed, I've updated the patch.
>
> For Lua-based TeX binaries, only when FORCE_SOURCE_DATE=1 and
> SOURCE_DATE_EPOCH are set, this will initialise the Lua seed to the
> value of SOURCE_DATE_EPOCH instead of a random value.
> With this patch, the .fmt files can be generated bit-for-bit identical.
>
> Regarding the patch:
> * This patch is intended only for Lua 5.3 that is embedded in
> texlive-binaries
> * A re-definition of `luai_makeseed` is unfortunately not sufficient for
> Lua 5.3, for 5.4.4 and later it would be. [1]
> * I've added no validation for the content of SOURCE_DATE_EPOCH:
> ** 1) That happens in other code locations already
> ** 2) Even if the value would be incorrect, the Lua seed will still be
> de-randomized
> * Do you want some comment lines?
> * The sorting from by previous patch is no longer required. Only
> lstate.c needs to be modified.
>
> With kind regards,
> Roland Clobus
>
>
> PS: If you later intend to upgrade to another version of Lua, the fixed
> seed value can help you in automated tests to see different behaviour
> due to the upgrade.
>
> [1]
> https://github.com/lua/lua/commit/97e394ba1805fbe394a5704de660403901559e54
>

Thank you very much for your patch, I will check it this weekend.

-- 
luigi
[Message part 2 (text/html, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Tue, 19 Apr 2022 09:21:03 GMT) (full text, mbox, link).


Acknowledgement sent to Roland Clobus <rclobus@rclobus.nl>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Tue, 19 Apr 2022 09:21:03 GMT) (full text, mbox, link).


Message #55 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: Roland Clobus <rclobus@rclobus.nl>
To: 1009196@bugs.debian.org, luatex development list <dev-luatex@ntg.nl>
Cc: Hans Hagen <j.hagen@xs4all.nl>, luigi scarso <luigi.scarso@gmail.com>
Subject: Re: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Tue, 19 Apr 2022 11:18:24 +0200
[Message part 1 (text/plain, inline)]
Hello list,

On 19/04/2022 09:52, luigi scarso wrote:
> Thank you very much for your patch, I will check it this weekend.

Another note:
While preparing for a generic change request for Lua, I found a mail by 
Hans Hagen [1], stating that all cases have been found in luatex. 
Sorting the table (as in my original patch) is also a solution, but my 
proposed patch in lstate.c will fix the root cause.

I would rather fix the root cause.
If you prefer the sorting patch, I'll adapt it to activate only when 
FORCE_SOURCE_DATE=1 and SOURCE_DATE_EPOCH are set.

With kind regards,
Roland Clobus

[1] http://lua-users.org/lists/lua-l/2014-07/msg00564.html
[OpenPGP_signature (application/pgp-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Wed, 04 May 2022 13:12:03 GMT) (full text, mbox, link).


Acknowledgement sent to Roland Clobus <rclobus@rclobus.nl>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Wed, 04 May 2022 13:12:03 GMT) (full text, mbox, link).


Message #60 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: Roland Clobus <rclobus@rclobus.nl>
To: luatex development list <dev-luatex@ntg.nl>, 1009196@bugs.debian.org
Subject: Re: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Wed, 4 May 2022 15:03:42 +0200
[Message part 1 (text/plain, inline)]
Hello luigi, list,

> On 19/04/2022 09:52, luigi scarso wrote: >> Thank you very much for your patch, I will check it this weekend.
Have you found the time already to review my patch? [1]

With kind regards,
Roland Clobus

[1] https://mailman.ntg.nl/pipermail/dev-luatex/2022-April/006659.html
[OpenPGP_signature (application/pgp-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Wed, 04 May 2022 13:21:02 GMT) (full text, mbox, link).


Acknowledgement sent to luigi scarso <luigi.scarso@gmail.com>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Wed, 04 May 2022 13:21:03 GMT) (full text, mbox, link).


Message #65 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: luigi scarso <luigi.scarso@gmail.com>
To: Roland Clobus <rclobus@rclobus.nl>
Cc: luatex development list <dev-luatex@ntg.nl>, 1009196@bugs.debian.org
Subject: Re: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Wed, 4 May 2022 15:16:58 +0200
[Message part 1 (text/plain, inline)]
On Wed, May 4, 2022 at 3:09 PM Roland Clobus <rclobus@rclobus.nl> wrote:

> Hello luigi, list,
>
> > On 19/04/2022 09:52, luigi scarso wrote: >> Thank you very much for your
> patch, I will check it this weekend.
> Have you found the time already to review my patch? [1]
>

Yes, Hans and I are discussing.
If possible, I would like to use a --reproducible switch at the command
line.

-- 
luigi
[Message part 2 (text/html, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian TeX Task Force <debian-tex-maint@lists.debian.org>:
Bug#1009196; Package texlive-binaries. (Wed, 04 May 2022 16:03:03 GMT) (full text, mbox, link).


Acknowledgement sent to Roland Clobus <rclobus@rclobus.nl>:
Extra info received and forwarded to list. Copy sent to Debian TeX Task Force <debian-tex-maint@lists.debian.org>. (Wed, 04 May 2022 16:03:03 GMT) (full text, mbox, link).


Message #70 received at 1009196@bugs.debian.org (full text, mbox, reply):

From: Roland Clobus <rclobus@rclobus.nl>
To: luatex development list <dev-luatex@ntg.nl>
Cc: 1009196@bugs.debian.org
Subject: Re: [Dev-luatex] Bug#1009196: texlive-binaries: Reproducible content of .fmt files
Date: Wed, 4 May 2022 18:00:52 +0200
[Message part 1 (text/plain, inline)]
On 04/05/2022 15:16, luigi scarso wrote:
> On Wed, May 4, 2022 at 3:09 PM Roland Clobus <rclobus@rclobus.nl 
>> On 19/04/2022 09:52, luigi scarso wrote:
>>> Thank you very much for your patch, I will check it this weekend.
>>     Have you found the time already to review my patch? [1]

> Yes, Hans and I are discussing.
> If possible, I would like to use a --reproducible switch at the command 
> line.

Adding a commandline argument is sometimes proposed by the development 
teams, instead of using SOURCE_DATE_EPOCH. I would rather suggest to use 
SOURCE_DATE_EPOCH, which is already in the code base, instead of adding 
a new code path.

If you find the time, please read the documentation on SOURCE_DATE_EPOCH 
[1] and the page that mentions a checklist [2].

The short summary: SOURCE_DATE_EPOCH has been standardized and is 
primarily intended to be used by rebuilders of the binaries, not the 
developers or end-users.

In the past, when SOURCE_DATE_EPOCH was getting established, texlive 
additionally added FORCE_SOURCE_DATE=1. Nowadays, if it can be avoided, 
I would recommend to use only SOURCE_DATE_EPOCH.
See [3] for all uses of FORCE_SOURCE_DATE_ in Debian. As you can see, it 
is mainly used in several tests to ensure that packages have output that 
can be compared against a reference.

With kind regards,
Roland Clobus

[1] https://reproducible-builds.org/docs/source-date-epoch/
[2] 
https://wiki.debian.org/ReproducibleBuilds/StandardEnvironmentVariables#Checklist
[3] https://codesearch.debian.net/search?q=FORCE_SOURCE_DATE&literal=0
[OpenPGP_signature (application/pgp-signature, attachment)]

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Wed May 17 10:41:58 2023; Machine Name: buxtehude

Debian Bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

Copyright © 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson, 2005-2017 Don Armstrong, and many other contributors.