Debian Bug report logs - #977177
mm-common: reproducible builds: Generated tarball includes user, group and file mode

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: submit@bugs.debian.org

Subject: mm-common: reproducible builds: Generated tarball includes user, group and file mode

Date: Fri, 11 Dec 2020 20:45:09 -0800

[Message part 1 (text/plain, inline)]

Source: mm-common
Severity: normal
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: username
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org

The generated tarball /usr/share/doc/mm-common/skeletonmm.tar.xz
includes user, uid, group, gid and file mode information that may
vary depending on the build environment:

  https://tests.reproducible-builds.org/debian/rb-pkg/bullseye/amd64/diffoscope-results/mm-common.html

  1 -rw-r--r--···0·pbuilder1··(1111)·pbuilder1··(1111)·······60·2020-09-25·11:54:04.045053·skeletonmm/.gitignore
  1 -rw-rw-r--···0·pbuilder2··(2222)·pbuilder2··(2222)·······60·2020-09-25·11:54:04.045053·skeletonmm/.gitignore


The attached patch fixes this by setting these values consistently
between builds in the python code used to generate the tarball.

If anyone has a better handle on python's tarfile mode handling code, it
might be worth taking a closer look. I'm not entirely sure how the file
modes work in this code (they don't appear to use modes similar to those
used by umask, chmod or python's file functions)... but after some trial
and error and print debugging this seems to work to reproducibly produce
the tarball in mm-common without loosing the execute bit on the one file
that has it.


Thanks for maintaining mm-common!


live well,
  vagrant

[0001-Set-uid-username-gid-group-name-and-mode-on-files-in.patch (text/x-diff, inline)]

From 024c121c844a4ec920133eb3f7e6b6ee8044c0b6 Mon Sep 17 00:00:00 2001
From: Vagrant Cascadian <vagrant@reproducible-builds.org>
Date: Sat, 12 Dec 2020 04:05:56 +0000
Subject: [PATCH] Set uid, username, gid, group name and mode on files in
 generated tarball.

The user, group and file mode may otherwise vary between builds on
different systems.

https://tests.reproducible-builds.org/debian/issues/unstable/users_and_groups_in_tarball_issue.html
https://tests.reproducible-builds.org/debian/issues/unstable/different_due_to_umask_issue.html
---
 util/meson_aux/skeletonmm-tarball.py | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/util/meson_aux/skeletonmm-tarball.py b/util/meson_aux/skeletonmm-tarball.py
index db9e650..89049b6 100755
--- a/util/meson_aux/skeletonmm-tarball.py
+++ b/util/meson_aux/skeletonmm-tarball.py
@@ -39,10 +39,24 @@ elif output_file.endswith('.gz'):
 else:
   mode = 'w'
 
+def reproducible(tarinfo):
+  # Set consistent user, group and mode on files in the tar archive
+  tarinfo.uid = 0
+  tarinfo.uname = 'root'
+  tarinfo.gid = 0
+  tarinfo.gname = 'root'
+  if str(tarinfo.mode).startswith('332'):
+    # executable... should be u=rwx,g=rx,o=rx
+    tarinfo.mode = 33261
+  else:
+    # regular... should be u=rw,g=r,o=r
+    tarinfo.mode = 33188
+  return tarinfo
+
 with tarfile.open(output_file, mode=mode) as tar_file:
   os.chdir(source_dir) # Input filenames are relative to source_dir.
   for file in sys.argv[3:]:
-    tar_file.add(file)
+    tar_file.add(file, filter=reproducible)
 # Errors raise exceptions. If an exception is raised, Meson+ninja will notice
 # that the command failed, despite exit(0).
 sys.exit(0)
-- 
2.29.2

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 977177@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Vagrant Cascadian <vagrant@reproducible-builds.org>, 977177@bugs.debian.org

Subject: Re: Bug#977177: mm-common: reproducible builds: Generated tarball includes user, group and file mode

Date: Sat, 12 Dec 2020 11:30:22 +0000

On Fri, 11 Dec 2020 at 20:45:09 -0800, Vagrant Cascadian wrote:
> If anyone has a better handle on python's tarfile mode handling code, it
> might be worth taking a closer look. I'm not entirely sure how the file
> modes work in this code (they don't appear to use modes similar to those
> used by umask, chmod or python's file functions)

It looks like they're encoded in the same way as st_mode in a struct
stat_buf: the low bits are Unix permissions (which start making sense
if you print them in octal) and the high bits are file type. See the
documentation for the stat Python module, and in particular stat.S_IMODE
and stat.S_IFMT.

I think the correct normalization would be something like this (untested!):

    if tarinfo.isdir() or (tarinfo.mode & 0o111) != 0:
        tarinfo.mode = stat.S_IFMT(tarinfo.mode) | 0o755
    else:
        tarinfo.mode = stat.S_IFMT(tarinfo.mode) | 0o644

(that's the same as chmod a+rX,og-w).

    smcv

Message #15 received at 977177@bugs.debian.org (full text, mbox, reply):

From: Vagrant Cascadian <vagrant@reproducible-builds.org>

To: 977177@bugs.debian.org

Cc: control@bugs.debian.org

Subject: Re: Mistakes were made ... klayout: reproducible builds: Binaries contain build timestamps

Date: Mon, 14 Dec 2020 14:33:46 -0800

[Message part 1 (text/plain, inline)]

user reproducible-builds@lists.alioth.debian.org
usertags 977412 - timestamps
thanks

Oops, meant for the klayout patch to go to a new bug report, sorry for
the noise!

live well,
  vagrant

[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.