Debian Bug report logs - #920732
strip-nondeterminism does not normalize Unix ownership from zip archives or .epub files

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: strip-nondeterminism should normalize file ownerships in epubs too

Date: Mon, 28 Jan 2019 17:19:42 +0100

[Message part 1 (text/plain, inline)]

Package: strip-nondeterminism
Version: 1.1.0-1
Severity: wishlist

Hi,

debian-edu-doc 2.10.11 when tested on tests.r-b.o shows this:

│ │ │ ├── ./usr/share/doc/debian-edu-doc-da/debian-edu-buster-manual.epub
│ │ │ │ ├── bsdtar -tvf {}
│ │ │ │ │ @@ -1,54 +1,54 @@
│ │ │ │ │  drwxr-xr-x  0 0      0           0 Jan 20 13:47 epub/
│ │ │ │ │  drwxr-xr-x  0 0      0           0 Jan 20 13:47 epub/META-INF/
│ │ │ │ │ --rw-r--r--  0 1111   1111      255 Jan  1  1970 epub/META-INF/container.xml
│ │ │ │ │ +-rw-r--r--  0 2222   2222      255 Jan  1  1970 epub/META-INF/container.xml

etc

despite this code (probably too much) in src:debian-edu-doc/documentation/common/Makefile.common

build-epub:
        # build the English EPUB version
        echo "Creating epub for en"
        $(DBTOEPUB) $(name).xml
        unzip -q $(name).epub -d epub
        sed -i 's#_idm[0-9]\{14\}#_idm1234567#' epub/OEBPS/content.opf
        sed -i 's#_idm[0-9]\{14\}#_idm1234567#' epub/OEBPS/toc.ncx
        find epub -exec touch -d @0 {} \;
        rm $(name).epub
        zip -q -r -o $(name).epub epub/
        rm -rf epub/
        strip-nondeterminism -t zip $(name).epub
        # build all other EPUB versions
        -for LINGUA in $(LANGUAGES) ; do \
                echo "Creating epub for $$LINGUA"; \
                po4a --translate-only $(name).$$LINGUA.xml po4a.cfg ; \
                mkdir images-tmp ; \
                cp images/*.* images-tmp/ ; \
                if [ -e images/$$LINGUA ] ; then \
                    cp -v images/$$LINGUA/*.* images-tmp/ ; \
                fi ; \
                sed -i "s#./images#./images-tmp#g" $(name).$$LINGUA.xml ; \
                $(DBTOEPUB) $(name).$$LINGUA.xml ; \
                sed -i "s#./images-tmp#./images/#g" $(name).$$LINGUA.xml ; \
                rm -rf images-tmp/ ; \
        done

IOW: we call strip-nondeterminism already and it would be nice if it would
normalize file ownerships.

And this 'find epub -exec touch -d @0 {} \;' also looks like something
strip-nd should do. Please close and retitle this bug report
accordingly.


-- 
tschüß,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

[signature.asc (application/pgp-signature, inline)]

Message #12 received at 920732@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 920732@bugs.debian.org, Holger Levsen <holger@layer-acht.org>

Subject: Re: Bug#920732: strip-nondeterminism should normalize file ownerships in epubs too

Date: Tue, 29 Jan 2019 15:48:02 +0100

Hi Holger,

> │ │ │ │ │ --rw-r--r--  0 1111   1111      255 Jan  1  1970 epub/META
> │ │ │ │ │ +-rw-r--r--  0 2222   2222      255 Jan  1  1970 epub/META

Hm, I can confirm this in Git (currently 3f4ba2fb8). Indeed, it
appears to be already handled with:

    my $uid_len = ord(substr($field, $pos + 5, 1));
    my $gid_len = ord(substr($field, $pos + 6 + $uid_len, 1));
    $result
      .= pack("CCx${uid_len}Cx${gid_len}", 1, $uid_len, $gid_len);
      
… and this code is run, padding with enough nulls (the "x" format
identifier for Perl's pack[0] method). However, it does not seem to
result in a normalised file when shown with bsdtar -tvf.

(I would immediately commit an "xfailing" test but the .epub in
question is 800K so I will hold off on that until I find a smaller
testcase. Do let me know if you have one.)

  [0] http://perldoc.perl.org/functions/pack.html
  

> [..] this 'find epub -exec touch -d @0 {} \;' also looks like
> something strip-nondeterminism should do.

We normalise the timestamps of .zip/.epub members so this should
not be a problem.


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Message #17 received at 920732@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: 920732@bugs.debian.org

Subject: Re: strip-nondeterminism should normalise file ownerships in epubs too

Date: Tue, 12 Mar 2019 12:56:48 -0400

forwarded 920732 https://salsa.debian.org/reproducible-builds/strip-nondeterminism/issues/2
thanks

I've forwarded this upstream here:

  https://salsa.debian.org/reproducible-builds/strip-nondeterminism/issues/2


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #24 received at 920732@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: 858431@bugs.debian.org, 920732@bugs.debian.org, "Mike Swanson" <mikeonthecomputer@gmail.com>

Subject: Re: strip-nondeterminism does not normalize Unix ownership from zip archives or .epub files

Date: Wed, 24 Apr 2019 11:38:14 -0400

forcemerge 858431 920732
retitle 858431 strip-nondeterminism does not normalize Unix ownership from zip archives or .epub files 
tags 858431 - wontfix
thanks

Two changes here:

 * .epub files are "just" .zip files, so merging and retitling to
    match.

 * Unmarking as wontfix; re-reading https://bugs.debian.org/858431#22,
   I believe we should indeed be normalising these.


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #45 received at 920732@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: 858431@bugs.debian.org, 920732@bugs.debian.org, "Holger Levsen" <holger@debian.org>, "Reiner Herrmann" <reiner@reiner-h.de>, "Santiago Vila" <sanvila@unex.es>

Cc: "Mike Swanson" <mikeonthecomputer@gmail.com>

Subject: Re: strip-nondeterminism does not normalize Unix ownership from zip archives or .epub files

Date: Sun, 28 Apr 2019 05:58:14 -0400

tags 858431 + pending
thanks

This is fixed in Git, pending upload:

  https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/f40f555085eeb086bfd4ee1fca1012550790a12d

  Makefile.PL                                  |  1 +
  lib/File/StripNondeterminism/handlers/zip.pm | 31 +++++++++++++++++++++++-----
  2 files changed, 27 insertions(+), 5 deletions(-)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #46 received at 858431-close@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 858431-close@bugs.debian.org

Subject: Bug#858431: fixed in strip-nondeterminism 1.1.3-1

Date: Sun, 28 Apr 2019 16:51:07 +0000

Source: strip-nondeterminism
Source-Version: 1.1.3-1

We believe that the bug you reported is fixed in the latest version of
strip-nondeterminism, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 858431@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Chris Lamb <lamby@debian.org> (supplier of updated strip-nondeterminism package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Sun, 28 Apr 2019 17:30:27 +0100
Source: strip-nondeterminism
Binary: dh-strip-nondeterminism libfile-stripnondeterminism-perl strip-nondeterminism
Architecture: source all
Version: 1.1.3-1
Distribution: experimental
Urgency: medium
Maintainer: Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>
Changed-By: Chris Lamb <lamby@debian.org>
Description:
 dh-strip-nondeterminism - file non-deterministic information stripper — Debhelper add-on
 libfile-stripnondeterminism-perl - file non-deterministic information stripper — Perl module
 strip-nondeterminism - file non-deterministic information stripper — stand-alone tool
Closes: 858431
Changes:
 strip-nondeterminism (1.1.3-1) experimental; urgency=medium
 .
   * Workaround Archive::Zip's incorrect handling of the localExtraField field
     by monkey-patching the accessor methods to always return normalised values.
     This fixes the normalisation of Unix ownership (uid/gid) within .zip
     archives, .epub files, etc.
     (Closes: #858431, reproducible-builds/strip-nondeterminism#4)
   * Check the return status from Archive::Zip when writing file to disk.
   * Catch an edgecase where/if we can't parse the provided length of an
     invalid field within .zip files.
Checksums-Sha1:
 6bc7b28c612301c2f1b65b7dd49aaaa0f096db1d 2555 strip-nondeterminism_1.1.3-1.dsc
 5f6f26842246ff130cba64152f6f091371035186 221786 strip-nondeterminism_1.1.3.orig.tar.bz2
 0cb0d7b371c3e8494058e7c3df1075a51d531175 31404 strip-nondeterminism_1.1.3-1.debian.tar.xz
 147b304fefd0b7e0011fb69d15205dafa458f177 13256 dh-strip-nondeterminism_1.1.3-1_all.deb
 15ae1f2565a48470121b7bf95edf736114034bdf 20424 libfile-stripnondeterminism-perl_1.1.3-1_all.deb
 d083865dac3523a7910c5c46df4630074b9d751c 13376 strip-nondeterminism_1.1.3-1_all.deb
 d5e40a525b2dcfbd4beda2e505b709bf8db0aee7 6362 strip-nondeterminism_1.1.3-1_amd64.buildinfo
Checksums-Sha256:
 f1c71f268d14960a170f0d4657e9b323fcb0a0f9370f0887804b7111905c8179 2555 strip-nondeterminism_1.1.3-1.dsc
 aade3e2855761824fb069ae773f344216b0285fc4cb118edea3eb32e888c955b 221786 strip-nondeterminism_1.1.3.orig.tar.bz2
 ad87b5d34ce0ba1463cfa5624fc4a29e0c52d4a241cb7018ab7f9dda9055cba5 31404 strip-nondeterminism_1.1.3-1.debian.tar.xz
 1411bba9541e0af554406d9512f9a5813f60a2280535c850a55a9fb707ada954 13256 dh-strip-nondeterminism_1.1.3-1_all.deb
 a7c9b3ea512cf7d2eb9928233edbac81c25f679e2a3a3e65aeb02411a4daec15 20424 libfile-stripnondeterminism-perl_1.1.3-1_all.deb
 70cd232e3b5e49e79d124a1a481443850a9f9bc3e2f7c1df09583f3cbf56312e 13376 strip-nondeterminism_1.1.3-1_all.deb
 30daaf0a6534b6e8a8843e76683e5c070aa11f83bbec06dcaeea661c77ef6f93 6362 strip-nondeterminism_1.1.3-1_amd64.buildinfo
Files:
 ec00328f67673602accf86dabdba9991 2555 devel optional strip-nondeterminism_1.1.3-1.dsc
 db196b77910287bfc2ec430168c9ad70 221786 devel optional strip-nondeterminism_1.1.3.orig.tar.bz2
 fd5422104cdde964cb532f0dedfbadd8 31404 devel optional strip-nondeterminism_1.1.3-1.debian.tar.xz
 0bf28ba05165008e21ef9193c510329e 13256 devel optional dh-strip-nondeterminism_1.1.3-1_all.deb
 71230b68b6badd1628861e7691256459 20424 perl optional libfile-stripnondeterminism-perl_1.1.3-1_all.deb
 cd86d4d82d1116317d15651e352915fb 13376 devel optional strip-nondeterminism_1.1.3-1_all.deb
 17a276ba3530462f61b365a91bb97853 6362 devel optional strip-nondeterminism_1.1.3-1_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEwv5L0nHBObhsUz5GHpU+J9QxHlgFAlzF1ZUACgkQHpU+J9Qx
HliAMw/+Ono4OGc2DHmAOnWBneXmScZ12PnVhcJPFSny/QRB7lA+ko7RVHm8KSPR
Qy2UoJEsw7itpWt+cizxz/kJELdvTEA1pq7/lBNgYq2eSszi+bQHP++Wd51u3ILr
LHpZG3i8i0lAdsruhAl//J/7Z/1oKn5qcXzBiu6iQz1oYLaMOdQdeLXRw9IsACIR
6boZmv3Mag8yomO5D1sxWRPNWToNkMa4Nkr0AcksfvOlPPkZPkvnO5ipRoQEU5aJ
3jLjhsq3EnyaQ1vUws7/Fji7OtpBZKUXkDFFUjeEyxigpDmbA3lOzGHYsXgQhk1t
IV8WnweX4mCwX428yjVvvg50+k1ZlKRPbH9f80D8CFrAbKYzXo3Xch+sywLFC1Zr
LlJLiR95xQA6DSnEJQt5BWL/1RPo99wb9EpQZDqPf6/zDVva+RzH/dN0UMQS1EqA
ltauil3xuP9f0b4cZZC/ksn9kk8Gb9vK2fxwnpPPGg5yVSjtQNUIyPwrkXmjzE9I
x8Bp9cZP2IKrOpCdjUyPsApm+KGs3bHxuFJMICiUas6A+xQ9NWvU061W3QJ4+dEr
cuMzjIIEhwYHe3g45hstjcqShn2SRwUsEUp1Gks2HB6JJZ6eXAzh0UW8vy1+IPVi
qGgXOs1GwRbKDa3as2Mxjg1SxxhqENSqdS0GLj8cId90m4Gr284=
=lenf
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.