#859103 - strip-nondeterminism: does not replace all timestamps in zip archives

Date: Thu, 30 Mar 2017 11:09:04 UTC

Severity: normal

Tags: fixed-upstream

Found in versions strip-nondeterminism/0.032-1, strip-nondeterminism/0.034-1

Forwarded to https://salsa.debian.org/reproducible-builds/strip-nondeterminism/issues/5

Report forwarded to debian-bugs-dist@lists.debian.org, Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>:
Bug#859103; Package strip-nondeterminism. (Thu, 30 Mar 2017 11:09:06 GMT) (full text, mbox, link).

Acknowledgement sent to beuc@beuc.net:
New Bug report received and forwarded. Copy sent to Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>. (Thu, 30 Mar 2017 11:09:06 GMT) (full text, mbox, link).

Package: strip-nondeterminism Version: 0.032-1 Hi, It seems strip-nondeterminism replaces correctly replaces timestamps in Unix-specific extra fields, however this may only happen in the central directory and not in the individual members themselves. reprotest - using a 0xAAAA timestamp (2004-09-10) for clarity: $ reprotest 'mkdir test; touch test/a test/b; zip -r test.zip test; strip-nondeterminism -T 1094795585 test.zip; chmod 644 test.zip; cp test.zip /tmp; sleep 2' 'test.zip' STARTING VIRTUAL SERVER ['/usr/lib/python3/dist-packages/reprotest/virt/null'] reprotest [12:29:15]: version @version@ reprotest [12:29:15]: host decision; command line: /usr/bin/reprotest 'mkdir test; touch test/a test/b; zip -r test.zip test; strip-nondeterminism -T 1094795585 test.zip; chmod 644 test.zip; cp test.zip /tmp; sleep 2' test.zip reprotest [12:29:15]: testbed dpkg architecture: amd64 reprotest [12:29:15]: testbed running kernel: Linux 4.9.0-1-amd64 #1 SMP Debian 4.9.6-3 (2017-01-28) will vary: environment will vary: fileordering will vary: home will vary: kernel will vary: locales will vary: exec_path will vary: time will vary: timezone will vary: umask copying /home/me/workdir/test/ over to virtual server's /tmp/autopkgtest.L1evFW/control/ copying /home/me/workdir/test/ over to virtual server's /tmp/autopkgtest.L1evFW/experiment/ starting build with source directory: /tmp/autopkgtest.L1evFW/control/, artifact pattern: test.zip executing: ( umask 0022 && cd /tmp/autopkgtest.L1evFW/control/ && linux64 --uname-2.6 sh -ec 'mkdir test; touch test/a test/b; zip -r test.zip test; strip-nondeterminism -T 1094795585 test.zip; chmod 644 test.zip; cp test.zip /tmp; sleep 2' ) adding: test/ (stored 0%) adding: test/a (stored 0%) adding: test/b (stored 0%) starting build with source directory: /tmp/autopkgtest.L1evFW/experiment/, artifact pattern: test.zip executing: if ( mv /tmp/autopkgtest.L1evFW/experiment/ /tmp/autopkgtest.L1evFW/experiment-before-disorderfs/ && mkdir -p /tmp/autopkgtest.L1evFW/experiment/ && sh -ec 'disorderfs --shuffle-dirents=yes --multi-user="$(if [ $(id -u) = 0 ]; then echo yes; else echo no; fi)" "$@"' - /tmp/autopkgtest.L1evFW/experiment-before-disorderfs/ /tmp/autopkgtest.L1evFW/experiment/ && umask 0002 && cd /tmp/autopkgtest.L1evFW/experiment/ && faketime +373days+7hours+13minutes linux32 sh -ec 'mkdir test; touch test/a test/b; zip -r test.zip test; strip-nondeterminism -T 1094795585 test.zip; chmod 644 test.zip; cp test.zip /tmp; sleep 2' ); then ( __c=0; fusermount -u /tmp/autopkgtest.L1evFW/experiment/ || __c=$?; rmdir /tmp/autopkgtest.L1evFW/experiment/ || __c=$?; mv /tmp/autopkgtest.L1evFW/experiment-before-disorderfs/ /tmp/autopkgtest.L1evFW/experiment/ || __c=$?; exit $__c; ); else __x=$?; if ( __c=0; fusermount -u /tmp/autopkgtest.L1evFW/experiment/ || __c=$?; rmdir /tmp/autopkgtest.L1evFW/experiment/ || __c=$?; mv /tmp/autopkgtest.L1evFW/experiment-before-disorderfs/ /tmp/autopkgtest.L1evFW/experiment/ || __c=$?; exit $__c; ); then exit $__x; else echo >&2; "cleanup failed with exit code $?"; exit $__x; fi; fi disorderfs: shuffling dirents disorderfs: reversing dirents adding: test/ (stored 0%) adding: test/b (stored 0%) adding: test/a (stored 0%) perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = "fr_CH.UTF-8:fr", LC_ALL = "fr_CH.UTF-8", LANG = "fr_CH.UTF-8" are supported and installed on your system. perl: warning: Falling back to the standard locale ("C"). copying /tmp/autopkgtest.L1evFW/control-dist/ back from virtual server's /tmp/tmpnnnyuzyg/control_artifact/ copying /tmp/autopkgtest.L1evFW/experiment-dist/ back from virtual server's /tmp/tmpnnnyuzyg/experiment_artifact/ Running diffoscope: ['diffoscope', '/tmp/tmpnnnyuzyg/control_artifact/', '/tmp/tmpnnnyuzyg/experiment_artifact/'] --- /tmp/tmpnnnyuzyg/control_artifact/ +++ /tmp/tmpnnnyuzyg/experiment_artifact/ │ --- /tmp/tmpnnnyuzyg/control_artifact/test.zip ├── +++ /tmp/tmpnnnyuzyg/experiment_artifact/test.zip │┄ No file format specific differences found inside, yet data differs (Zip archive data, at least v1.0 to extract) │ @@ -1,18 +1,18 @@ │ 00000000: 504b 0304 0a00 0000 0000 a22e 2a31 0000 PK..........*1.. │ 00000010: 0000 0000 0000 0000 0000 0500 1c00 7465 ..............te │ 00000020: 7374 2f55 5409 0003 4141 4141 4141 4141 st/UT...AAAAAAAA │ 00000030: 7578 0b00 0104 0000 0000 0400 0000 0050 ux.............P │ 00000040: 4b03 040a 0000 0000 00a2 2e2a 3100 0000 K..........*1... │ 00000050: 0000 0000 0000 0000 0006 001c 0074 6573 .............tes │ -00000060: 742f 6155 5409 0003 fbdd dc58 fbdd dc58 t/aUT......X...X │ +00000060: 742f 6155 5409 0003 fddd dc58 fddd dc58 t/aUT......X...X │ 00000070: 7578 0b00 0104 e803 0000 04e8 0300 0050 ux.............P │ 00000080: 4b03 040a 0000 0000 00a2 2e2a 3100 0000 K..........*1... │ 00000090: 0000 0000 0000 0000 0006 001c 0074 6573 .............tes │ -000000a0: 742f 6255 5409 0003 fbdd dc58 fbdd dc58 t/bUT......X...X │ +000000a0: 742f 6255 5409 0003 fddd dc58 fddd dc58 t/bUT......X...X │ 000000b0: 7578 0b00 0104 e803 0000 04e8 0300 0050 ux.............P │ 000000c0: 4b01 021e 030a 0000 0000 00a2 2e2a 3100 K............*1. │ 000000d0: 0000 0000 0000 0000 0000 0005 0018 0000 ................ │ 000000e0: 0000 0000 0010 00ed 4100 0000 0074 6573 ........A....tes │ 000000f0: 742f 5554 0500 0341 4141 4175 780b 0001 t/UT...AAAAux... │ 00000100: 0400 0000 0004 0000 0000 504b 0102 1e03 ..........PK.... │ 00000110: 0a00 0000 0000 a22e 2a31 0000 0000 0000 ........*1...... 5 ouf of 9 timestamps were replaced, 4 of them are remaining as confirmed by bsdtar: $ bsdtar -tvf /tmp/test.zip drwxr-xr-x 0 0 0 0 sept. 10 2004 test/ -rw-r--r-- 0 1000 1000 0 mars 30 12:29 test/a -rw-r--r-- 0 1000 1000 0 mars 30 12:29 test/b I couldn't precisely locate where the error is in the File/StripNondeterminism/handlers/zip.pm - maybe removing and adding back a zip member only resets the directory extra fields. Cheers! Sylvain

Information forwarded to debian-bugs-dist@lists.debian.org, Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>:
Bug#859103; Package strip-nondeterminism. (Tue, 27 Mar 2018 23:12:04 GMT) (full text, mbox, link).

Acknowledgement sent to Benjamin Moody <benjamin.moody@gmail.com>:
Extra info received and forwarded to list. Copy sent to Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>. (Tue, 27 Mar 2018 23:12:04 GMT) (full text, mbox, link).

Package: strip-nondeterminism Version: 0.034-1 Followup-For: Bug #859103 This is especially annoying because the "local extra field" includes the file *access* time: $ rm -f foo 1.zip 2.zip $ touch -d 2015-01-01 foo $ zip 1.zip foo $ zip 2.zip foo $ strip-nondeterminism 1.zip 2.zip $ diffoscope 1.zip 2.zip ... 00000010: 0000 0000 0000 0000 0000 0300 1c00 666f ..............fo -00000020: 6f55 5409 0003 50d4 a454 50d4 a454 7578 oUT...P..TP..Tux +00000020: 6f55 5409 0003 50d4 a454 62a1 ba5a 7578 oUT...P..Tb..Zux 00000030: 0b00 0104 e803 0000 04e8 0300 0050 4b01 .............PK. ... (Which makes me think, for testing build reproducibility, it'd be wise to try one build using noatime and another using strictatime. But anyway...) The problem here is that Archive::Zip::ZipFileMember is not designed to allow modifying the localExtraField() at all. From the man page: localExtraField( [ $newField ] ) localExtraField( [ { field => $newField } ] ) Gets or sets the extra field that was read from the local header. This is not set for a member from a zip file until after the member has been written out. The extra field must be in the proper format. That is to say, before calling $zip->overwrite(), localExtraField() returns an empty string. Moreover, for ZipFileMembers, manually setting the field has no effect - even if you call overwrite(), then go back and modify the localExtraFields, then call overwrite() again, it will re-read the fields from the zip file. (As far as I can tell, the *only* way to manually specify a localExtraField using Archive::Zip is to decompress and recompress each member.) Here is a rather kludgy patch to make Archive::Zip behave the way that strip-nondeterminism seems to expect: --- /usr/share/perl5/Archive/Zip/ZipFileMember.pm +++ Archive/Zip/ZipFileMember.pm @@ -43,6 +43,25 @@ and $self->uncompressedSize == 0); } +sub localExtraField { + my $self = shift; + + # If this function is called with an argument, it overrides the + # original field contents from the source archive. + if (@_) { + $self->{'_localExtraFieldUserDefined'} = 1; + } + # Otherwise, the value is loaded lazily, the first time it is needed. + elsif (!defined $self->{'_localExtraFieldUserDefined'} + and defined $self->{'externalFileName'}) { + my $origpos = $self->fh()->tell(); + $self->rewindData(); + $self->fh()->seek($origpos, IO::Seekable::SEEK_SET); + } + + return $self->SUPER::localExtraField(@_); +} + # Seek to the beginning of the local header, just past the signature. # Verify that the local header signature is in fact correct. # Update the localHeaderRelativeOffset if necessary by adding the possibleEocdOffset. @@ -156,10 +175,17 @@ } if ($extraFieldLength) { - $bytesRead = - $self->fh()->read($self->{'localExtraField'}, $extraFieldLength); - if ($bytesRead != $extraFieldLength) { - return _ioError("reading local extra field"); + if ($self->{'_localExtraFieldUserDefined'}) { + $self->fh()->seek($extraFieldLength, IO::Seekable::SEEK_CUR) + or return _ioError("skipping local extra field"); + } + else { + $bytesRead = + $self->fh()->read($self->{'localExtraField'}, $extraFieldLength); + if ($bytesRead != $extraFieldLength) { + return _ioError("reading local extra field"); + } + $self->{'_localExtraFieldUserDefined'} = 0; } } Here is a different kludgy approach to work around the issue in strip-nondeterminism, rewriting the local file headers by hand: --- /usr/share/perl5/File/StripNondeterminism/handlers/zip.pm +++ File/StripNondeterminism/handlers/zip.pm @@ -23,6 +23,7 @@ use File::Temp; use Archive::Zip qw/:CONSTANTS :ERROR_CODES/; +use Fcntl q/SEEK_SET/; # A magic number from Archive::Zip for the earliest timestamp that # can be represented by a Zip file. From the Archive::Zip source: @@ -207,11 +208,36 @@ } $member->cdExtraField( normalize_extra_fields($member->cdExtraField(), CENTRAL_HEADER)); - $member->localExtraField( - normalize_extra_fields($member->localExtraField(), LOCAL_HEADER)); } my $old_perms = (stat($zip_filename))[2] & oct(7777); $zip->overwrite(); + + # Archive::Zip::ZipFileMembers do not allow modifying the + # local extra field, so we need to rewrite the local file + # headers by hand. This assumes that normalize_extra_fields + # does not change the length of the field(s). + + open(my $fh, '+<', $zip_filename) or die "Unable to open $zip_filename: $!"; + for my $member ($zip->members()) { + my $extra_field = normalize_extra_fields($member->localExtraField(), LOCAL_HEADER); + my $offset = $member->writeLocalHeaderRelativeOffset(); + my ($header, $signature, $namelength, $extralength); + if (seek($fh, $offset, SEEK_SET) and read($fh, $header, 30) == 30) { + $signature = unpack("V", substr($header, 0, 4)); + $namelength = unpack("v", substr($header, 26, 2)); + $extralength = unpack("v", substr($header, 28, 2)); + } + if ($signature == 0x04034b50 + and $extralength == length($extra_field) + and seek($fh, $offset + 30 + $namelength, SEEK_SET)) { + syswrite($fh, $extra_field); + } + else { + die "Cannot find local file header(s) in $zip_filename"; + } + } + close($fh) or die "Error writing $zip_filename: $!"; + chmod($old_perms, $zip_filename); return 1; } -- System Information: Debian Release: 9.4 APT prefers stable APT policy: (990, 'stable'), (500, 'stable-updates'), (500, 'stable-debug') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 4.9.0-6-amd64 (SMP w/40 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages strip-nondeterminism depends on: ii libfile-stripnondeterminism-perl 0.034-1 ii perl 5.24.1-3+deb9u2 strip-nondeterminism recommends no packages. strip-nondeterminism suggests no packages. -- no debconf information

Acknowledgement sent to "Chris Lamb" <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>. (Tue, 12 Mar 2019 17:00:08 GMT) (full text, mbox, link).

forwarded 859103 https://salsa.debian.org/reproducible-builds/strip-nondeterminism/issues/5 thanks I've forwarded this upstream here: https://salsa.debian.org/reproducible-builds/strip-nondeterminism/issues/5 Regards, -- ,''`. : :' : Chris Lamb `. `'` lamby@debian.org / chris-lamb.co.uk `-

Set Bug forwarded-to-address to 'https://salsa.debian.org/reproducible-builds/strip-nondeterminism/issues/5'. Request was from "Chris Lamb" <lamby@debian.org> to control@bugs.debian.org. (Tue, 12 Mar 2019 17:00:15 GMT) (full text, mbox, link).

Added tag(s) fixed-upstream. Request was from debian-bts-link@lists.debian.org to control@bugs.debian.org. (Thu, 18 Jul 2019 19:51:08 GMT) (full text, mbox, link).

Debian Bug report logs - #859103 strip-nondeterminism: does not replace all timestamps in zip archives

Debian Bug report logs - #859103
strip-nondeterminism: does not replace all timestamps in zip archives