Debian Bug report logs - #903401
diffoscope fails on deb archives that contain non-compressed data.tar

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Roderich Schupp <roderich.schupp@gmail.com>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: diffoscope fails on deb archives that contain non-compressed data.tar

Date: Mon, 09 Jul 2018 15:54:39 +0200

Package: diffoscope
Version: 99
Severity: normal

Following up #903391 let's try the same with uncompressed data.tar (using again
modified tests/data/test{1,2}.deb for diffoscope source and dc9ee98bdfce7d1a
from git):

$ diffoscope non-compressed-data*.deb
...
2018-07-09 13:51:51 D: diffoscope.comparators.utils.libarchive: Extracted 4
entries from /tmp/diffoscope_a9dq4e2h/tmppp059aii/0/2.tar to
/tmp/diffoscope_a9dq4e2h/tmpk1so9k_u
2018-07-09 13:51:51 D: diffoscope.comparators.utils.archive: Unpacking
./usr/share/doc/test/copyright from data.tar
2018-07-09 13:51:51 D: diffoscope.comparators.utils.archive: Unpacking
./usr/share/doc/test/README.Debian from data.tar
2018-07-09 13:51:51 D: diffoscope.comparators.utils.archive: Unpacking
./usr/share/doc/test/text from data.tar
2018-07-09 13:51:51 D: diffoscope.comparators.utils.compare: Comparing
./usr/share/doc/test/changelog.gz (LibarchiveMember) and
./usr/share/doc/test/changelog.gz (LibarchiveMember)
2018-07-09 13:51:51 D: diffoscope.comparators.utils.file:
File.has_same_content: <<class
'diffoscope.comparators.utils.libarchive.LibarchiveMember'>
./usr/share/doc/test/changelog.gz> <<class
'diffoscope.comparators.utils.libarchive.LibarchiveMember'>
./usr/share/doc/test/changelog.gz>
2018-07-09 13:51:51 D: diffoscope.comparators.utils.specialize: Using
DebDataTarFile for ./usr/share/doc/test/changelog.gz
2018-07-09 13:51:51 D: diffoscope.comparators.utils.specialize: Using
DebDataTarFile for ./usr/share/doc/test/changelog.gz
2018-07-09 13:51:51 D: diffoscope.diff: Running diff -aU7
/tmp/diffoscope_a9dq4e2h/tmph5lsxwjh/fifo1
/tmp/diffoscope_a9dq4e2h/tmph5lsxwjh/fifo2
2018-07-09 13:51:51 D: diffoscope.diff: diff -aU7
/tmp/diffoscope_a9dq4e2h/tmph5lsxwjh/fifo1
/tmp/diffoscope_a9dq4e2h/tmph5lsxwjh/fifo2: returncode 0, parsed True
Traceback (most recent call last):
  File "/build/work/diffoscope/diffoscope-99/diffoscope/diff.py", line 327, in
diff
    return run_diff(fifo1_path, fifo2_path, fifo1.end_nl_q, fifo2.end_nl_q)
  File "/build/work/diffoscope/diffoscope-99/diffoscope/diff.py", line 220, in
__exit__
    self.join()
  File "/build/work/diffoscope/diffoscope-99/diffoscope/diff.py", line 253, in
join
    raise self._exception
  File "/build/work/diffoscope/diffoscope-99/diffoscope/diff.py", line 244, in
run
    end_nl = self.feeder(fifo)
  File "/build/work/diffoscope/diffoscope-99/diffoscope/feeders.py", line 45,
in feeder
    for buf in in_file:
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/libarchive.py",
line 99, in list_libarchive
    with libarchive.file_reader(path) as archive:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/libarchive/read.py", line 73, in
file_reader
    ffi.read_open_filename_w(archive_p, path, block_size)
  File "/usr/lib/python3/dist-packages/libarchive/ffi.py", line 85, in
check_int
    raise archive_error(args[0], retcode)
  File "/usr/lib/python3/dist-packages/libarchive/ffi.py", line 69, in
archive_error
    raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
libarchive.exception.ArchiveError: Unrecognized archive format (errno=84,
retcode=-30, archive_p=139857020076192)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/build/work/diffoscope/diffoscope-99/diffoscope/main.py", line 460, in
main
    sys.exit(run_diffoscope(parsed_args))
  File "/build/work/diffoscope/diffoscope-99/diffoscope/main.py", line 432, in
run_diffoscope
    difference = compare_root_paths(path1, path2)
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/compare.py",
line 68, in compare_root_paths
    difference = compare_files(file1, file2)
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/compare.py",
line 118, in compare_files
    return file1.compare(file2, source)
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/file.py",
line 366, in compare
    difference = self._compare_using_details(other, source)
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/file.py",
line 321, in _compare_using_details
    other.as_container, no_recurse=no_recurse))
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/container.py",
line 177, in compare_pair
    file1, file2, source=None, diff_content_only=no_recurse)
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/compare.py",
line 118, in compare_files
    return file1.compare(file2, source)
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/file.py",
line 366, in compare
    difference = self._compare_using_details(other, source)
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/file.py",
line 321, in _compare_using_details
    other.as_container, no_recurse=no_recurse))
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/container.py",
line 177, in compare_pair
    file1, file2, source=None, diff_content_only=no_recurse)
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/compare.py",
line 118, in compare_files
    return file1.compare(file2, source)
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/file.py",
line 366, in compare
    difference = self._compare_using_details(other, source)
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/file.py",
line 304, in _compare_using_details
    details.extend(self.compare_details(other, source))
  File "/build/work/diffoscope/diffoscope-99/diffoscope/comparators/deb.py",
line 217, in compare_details
    self.path, other.path, source="file list")]
  File "/build/work/diffoscope/diffoscope-99/diffoscope/difference.py", line
228, in from_text_readers
    **kwargs
  File "/build/work/diffoscope/diffoscope-99/diffoscope/difference.py", line
186, in from_feeder
    unified_diff = diff(feeder1, feeder2)
  File "/build/work/diffoscope/diffoscope-99/diffoscope/diff.py", line 327, in
diff
    return run_diff(fifo1_path, fifo2_path, fifo1.end_nl_q, fifo2.end_nl_q)
  File "/build/work/diffoscope/diffoscope-99/diffoscope/diff.py", line 220, in
__exit__
    self.join()
  File "/build/work/diffoscope/diffoscope-99/diffoscope/diff.py", line 253, in
join
    raise self._exception
  File "/build/work/diffoscope/diffoscope-99/diffoscope/diff.py", line 244, in
run
    end_nl = self.feeder(fifo)
  File "/build/work/diffoscope/diffoscope-99/diffoscope/feeders.py", line 45,
in feeder
    for buf in in_file:
  File
"/build/work/diffoscope/diffoscope-99/diffoscope/comparators/utils/libarchive.py",
line 99, in list_libarchive
    with libarchive.file_reader(path) as archive:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/libarchive/read.py", line 73, in
file_reader
    ffi.read_open_filename_w(archive_p, path, block_size)
  File "/usr/lib/python3/dist-packages/libarchive/ffi.py", line 85, in
check_int
    raise archive_error(args[0], retcode)
  File "/usr/lib/python3/dist-packages/libarchive/ffi.py", line 69, in
archive_error
    raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
libarchive.exception.ArchiveError: Unrecognized archive format (errno=84,
retcode=-30, archive_p=139856885857072)



-- System Information:
Debian Release: buster/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'unstable'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.17.4 (SMP w/4 CPU cores)
Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8), LANGUAGE=en_US.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  libpython3.6-stdlib    3.6.6-1
ii  python3                3.6.6-1
ii  python3-distro         1.0.1-2
ii  python3-distutils      3.6.6-1
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic          2:0.4.15-1
ii  python3-pkg-resources  39.2.0-1

Versions of packages diffoscope recommends:
pn  abootimg                         <none>
ii  acl                              2.2.52-3+b1
pn  apktool                          <none>
pn  binutils-multiarch               <none>
ii  bzip2                            1.0.6-8.1
ii  caca-utils                       0.99.beta19-2+b3
ii  colord                           1.3.3-2
ii  db-util                          5.3.1
ii  default-jdk [java-sdk]           2:1.10-67
ii  default-jdk-headless             2:1.10-67
ii  device-tree-compiler             1.4.6-1
pn  docx2txt                         <none>
ii  e2fsprogs                        1.44.3~rc2-1
pn  enjarify                         <none>
pn  fontforge-extras                 <none>
pn  fp-utils                         <none>
ii  genisoimage                      9:1.1.11-3+b2
ii  gettext                          0.19.8.1-6+b1
ii  ghc                              8.2.2-4
ii  ghostscript                      9.22~dfsg-2.1
ii  giflib-tools                     5.1.4-3
pn  gnumeric                         <none>
ii  gnupg                            2.2.8-3
ii  imagemagick                      8:6.9.10.2+dfsg-2
ii  imagemagick-6.q16 [imagemagick]  8:6.9.10.2+dfsg-2
pn  jsbeautifier                     <none>
ii  libarchive-tools                 3.2.2-4
pn  llvm                             <none>
ii  lz4                              1.8.2-1
ii  mono-utils                       4.6.2.7+dfsg-2
pn  odt2txt                          <none>
pn  oggvideotools                    <none>
ii  openjdk-10-jdk [java-sdk]        10.0.1+10-4
ii  openjdk-11-jdk [java-sdk]        11~21-2
ii  openssh-client                   1:7.7p1-2
pn  pgpdump                          <none>
ii  poppler-utils                    0.63.0-2
pn  procyon-decompiler               <none>
ii  python3-argcomplete              1.8.1-1
pn  python3-binwalk                  <none>
ii  python3-debian                   0.1.32
ii  python3-defusedxml               0.5.0-1
pn  python3-guestfs                  <none>
pn  python3-jsondiff                 <none>
pn  python3-progressbar              <none>
pn  python3-pyxattr                  <none>
ii  python3-tlsh                     3.4.4+20151206-1+b4
pn  r-base-core                      <none>
ii  rpm2cpio                         4.14.1+dfsg1-3
pn  sng                              <none>
ii  sqlite3                          3.24.0-1
ii  squashfs-tools                   1:4.3-6
ii  tcpdump                          4.9.2-3
ii  unzip                            6.0-21
ii  vim-common                       2:8.1.0089-1
ii  xmlbeans                         2.6.0+dfsg-3
ii  xxd                              2:8.1.0089-1
ii  xz-utils                         5.2.2-1.3

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.2.1-1

-- no debconf information

Message #10 received at 903401@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Roderich Schupp <roderich.schupp@gmail.com>, 903401@bugs.debian.org

Subject: Re: Bug#903401: diffoscope fails on deb archives that contain non-compressed data.tar

Date: Mon, 09 Jul 2018 15:35:12 +0100

Hi Roderich,

> libarchive.exception.ArchiveError: Unrecognized archive format 
> (errno=84,
> retcode=-30, archive_p=139857020076192)

This one is a bit more complicated as the uncompressed data.tar is written to (for example)

  /tmp/diffoscope_h6c7nido/tmp4eacglys/0/5.gz

… and then libarchive uses the .gz extension.


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #15 received at 903401@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 903401@bugs.debian.org

Cc: Roderich Schupp <roderich.schupp@gmail.com>

Subject: Re: diffoscope fails on deb archives that contain non-compressed data.tar

Date: Mon, 09 Jul 2018 18:18:43 +0100

tags 903401 + pending
thanks

Fixed in Git, pending upload:

  https://salsa.debian.org/reproducible-builds/diffoscope/commit/185077c843e2f61c3915dab73770867067cb9358

  diffoscope/comparators/deb.py              |   4 +-
  diffoscope/comparators/utils/libarchive.py |  69 ++++++++++++++++-------------
  tests/comparators/test_deb.py              |   6 +++
  tests/data/bug903401_1.deb                 | Bin 0 -> 10876 bytes
  tests/data/bug903401_2.deb                 | Bin 0 -> 10874 bytes
  5 files changed, 46 insertions(+), 33 deletions(-)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #22 received at 903401-close@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 903401-close@bugs.debian.org

Subject: Bug#903401: fixed in diffoscope 100

Date: Sat, 01 Sep 2018 08:34:14 +0000

Source: diffoscope
Source-Version: 100

We believe that the bug you reported is fixed in the latest version of
diffoscope, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 903401@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Chris Lamb <lamby@debian.org> (supplier of updated diffoscope package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Sat, 01 Sep 2018 09:09:48 +0100
Source: diffoscope
Binary: diffoscope
Built-For-Profiles: nocheck
Architecture: source all
Version: 100
Distribution: unstable
Urgency: medium
Maintainer: Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>
Changed-By: Chris Lamb <lamby@debian.org>
Description:
 diffoscope - in-depth comparison of files, archives, and directories
Closes: 903391 903401 903447 903449 903565 904685 905598 906967
Changes:
 diffoscope (100) unstable; urgency=medium
 .
   [ Alexis Murzeau ]
   * Correct matching of .deb archive members. (Closes: #903565)
 .
   [ Chris Lamb ]
   * Support .deb archives that contain an uncompressed control.tar and data.tar
     Thanks to Roderich Schupp <roderich.schupp@gmail.com>.
     (Closes: #903391, #903401)
   * Wrap jsondiff calls with try-except to prevent fatal errors.
     (Closes: #903447, #903449)
   * Don't include the filename in llvm-bcanalyzer results. (Closes: #905598)
   * Update generated debian/tests/control.
   * Bump Standards-Version to 4.2.1.
 .
   [ Daniel Kahn Gillmor ]
   * Avoid line eraser error on dumb terminals. (Closes: #906967)
   * Correct spelling of ereser to eraser.
 .
   [ Mattia Rizzolo ]
   * On Debian, do not require apktool on ppc64el and s390x, ogvideotools on
     s390x and fp-utils on ppc64el and s390x as they are not installable there.
   * Explicitly add `file` to the dependencies of autopkgtest to have the tests
     triggered whenever `file` changes
 .
   [ Ricardo Gaviria ]
   * Handle errors with encrypted archive files. (Closes: #904685)
Checksums-Sha1:
 ee418ed8178f58ab456f7f7f99330c9ea4b3da42 4050 diffoscope_100.dsc
 bcf9d36178a98c9aeb24457588ba0638152661ed 9251936 diffoscope_100.tar.xz
 a60721094ec8ed73f7658baed694bfc94e128ff2 122780 diffoscope_100_all.deb
 af050ca0c265c4d181bd322c9f65323bc1bc4a45 6725 diffoscope_100_amd64.buildinfo
Checksums-Sha256:
 61cbdfa21cfbe6a6a8c32be511a13fbed0d0d1e2788ef5cc859cd2dc54c2c092 4050 diffoscope_100.dsc
 98af4a69d2ded2de221d050a110d60beb7a8b737a3bba1ac40755682c7a45b4b 9251936 diffoscope_100.tar.xz
 9f56af77c7756c9265c615117f0704d0009b8b6de2d164d6afcc8ce78d208e6f 122780 diffoscope_100_all.deb
 6fca17529ee1260aa9576840b97f3dffb0e16476435a655986bf03bc280a1fad 6725 diffoscope_100_amd64.buildinfo
Files:
 30b68b4a6dbcba4bc3aa7d948bd3f4da 4050 devel optional diffoscope_100.dsc
 148c2d1675b1f3475fc6ab28bc39eb2a 9251936 devel optional diffoscope_100.tar.xz
 cd5e06d93db32e437ca32ff5e36cce78 122780 devel optional diffoscope_100_all.deb
 f87adefc92b27a4441634aba05448839 6725 devel optional diffoscope_100_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEwv5L0nHBObhsUz5GHpU+J9QxHlgFAluKSc8ACgkQHpU+J9Qx
Hlg1lxAAjgTqnKwdWr3cQYA7MO4VqB11p9185NkXrR2vkHcUNvN1xv95J7/A4BZx
3BXeK2dfi8Qu1gN2NDeg6+m5FTUMDHesmXMlqrDG5JLVRHXM8a/FqpHHCAGJKLR6
QALd1uE24VSdbxBc2hJvTWDu6t1DbKvhXp83D9kBc8APvOsaxWpFNE7dDHDKEICa
EC8Ne925EEU/8xwl+LTtAN9DQEw0Pmi521NK2pBhX4NfETm6OJODb2cH/DCCMB9x
65IdU0hF4tMDzZOQ2utgWQ5Cx7Se66tcqNmamyunBocfZ77d5EA+IKsiYD2QQ7Iy
G+fhMJYTiiYPfxZvGS8ln3u29p1CFrj5vJ+Cggf7Dl0w+a8WEXIOaKPZ4KIddT7X
k2bg+dtMzr/4VcTFAy6zmtyn86hMNNkF2rdkG2EkeHNEHuP+HwnIPnmMpS5o7BEW
S9jtd93khRD+/Jp/Zen6K8X2As2Gvd3s97+0hInTPezweny/DhapNxwRc/sYLfvv
iuvrkssM4hJ+yXFogvCbI+Q4MLZpUtvXLU/IKDhTVQuPIBDNrSDQChPGOEeWYEsz
2PW+ls6ipNPWUUkL556OiN7V0dbZIEMlSLHzWdzr6FrRZwWZmJAfulO+1GIZTMyu
xwzo6q+pne8fLX7ynE53uXVpzvdAMDPDRkj4cJMMVcKojoCih0E=
=Uc9v
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.