Debian Bug report logs - #903565
diffoscope fails when comparing deb's containing differently compressed data.tar members

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Roderich Schupp <roderich.schupp@gmail.com>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: diffoscope fails when comparing deb's containing differently compressed data.tar members

Date: Wed, 11 Jul 2018 14:10:21 +0200

Package: diffoscope
Version: 99
Severity: normal

How to reproduce:
I used the versions of libjson-glib-1.0-0 in buster and sid
(tests/data/test*.deb don't exhibit the problem). Note that

$ diffoscope libjson-glib-1.0-0_1.2.6-1_amd64.deb libjson-
glib-1.0-0_1.4.2-4_amd64.deb

works and the output looks very reasonable (compares the contained shared libs,
even though they have different version!). Both deb's contain data.tar files
compressed with xz. Now recompress data.tar in the first deb with gzip
resulting in libjson-glib-1.0-0_1.2.6-1+gzipped-data-tar_amd64.deb.

$ diffoscope  libjson-glib-1.0-0_1.2.6-1+gzipped-data-tar_amd64.deb libjson-
glib-1.0-0_1.2.6-1_amd64.deb

works and shows

--- libjson-glib-1.0-0_1.2.6-1+gzipped-data-tar_amd64.deb
+++ libjson-glib-1.0-0_1.2.6-1_amd64.deb
├── file list
│ @@ -1,3 +1,3 @@
│ -?rw-r--r--   0        0        0        4 1970-01-01 00:00:00.000000 debian-
binary
│ -?rw-r--r--   0        0        0     2074 1970-01-01 00:00:00.000000
control.tar.gz
│ -?rw-r--r--   0        0        0   187626 1970-01-01 00:00:00.000000
data.tar.gz
│ +-rw-r--r--   0        0        0        4 2017-03-16 19:01:26.000000 debian-
binary
│ +-rw-r--r--   0        0        0     2074 2017-03-16 19:01:26.000000
control.tar.gz
│ +-rw-r--r--   0        0        0   175512 2017-03-16 19:01:26.000000
data.tar.xz
│   --- data.tar.gz
├── +++ data.tar.xz
│┄ Files similar despite different names (difference score: 0)
│ ├── filetype from file(1)
│ │ @@ -1 +1 @@
│ │ -gzip compressed data, was "data.tar", last modified: Wed Jul 11 08:52:12
2018, from Unix, original size 296960
│ │ +XZ compressed data
│ ├── filetype from diffoscope
│ │ @@ -1 +1 @@
│ │ -GzipFile
│ │ +XzFile


But this doesn't work:

$ diffoscope libjson-glib-1.0-0_1.2.6-1+gzipped-data-tar_amd64.deb libjson-
glib-1.0-0_1.4.2-4_amd64.deb
Traceback (most recent call last):
  File "/build/work/diffoscope/diffoscope.git/diffoscope/main.py", line 460, in
main
    sys.exit(run_diffoscope(parsed_args))
  File "/build/work/diffoscope/diffoscope.git/diffoscope/main.py", line 432, in
run_diffoscope
    difference = compare_root_paths(path1, path2)
  File
"/build/work/diffoscope/diffoscope.git/diffoscope/comparators/utils/compare.py",
line 68, in compare_root_paths
    difference = compare_files(file1, file2)
  File
"/build/work/diffoscope/diffoscope.git/diffoscope/comparators/utils/compare.py",
line 118, in compare_files
    return file1.compare(file2, source)
  File
"/build/work/diffoscope/diffoscope.git/diffoscope/comparators/utils/file.py",
line 366, in compare
    difference = self._compare_using_details(other, source)
  File
"/build/work/diffoscope/diffoscope.git/diffoscope/comparators/utils/file.py",
line 321, in _compare_using_details
    other.as_container, no_recurse=no_recurse))
  File
"/build/work/diffoscope/diffoscope.git/diffoscope/comparators/utils/container.py",
line 158, in comparisons
    for my_name, other_name, score in self.perform_fuzzy_matching(my_members,
other_members):
  File "/build/work/diffoscope/diffoscope.git/diffoscope/comparators/deb.py",
line 94, in perform_fuzzy_matching
    for name1 in my_members.keys():
RuntimeError: OrderedDict mutated during iteration


This is with diffoscope from git head.

Cheers, Roderich




-- System Information:
Debian Release: buster/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'unstable'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.17.5 (SMP w/4 CPU cores)
Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8), LANGUAGE=en_US.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  libpython3.6-stdlib    3.6.6-1
ii  python3                3.6.6-1
ii  python3-distro         1.0.1-2
ii  python3-distutils      3.6.6-1
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic          2:0.4.15-1
ii  python3-pkg-resources  39.2.0-1

Versions of packages diffoscope recommends:
pn  abootimg                         <none>
ii  acl                              2.2.52-3+b1
pn  apktool                          <none>
pn  binutils-multiarch               <none>
ii  bzip2                            1.0.6-8.1
ii  caca-utils                       0.99.beta19-2+b3
ii  colord                           1.3.3-2
ii  db-util                          5.3.1
ii  default-jdk [java-sdk]           2:1.10-67
ii  default-jdk-headless             2:1.10-67
ii  device-tree-compiler             1.4.6-1
pn  docx2txt                         <none>
ii  e2fsprogs                        1.44.3-1
pn  enjarify                         <none>
pn  fontforge-extras                 <none>
pn  fp-utils                         <none>
ii  genisoimage                      9:1.1.11-3+b2
ii  gettext                          0.19.8.1-6+b1
ii  ghc                              8.2.2-4
ii  ghostscript                      9.22~dfsg-2.1
ii  giflib-tools                     5.1.4-3
pn  gnumeric                         <none>
ii  gnupg                            2.2.8-3
ii  imagemagick                      8:6.9.10.2+dfsg-3
ii  imagemagick-6.q16 [imagemagick]  8:6.9.10.2+dfsg-3
ii  jsbeautifier                     1.6.4-7
ii  libarchive-tools                 3.2.2-4
pn  llvm                             <none>
ii  lz4                              1.8.2-1
ii  mono-utils                       4.6.2.7+dfsg-2
pn  odt2txt                          <none>
pn  oggvideotools                    <none>
ii  openjdk-10-jdk [java-sdk]        10.0.1+10-4
ii  openjdk-11-jdk [java-sdk]        11~21-2
ii  openssh-client                   1:7.7p1-3
pn  pgpdump                          <none>
ii  poppler-utils                    0.63.0-2
pn  procyon-decompiler               <none>
ii  python3-argcomplete              1.8.1-1
pn  python3-binwalk                  <none>
ii  python3-debian                   0.1.32
ii  python3-defusedxml               0.5.0-1
pn  python3-guestfs                  <none>
pn  python3-jsondiff                 <none>
pn  python3-progressbar              <none>
pn  python3-pyxattr                  <none>
ii  python3-tlsh                     3.4.4+20151206-1+b4
pn  r-base-core                      <none>
ii  rpm2cpio                         4.14.1+dfsg1-3
pn  sng                              <none>
ii  sqlite3                          3.24.0-1
ii  squashfs-tools                   1:4.3-6
ii  tcpdump                          4.9.2-3
ii  unzip                            6.0-21
ii  vim-common                       2:8.1.0089-1
ii  xmlbeans                         2.6.0+dfsg-3
ii  xxd                              2:8.1.0089-1
ii  xz-utils                         5.2.2-1.3

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.2.1-1

-- no debconf information

Message #10 received at 903565@bugs.debian.org (full text, mbox, reply):

From: Alexis Murzeau <amubtdx@gmail.com>

To: 903565@bugs.debian.org, Roderich Schupp <roderich.schupp@gmail.com>

Subject: Re: diffoscope fails when comparing deb's containing differently compressed data.tar members

Date: Mon, 13 Aug 2018 01:21:55 +0200

[Message part 1 (text/plain, inline)]

Control: tag -1 + patch

Hi,

On Wed, 11 Jul 2018 14:10:21 +0200 Roderich Schupp
<roderich.schupp@gmail.com> wrote>
> But this doesn't work:
> 
> $ diffoscope libjson-glib-1.0-0_1.2.6-1+gzipped-data-tar_amd64.deb w
> Traceback (most recent call last):
>   File "/build/work/diffoscope/diffoscope.git/diffoscope/main.py", line 460, in
> main
>     sys.exit(run_diffoscope(parsed_args))
>   File "/build/work/diffoscope/diffoscope.git/diffoscope/main.py", line 432, in

I propose a patch here [0] with a test case.
This fix the issue for me.

[0]
https://salsa.debian.org/reproducible-builds/diffoscope/merge_requests/11

-- 
Alexis Murzeau
PGP: B7E6 0EBB 9293 7B06 BDBC  2787 E7BD 1904 F480 937F

[signature.asc (application/pgp-signature, attachment)]

Message #17 received at 903565@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 903565@bugs.debian.org

Cc: Roderich Schupp <roderich.schupp@gmail.com>

Subject: Re: diffoscope fails when comparing deb's containing differently compressed data.tar members

Date: Mon, 13 Aug 2018 07:57:26 +0100

tags 903565 + pending
thanks

Thanks! Applied in Git with some very minor changes:

  https://salsa.debian.org/reproducible-builds/diffoscope/commit/8b90ec1a4e0c0c73bb67936e80eb898366b67a6f

  diffoscope/comparators/deb.py         |   5 +++++
  diffoscope/comparators/utils/fuzzy.py |   2 +-
  tests/comparators/test_deb.py         |   7 +++++++
  tests/data/bug903565_1.deb            | Bin 0 -> 3332 bytes
  tests/data/bug903565_2.deb            | Bin 0 -> 3392 bytes
  5 files changed, 13 insertions(+), 1 deletion(-)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #24 received at 903565-close@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 903565-close@bugs.debian.org

Subject: Bug#903565: fixed in diffoscope 100

Date: Sat, 01 Sep 2018 08:34:14 +0000

Source: diffoscope
Source-Version: 100

We believe that the bug you reported is fixed in the latest version of
diffoscope, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 903565@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Chris Lamb <lamby@debian.org> (supplier of updated diffoscope package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Sat, 01 Sep 2018 09:09:48 +0100
Source: diffoscope
Binary: diffoscope
Built-For-Profiles: nocheck
Architecture: source all
Version: 100
Distribution: unstable
Urgency: medium
Maintainer: Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>
Changed-By: Chris Lamb <lamby@debian.org>
Description:
 diffoscope - in-depth comparison of files, archives, and directories
Closes: 903391 903401 903447 903449 903565 904685 905598 906967
Changes:
 diffoscope (100) unstable; urgency=medium
 .
   [ Alexis Murzeau ]
   * Correct matching of .deb archive members. (Closes: #903565)
 .
   [ Chris Lamb ]
   * Support .deb archives that contain an uncompressed control.tar and data.tar
     Thanks to Roderich Schupp <roderich.schupp@gmail.com>.
     (Closes: #903391, #903401)
   * Wrap jsondiff calls with try-except to prevent fatal errors.
     (Closes: #903447, #903449)
   * Don't include the filename in llvm-bcanalyzer results. (Closes: #905598)
   * Update generated debian/tests/control.
   * Bump Standards-Version to 4.2.1.
 .
   [ Daniel Kahn Gillmor ]
   * Avoid line eraser error on dumb terminals. (Closes: #906967)
   * Correct spelling of ereser to eraser.
 .
   [ Mattia Rizzolo ]
   * On Debian, do not require apktool on ppc64el and s390x, ogvideotools on
     s390x and fp-utils on ppc64el and s390x as they are not installable there.
   * Explicitly add `file` to the dependencies of autopkgtest to have the tests
     triggered whenever `file` changes
 .
   [ Ricardo Gaviria ]
   * Handle errors with encrypted archive files. (Closes: #904685)
Checksums-Sha1:
 ee418ed8178f58ab456f7f7f99330c9ea4b3da42 4050 diffoscope_100.dsc
 bcf9d36178a98c9aeb24457588ba0638152661ed 9251936 diffoscope_100.tar.xz
 a60721094ec8ed73f7658baed694bfc94e128ff2 122780 diffoscope_100_all.deb
 af050ca0c265c4d181bd322c9f65323bc1bc4a45 6725 diffoscope_100_amd64.buildinfo
Checksums-Sha256:
 61cbdfa21cfbe6a6a8c32be511a13fbed0d0d1e2788ef5cc859cd2dc54c2c092 4050 diffoscope_100.dsc
 98af4a69d2ded2de221d050a110d60beb7a8b737a3bba1ac40755682c7a45b4b 9251936 diffoscope_100.tar.xz
 9f56af77c7756c9265c615117f0704d0009b8b6de2d164d6afcc8ce78d208e6f 122780 diffoscope_100_all.deb
 6fca17529ee1260aa9576840b97f3dffb0e16476435a655986bf03bc280a1fad 6725 diffoscope_100_amd64.buildinfo
Files:
 30b68b4a6dbcba4bc3aa7d948bd3f4da 4050 devel optional diffoscope_100.dsc
 148c2d1675b1f3475fc6ab28bc39eb2a 9251936 devel optional diffoscope_100.tar.xz
 cd5e06d93db32e437ca32ff5e36cce78 122780 devel optional diffoscope_100_all.deb
 f87adefc92b27a4441634aba05448839 6725 devel optional diffoscope_100_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEwv5L0nHBObhsUz5GHpU+J9QxHlgFAluKSc8ACgkQHpU+J9Qx
Hlg1lxAAjgTqnKwdWr3cQYA7MO4VqB11p9185NkXrR2vkHcUNvN1xv95J7/A4BZx
3BXeK2dfi8Qu1gN2NDeg6+m5FTUMDHesmXMlqrDG5JLVRHXM8a/FqpHHCAGJKLR6
QALd1uE24VSdbxBc2hJvTWDu6t1DbKvhXp83D9kBc8APvOsaxWpFNE7dDHDKEICa
EC8Ne925EEU/8xwl+LTtAN9DQEw0Pmi521NK2pBhX4NfETm6OJODb2cH/DCCMB9x
65IdU0hF4tMDzZOQ2utgWQ5Cx7Se66tcqNmamyunBocfZ77d5EA+IKsiYD2QQ7Iy
G+fhMJYTiiYPfxZvGS8ln3u29p1CFrj5vJ+Cggf7Dl0w+a8WEXIOaKPZ4KIddT7X
k2bg+dtMzr/4VcTFAy6zmtyn86hMNNkF2rdkG2EkeHNEHuP+HwnIPnmMpS5o7BEW
S9jtd93khRD+/Jp/Zen6K8X2As2Gvd3s97+0hInTPezweny/DhapNxwRc/sYLfvv
iuvrkssM4hJ+yXFogvCbI+Q4MLZpUtvXLU/IKDhTVQuPIBDNrSDQChPGOEeWYEsz
2PW+ls6ipNPWUUkL556OiN7V0dbZIEMlSLHzWdzr6FrRZwWZmJAfulO+1GIZTMyu
xwzo6q+pne8fLX7ynE53uXVpzvdAMDPDRkj4cJMMVcKojoCih0E=
=Uc9v
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.