Debian Bug report logs - #1012258
diffoscope crashes with IndexError: tuple index out of range

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Sander van Grieken <sander@outrightsolutions.nl>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: diffoscope crashes with IndexError: tuple index out of range

Date: Thu, 02 Jun 2022 13:20:38 +0200

Package: diffoscope
Version: 214
Severity: important
X-Debbugs-Cc: sander@outrightsolutions.nl

Dear Maintainer,

Using diffoscope to compare two files (tar.gz) diffoscope crashes with the
stacktrace below.

The files in question are assets/private.mp3 from two builds of Electrum APK,
which are tar-gzipped file trees containing python code.

Testing out older versions of diffoscope reveals that the problem starts from
diffoscope-187


-- Stacktrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/diffoscope/main.py", line 762,
in main
    sys.exit(run_diffoscope(parsed_args))
  File "/usr/local/lib/python3.9/dist-packages/diffoscope/main.py", line 716,
in run_diffoscope
    difference = compare_root_paths(path1, path2)
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/compare.py", line 69, in
compare_root_paths
    difference = compare_files(file1, file2)
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/compare.py", line 146, in compare_files
    return file1.compare(file2, source)
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/file.py", line 513, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/file.py", line 448, in
_compare_using_details
    details.extend(
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/container.py", line 197, in compare_pair
    difference = compare_files(
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/compare.py", line 146, in compare_files
    return file1.compare(file2, source)
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/file.py", line 513, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/file.py", line 448, in
_compare_using_details
    details.extend(
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/container.py", line 197, in compare_pair
    difference = compare_files(
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/compare.py", line 146, in compare_files
    return file1.compare(file2, source)
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/file.py", line 513, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/utils/file.py", line 414, in
_compare_using_details
    details.extend(self.compare_details(other, source))
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/python.py", line 49, in compare_details
    describe_pyc(self.path),
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/python.py", line 63, in describe_pyc
    return "\n".join(parse_pyc(f))
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/python.py", line 80, in parse_pyc
    yield from show_code(code)
  File "/usr/local/lib/python3.9/dist-
packages/diffoscope/comparators/python.py", line 93, in show_code
    dis.disassemble(code, file=s)
  File "/usr/lib/python3.9/dis.py", line 369, in disassemble
    _disassemble_bytes(co.co_code, lasti, co.co_varnames, co.co_names,
  File "/usr/lib/python3.9/dis.py", line 401, in _disassemble_bytes
    for instr in _get_instructions_bytes(code, varnames, names,
  File "/usr/lib/python3.9/dis.py", line 347, in _get_instructions_bytes
    argval = cmp_op[arg]
IndexError: tuple index out of range


-- System Information:
Debian Release: bookworm/sid
  APT prefers testing
  APT policy: (750, 'testing'), (700, 'stable'), (600, 'unstable'), (500, 'stable-security')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.17.5+ (SMP w/12 CPU threads; PREEMPT)
Kernel taint flags: TAINT_WARN, TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
pn  diffoscope-minimal  <none>

Versions of packages diffoscope recommends:
ii  abootimg                         0.6-1+b2
ii  acl                              2.3.1-1
ii  androguard                       3.4.0~a1-2
ii  apksigner                        31.0.2-1
ii  apktool                          2.6.1+dfsg.1-2
ii  binutils-multiarch               2.38-4
ii  bzip2                            1.0.8-5
ii  caca-utils                       0.99.beta19-3
ii  colord                           1.4.6-1
ii  db-util                          5.3.1+nmu1
ii  device-tree-compiler             1.6.1-1
ii  docx2txt                         1.4-5
ii  e2fsprogs                        1.46.5-2
ii  enjarify                         1:1.0.3-5
ii  ffmpeg                           10:4.4.1-dmo6
ii  fontforge-extras                 1:20201107~dfsg-4+b1
ii  fp-utils                         3.2.2+dfsg-11
ii  fp-utils-3.2.2 [fp-utils]        3.2.2+dfsg-11
ii  genisoimage                      9:1.1.11-3.2
ii  gettext                          0.21-6
ii  ghc                              8.8.4-3
ii  ghostscript                      9.56.1~dfsg-1
ii  giflib-tools                     5.2.1-2.4
ii  gnumeric                         1.12.52-1
ii  gnupg                            2.2.35-2
ii  gnupg-utils                      2.2.35-2
ii  hdf5-tools                       1.10.7+repack-4
ii  imagemagick                      8:6.9.11.60+dfsg-1.3+b2
ii  imagemagick-6.q16 [imagemagick]  8:6.9.11.60+dfsg-1.3+b2
ii  jsbeautifier                     1.14.3-1
ii  libarchive-tools                 3.6.0-1
ii  libxmlb-dev                      0.3.8-1
ii  llvm                             1:13.0-54
ii  lz4 [liblz4-tool]                1.9.3-2
ii  mono-utils                       6.8.0.105+dfsg-3.2
ii  ocaml-nox                        4.13.1-3
ii  odt2txt                          0.5-7
ii  oggvideotools                    0.9.1-6
ii  openjdk-11-jdk [java-sdk]        11.0.14.1+1-1
ii  openjdk-8-jdk [java-sdk]         8u332-ga-1
ii  openssh-client                   1:9.0p1-1
ii  openssl                          1.1.1o-1
ii  pgpdump                          0.34-1
ii  poppler-utils                    22.02.0-3
pn  procyon-decompiler               <none>
ii  python3-argcomplete              2.0.0-1
ii  python3-binwalk                  2.3.3+dfsg1-2
ii  python3-debian                   0.1.44
ii  python3-defusedxml               0.7.1-1
pn  python3-guestfs                  <none>
ii  python3-jsondiff                 1.3.1-2
ii  python3-pdfminer                 20220319+dfsg-1
ii  python3-progressbar              2.5-2
ii  python3-pypdf2                   1.26.0-4
ii  python3-pyxattr                  0.7.2-2
ii  python3-rpm                      4.17.0+dfsg1-4
ii  python3-tlsh                     3.4.4+20151206-1.4+b1
ii  r-base-core                      4.2.0-1+b1
ii  radare2                          5.5.0+dfsg-1
ii  rpm2cpio                         4.17.0+dfsg1-4
ii  sng                              1.1.0-4
ii  sqlite3                          3.38.5-1
ii  squashfs-tools                   1:4.5.1-1
ii  tcpdump                          4.99.1-4
ii  u-boot-tools                     2022.04+dfsg-2
ii  unzip                            6.0-26
ii  vim-common                       2:8.2.4793-1
ii  wabt                             1.0.29-1
ii  xmlbeans                         4.0.0-1
ii  xxd                              2:8.2.4793-1
ii  xz-utils                         5.2.5-2.1
ii  zip                              3.0-12
ii  zstd                             1.5.2+dfsg-1

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.6.0+dfsg+~3.5.13-1

Message #10 received at 1012258@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: "Sander van Grieken" <sander@outrightsolutions.nl>, 1012258@bugs.debian.org

Subject: Re: Bug#1012258: diffoscope crashes with IndexError: tuple index out of range

Date: Thu, 02 Jun 2022 12:42:11 +0100

Hi Sander,

> Using diffoscope to compare two files (tar.gz) diffoscope crashes with the
> stacktrace below.

Thanks for the report. Given that you mention .tar.gz & .mp3 files,
and that diffoscope is crashing within code for comparing Python .pyc
files (!), I suspect what is happening is that diffoscope is not
detecting the filetype correctly. (Still, diffoscope shouldn't be
crashing when encountering an "invalid" .pyc file. In other words,
there is at least one problem here.)

Please could you provide:

  a) The two .tar.gz files in question

and:

  b) The result of running diffoscope and appending "--debug 2>debug.log"
     and attaching the resulting debug.log.

With that, I'm sure I can quite easily diagnose and fix both potential
issues for you.


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Message #15 received at 1012258@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: 1012258@bugs.debian.org

Subject: Fwd: Re: Bug#1012258: diffoscope crashes with IndexError: tuple index out of range

Date: Thu, 02 Jun 2022 13:02:04 +0100

[Message part 1 (text/plain, inline)]

----- Original message -----
From: Sander van Grieken <sander@outrightsolutions.nl>
To: Chris Lamb <lamby@debian.org>
Subject: Re: Bug#1012258: diffoscope crashes with IndexError: tuple index out of range
Date: Thursday, 2 June 2022 12:52 PM

On donderdag 2 juni 2022 13:42:11 CEST you wrote:

> Hi Sander,
> 
> > Using diffoscope to compare two files (tar.gz) diffoscope crashes with the
> > stacktrace below.
> 
> Thanks for the report. Given that you mention .tar.gz & .mp3 files,
> and that diffoscope is crashing within code for comparing Python .pyc
> files (!), I suspect what is happening is that diffoscope is not
> detecting the filetype correctly. (Still, diffoscope shouldn't be
> crashing when encountering an "invalid" .pyc file. In other words,
> there is at least one problem here.)
> 
> Please could you provide:
> 
>   a) The two .tar.gz files in question
> 
> and:
> 
>   b) The result of running diffoscope and appending "--debug 2>debug.log"
>      and attaching the resulting debug.log.
> 
> With that, I'm sure I can quite easily diagnose and fix both potential
> issues for you.
> 
> 
> Regards,

debug.log attached. The two archives are provided here: 
http://www.outrightsolutions.nl/~sander/debian/private.mp3
_http://www.outrightsolutions.nl/~sander/debian/private_2_.mp3_

Regards,

Sander

[debug.log (text/x-log, attachment)]

Message #18 received at 1012258-submitter@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 1012258-submitter@bugs.debian.org

Subject: Bug#1012258 marked as pending in diffoscope

Date: Thu, 02 Jun 2022 12:03:46 +0000

Control: tag -1 pending

Hello,

Bug #1012258 in diffoscope reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:

https://salsa.debian.org/reproducible-builds/diffoscope/-/commit/35a9bfbee0c1a5f00fb0a66b689514d21bacd6a5

------------------------------------------------------------------------
Also catch IndexError (in addition to ValueError) when parsing .pyc files. (Closes: #1012258)
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/1012258

Message #25 received at 1012258@bugs.debian.org (full text, mbox, reply):

From: "Chris Lamb" <lamby@debian.org>

To: "Sander van Grieken" <sander@outrightsolutions.nl>, 1012258@bugs.debian.org

Subject: Re: Bug#1012258: diffoscope crashes with IndexError: tuple index out of range

Date: Thu, 02 Jun 2022 13:07:06 +0100

(Don't forget to keep all recipients in the To/CC when replying; thanks...)

> debug.log attached. The two archives are provided here:
> http://www.outrightsolutions.nl/~sander/debian/private.mp3
> http://www.outrightsolutions.nl/~sander/debian/private2.mp3

Cheers for these. I've pushed a fix and I'll release it tomorrow in
diffoscope version 215.

Out of interest, how come you name these .tgz files .mp3?


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

Message #30 received at 1012258-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 1012258-close@bugs.debian.org

Subject: Bug#1012258: fixed in diffoscope 215

Date: Fri, 03 Jun 2022 06:03:56 +0000

Source: diffoscope
Source-Version: 215
Done: Chris Lamb <lamby@debian.org>

We believe that the bug you reported is fixed in the latest version of
diffoscope, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1012258@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Chris Lamb <lamby@debian.org> (supplier of updated diffoscope package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Fri, 03 Jun 2022 06:54:48 +0100
Source: diffoscope
Built-For-Profiles: nocheck
Architecture: source
Version: 215
Distribution: unstable
Urgency: medium
Maintainer: Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>
Changed-By: Chris Lamb <lamby@debian.org>
Closes: 1011635 1012258
Changes:
 diffoscope (215) unstable; urgency=medium
 .
   [ Chris Lamb ]
   * Bug fixes:
     - Also catch IndexError (in addition to ValueError) when parsing .pyc
       files. (Closes: #1012258)
     - Strip "sticky" etc. from "x.deb: sticky Debian binary package […]".
       Thanks to David Prévot for the report. (Closes: #1011635)
     - Correctly package diffoscope's scripts/ directory, fixing the extraction
       of vmlinuz kernel images. (Closes: reproducible-builds/diffoscope#305)
     - Correct the logic for supporting different versions of argcomplete in
       debian/rules.
 .
   * New features:
     - Support both PyPDF 1.x and 2.x.
 .
   * Codebase improvements:
     - Don't call re.compile and then call .sub on the result; just call
       re.sub directly.
     - Clarify the logic around the difference between --usage and --help.
 .
   * Testsuite improvements:
     - Integrate test coverage with GitLab's concept of artifacts.
     - Re-enable Gnumeric tests as its now available again.
     - Test --help and --usage, and additionally test that --help includes
       the programmatically-generated file format list as well.
 .
   [ Holger Levsen ]
   * Bump Standards Version field in debian/control to 4.6.1.
Checksums-Sha1:
 3952f5b86fb57c88f1f7dbb0a7641b213cede492 4966 diffoscope_215.dsc
 bb1fc71692d804d673bb46edf1f23266317eb584 2097172 diffoscope_215.tar.xz
 dfc79e2ba8651808b1b20c4933a602458d857503 7119 diffoscope_215_amd64.buildinfo
Checksums-Sha256:
 60ecf0d5e95e3b69651e0881ab373c332b7f7fb942ad467adc30ea6afba79f2c 4966 diffoscope_215.dsc
 fa920480e97118e53b39cee0ced2dfec219528334eacd5d50fa9f081162ad483 2097172 diffoscope_215.tar.xz
 87a4d2e842e85fa5a29ba6910fa9634ed99a13b1bc5135f3b0d76871f35d43b6 7119 diffoscope_215_amd64.buildinfo
Files:
 934c25b7fafa4ac8de03f5e12dc55099 4966 devel optional diffoscope_215.dsc
 de5f4e9d5d06897a2516a8d0fc46f019 2097172 devel optional diffoscope_215.tar.xz
 d219047424e8b161a1a777aee7f4d750 7119 devel optional diffoscope_215_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEwv5L0nHBObhsUz5GHpU+J9QxHlgFAmKZonAACgkQHpU+J9Qx
HlgUHhAAwOtGabSktlUwjtMemIgw6zkwG+fHmmSqpCCEyUkv7cM8FSD5s7kMYQBr
FYAH54wZb/MS1YapUKdVVCyJZUnPBOsZM5DeRtF1KUEIh4raiRhNGxCTPifo0ACe
zh02DkK36IxrUNPNSWxuBpKmph4yBsI9XI9XqdEPmudnVjY6mK014P+H+rxX62vC
Yl1BV848JK77A8PYsIRnftWMg7BhEeU4+niubkEHf7+2xf1/lOK57Axq+CNuxTYI
0CPJJtvXEX/VygYX8qKr9JB++HQCaEcHLFMte9O+vSloOBzds7DJZX/KCzAymmt7
BuxvLB/zCstNgOjGIug5CxPQtW054ZR05HDPwUyob0FBeEeORARsuSxpGaicUnPS
AieWfehwY77hwAwWhqyhOxa9xwGjns+6u4D304mlprGlOwjHI6gh7G81CWPvQIn6
6aV0++Wm8dfywX55rNDajOAXhFdHItFJaAWFL7vupGjVynCD0meB/KbaP659UbqE
LYXG+Q9y3hZxflhAg7/K8uQeWNogTx+BhYaPgYUN/1IoUQoV2f/IFYBkdyvB3FAO
2OFn8Pd6hTwlBW3Q3LfHwiU7bZh6nXlAarOJbBfttcCftebiiPuc7yfwiCO3ExwY
67DcBUjq3oedOJK+aPLEAGgpdB3z09UilO8OtsnuNy85uFWnmT0=
=oJ4F
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.