Debian Bug report logs - #780761
diffoscope: fails noisily when run over perversely recursive input files

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: debbindiff fails noisily when run over perversely recursive input files

Date: Wed, 18 Mar 2015 16:37:18 -0400

[Message part 1 (text/plain, inline)]

Package: debbindiff
Version: 9
Severity: wishlist

I'm attaching two ugly fake little .debs that look roughly the same
from the outside.  they each ship only one file in data.tar.gz, which
is r.zip.

in both cases, r.zip is a nasty file that unpacks to show another
file, r.zip, which happens to be identical to the original file [0].

however, the r.zip files in each .deb differ from each other a little
bit.

the result is a huge python backtrace that ultimately ends with:

  File "/usr/lib/python2.7/dist-packages/debbindiff/comparators/zip.py", line 61, in compare_zip_files
    source=name))
  File "/usr/lib/python2.7/dist-packages/debbindiff/comparators/__init__.py", line 119, in compare_files
    return comparator(path1, path2, source)
  File "/usr/lib/python2.7/dist-packages/debbindiff/comparators/utils.py", line 55, in with_fallback
    inside_differences = original_function(path1, path2, source)
  File "/usr/lib/python2.7/dist-packages/debbindiff/comparators/zip.py", line 43, in compare_zip_files
    with ZipFile(path1, 'r') as zip1:
  File "/usr/lib/python2.7/zipfile.py", line 770, in __init__
    self._RealGetContents()
  File "/usr/lib/python2.7/zipfile.py", line 844, in _RealGetContents
    x = ZipInfo(filename)
RuntimeError: maximum recursion depth exceeded


Perhaps leaving debbindiff to the mercies of the python recursion
limits is the way to go here -- at least it's a relatively quick
failure.  But it's conceivable that we want to do something marginally
cleaner.

Some options might be:

 0) limit the unpacking recursion ourselves (mayve 20 levels?), and
    having a graceful termination suggesting that the packages are
    likely to be bogus?

 1) caching the digest of every archive unpacked on the stack, and
    bailing gracefully when we find another internal archive with a
    digest already unpacked.  (maybe this doesn't need to be an error
    for debbindiff -- could we just stop the nested recursion there
    and treat the archive as a non-archive?)

or we could do both of the above, i guess.  I'm open to other
suggestions.

    --dkg

[0] hat tip to http://research.swtch.com/zip for the construction

-- System Information:
Debian Release: 8.0
  APT prefers testing
  APT policy: (500, 'testing'), (200, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages debbindiff depends on:
ii  binutils-multiarch  2.25-5
ii  bzip2               1.0.6-7+b2
ii  cpio                2.11+dfsg-4.1
ii  diffutils           1:3.3-1+b1
ii  file                1:5.22+15-2
ii  fontforge-extras    0.3-4
ii  gettext             0.19.3-2
ii  ghc                 7.6.3-20
ii  gnupg               1.4.19-1
ii  pdftk               2.02-2
ii  poppler-utils       0.26.5-2
ii  python              2.7.8-4
ii  python-debian       0.1.25
ii  python-magic        1:5.22+15-2
ii  python-rpm          4.11.3-1.1
ii  rpm2cpio            4.11.3-1.1
ii  sng                 1.0.2-7
ii  unzip               6.0-16
ii  vim                 2:7.4.488-4
ii  vim-common          2:7.4.488-4
ii  xz-utils            5.1.1alpha+20120614-2+b3

debbindiff recommends no packages.

debbindiff suggests no packages.

-- debconf-show failed

[turtles_1_all.deb (application/vnd.debian.binary-package, attachment)]

[turtles_2_all.deb (application/vnd.debian.binary-package, attachment)]

Message #18 received at 780761@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 780761@bugs.debian.org

Cc: Daniel Kahn Gillmor <dkg@fifthhorseman.net>

Subject: Re: diffoscope: fails noisily when run over perversely recursive input files

Date: Sat, 13 May 2017 07:56:32 +0100

Hi,

> diffoscope: fails noisily when run over perversely recursive input files

I've added some failing-but-caught tests in:

  https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=9fd7be6f16fd5988e39ca6c27ae2beacc69d690e


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #23 received at 780761@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 780761@bugs.debian.org

Cc: Daniel Kahn Gillmor <dkg@fifthhorseman.net>

Subject: Re: diffoscope: fails noisily when run over perversely recursive input files

Date: Sat, 13 May 2017 09:41:00 +0100

tags 780761 + pending
thanks

Fixed in Git:

  https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=80d380823b2b39109659084c57d26ee2ef5e9ad4


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #30 received at 780761-close@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 780761-close@bugs.debian.org

Subject: Bug#780761: fixed in diffoscope 83

Date: Sun, 18 Jun 2017 09:05:15 +0000

Source: diffoscope
Source-Version: 83

We believe that the bug you reported is fixed in the latest version of
diffoscope, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 780761@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Chris Lamb <lamby@debian.org> (supplier of updated diffoscope package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Sun, 18 Jun 2017 15:34:59 +0800
Source: diffoscope
Binary: diffoscope
Built-For-Profiles: nocheck
Architecture: source
Version: 83
Distribution: unstable
Urgency: medium
Maintainer: Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>
Changed-By: Chris Lamb <lamby@debian.org>
Description:
 diffoscope - in-depth comparison of files, archives, and directories
Closes: 780761 833697 850758 856446 856447 857610 857940 858223 858867 859034 859056 859117 861109 861286
Changes:
 diffoscope (83) unstable; urgency=medium
 .
   [ Chris Lamb ]
   * New features:
     * Add comparator for Fontconfig cache files.
   * Bug fixes:
     - Don't fail when run under perversely recursive input files. See
       <https://research.swtch.com/zip> for more information. (Closes: #780761)
     - Prevent a traceback when using --new-file with containers by passing
       progress_name of the Member and not the member itself. (Closes: #861286)
     - Drop passing has_differences around, fixing an issue with generating
       files called '-'.
   * Cleanups & refactoring:
     - Reorder and rename FIFOFeeder arguments to prevent tools incorrectly
       parsing as a SyntaxError.
     - Refactor HTML and text presenters so they fit the same interface.
     - Refactor Presenter to a singleton manager.
     - Split output and configuration of presenters.
     - Prevent abstraction-level violation by defining visual diff support on
       the Presenter classes.
     - Split diffoscope.difference into a feeders module.
     - Document various variables.
     - PEP8-ify and tidy a lot of files.
 .
   [ Ximin Luo ]
   * New features:
     - Add --max-container-depth CLI option.
     - Add various traverse_* methods to Difference.
     - Weigh elements in progress bar by their size.
     - Add a reader for the JSON format.
     - Add a --exclude-command CLI for filtering out long-running commands like
       "readelf --debug-dump=info".
     - Don't show +/- controls for differences with no children.
     - Use unicode chars for the +/- controls instead of punctuation.
   * Bug fixes:
     - Fix --exclude control.tar.gz.
     - Make the progress bar play nicely with --debug.
     - When enforcing max-container-depth, show which internal files differ,
       without showing details.
     - Fix JSON presenter to handle recursion properly.
     - Avoid a ZeroDivisionError in the progress bar
     - Fix create_limited_print_func.
   * Tests:
     - Fix failing tests due to logging global state.
     - Add tips about running with TMPDIR=/run/shm.
   * Cleanups & refactoring:
     - Remove unnecessary "dest" args to argparse.
     - Refactor DirectoryContainer to be more similar to Container.
     - Refactor Container abstract method names.
     - Remove unused imports and tidy up Container.comparisons().
     - rename get_{filtered => adjusted}_members_sizes for consistency.
     - Move tests/comparators/utils up one directory.
     - html-dir: show/hide diff comments which can be very large.
     - Refactor html-dir presenter to a class instance avoiding global state.
     - Move side-by-side and linediff algorithms to difference.py.
     - difference: has_children -> has_visible_children, and take into account
       comments.
     - Move ydiff/linediff from diffoscope.{difference => diff} to group
       unified_diff-related things together
 .
   [ Maria Glukhova ]
   * New features:
     - Add visual comparisons for JPEG, ICO, PNG and static GIF images.
   * Test improvements:
     - Test that external tools providers are being returned when tool is not
       found.
     - Add tests for OutputParserError and ContainerExtractionError.
   * Cleanups & refactoring:
     * Ignore text difference if we have a visual one.
     * Fix link formatting and typo in README.
     * Rename html_output to compute_visual_diffs and explain its purpose.
     * Removed duplicated functions from diff.py.
 .
   [ Mattia Rizzolo ]
   * Export junit-xml style test report when building on Jenkins.
 .
   [ anthraxx ]
   * Extend external Arch Linux tools list.
 .
 diffoscope (82) experimental; urgency=medium
 .
   [ Chris Lamb ]
   * New features:
     - Add support for comparing Pcap files. (Closes: #858867)
     - Add support for .docx and .odt files via docx2txt & odt2txt.
       (Closes: #859056)
     - Add support for PGP files via pgpdump. (Closes: #859034)
     - Add support for comparing Ogg Vorbis files.
   * Bug fixes:
     - Don't crash on invalid archives; print a useful error instead.
       (Closes: #833697)
     - Ensure tests and the runtime environment can locate binaries in
       /usr/sbin (eg. tcpdump)
   * Tests:
     - Ensure that PATH is modified.
     - Ensure @tool_required raises RequiredToolNotFound.
     - Don't assume that /sbin/init exists; locate any /sbin binary manually and
       then test for that. This should prevent FTBFS on travis-ci.org.
     - Show packages installed in report output.
   * Misc:
     - comparators.bzip2: Don't print error output from bzip2 call.
     - comparators.pcap: Show the delta, not the absolute time.
     - Use /usr/share/dpkg/pkg-info.mk over manual calls to dpkg-parsechangelog
       in debian/rules.
     - Document PYTHONPATH usage when running tests in README.Source.
     - Add internal documentation for @tool_required decorator.
 .
   [ beuc@beuc.net ]
   * Display differences in zip platform-specific timestamps. (Closes: #859117)
 .
   [ Ximin Luo ]
   * Add support for R .rds and .rdb object files.
 .
   [ Vagrant Cascadian ]
   * Add support for .dtb (device tree blob) files (Closes: #861109).
 .
 diffoscope (81) experimental; urgency=medium
 .
   [ Chris Lamb ]
   * Correct meaningless "1234-content" metadata when introspecting files
     within archives. This was a regression since #854723 due to the use of
     auto-incrementing on-disk filenames. (Closes: #858223)
   * Refactor get_compressed_content_name.
 .
   [ Ximin Luo ]
   * Improve ISO9660/DOS/MBR check.
 .
 diffoscope (80) experimental; urgency=medium
 .
   * Ensure that we really are using ImageMagick and not, for example, the
     GraphicsMagick compatibility layer installed by
     graphicsmagick-imagemagick-compat. (Closes: #857940)
   * Factor out the unicode decoding of the identify -version output.
   * travis.yml: Don't build tags.
 .
 diffoscope (79) experimental; urgency=medium
 .
   [ Chris Lamb ]
   * Extract SquashFS images in one go rather than per-file, speeding up (eg.)
     Tails ISO comparison by ~10x.
   * Support newer versions of cbfstool to avoid test failures.
     (Closes: #856446)
   * Skip icc test that varies on endian if the Debian-specific patch is not
     present. (Closes: #856447)
   * Compare GIF images using gifbuild. (Closes: #857610)
   * Also interpret "DOS/MBR boot sector" files as ISO images as they may have
     been processed by isohybrid.
   * Progress bar:
     - Hide bar if we are running with --debug mode.
     - Update prior to working on an item so the displayed filename is correct.
 .
   [ Maria Glukhova ]
   * Improve AndroidManifest.xml comparison for APK files.
     - Indicate the AndroidManifest.xml type. (Closes: #850758)
Checksums-Sha1:
 8ee2f30785e4c2eb1509acb9d682eb5ed00fb216 3217 diffoscope_83.dsc
 35b07d8349c804bd8d5028eefc9ccfd17bfe0d0e 641012 diffoscope_83.tar.xz
 8b811f4c2dbb8f9c4be03436326b503ad993d021 6370 diffoscope_83_amd64.buildinfo
Checksums-Sha256:
 487d50c63ee1c67fb304b7244038b935ae756c231f649156a23dc730e8fd1673 3217 diffoscope_83.dsc
 09fc25285eb9ad9ebe6dec80b11c2b3f8c5b7250b3e047be6fefbf0f9149308f 641012 diffoscope_83.tar.xz
 58e3e6df90a0b306f0bb0a419133dd8527ffd7119817a27702bbf87ebaa8c456 6370 diffoscope_83_amd64.buildinfo
Files:
 2f3c4b9bc0f16ea6ac6257ec0283b382 3217 devel optional diffoscope_83.dsc
 dffd5533f8b0bfaf5a5ef59c6ff90f21 641012 devel optional diffoscope_83.tar.xz
 e5852580906da10eb8028c6f4af2c19c 6370 devel optional diffoscope_83_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEwv5L0nHBObhsUz5GHpU+J9QxHlgFAllGNGIACgkQHpU+J9Qx
HliPlA//bthdQrAQR8qfpuJCGj4wNS4j8V4b+ZgyCVLzHtBYmgiPjE69W6CTZZNf
pkZlLTtEkbnlbTL39bgKcNp3PAu8+4pWLj7blG2p/XxCnDrNPQvS0GN5zSI+hy+X
7I51641tjTzJQvYXFk7ZfSBAkwbNrsRm87WZySCOhU+TBSBYASBKpqHRqTpB3Csi
bVIW7z0Yli43h5+Ge9lpazp42ZXffBwt4rDhJHt/tKfoXhh2AXRL5Rer93Pih/Ah
akX/yL9LY691nfxNsv04kBNl4bBGBSKQDzsYuF71GO824R61XmsancMGlUT6QByP
czjILbdzbecFC6N3kikVXciwxPhXdAC3uRzGPIAAG+7Acqee7I270KR2nOxVZIoy
uVPPasXs4nK2pR50E9eHXh8P+TZxHHjTpVaYLh8aJwTBdpoGF/DUB1fmMkpZBvBD
5y7WxQlV2a/ZvwF4Hxt2nu5lJjvfb4zZH/2O07pVGnhPi/gmtAxGw8wLWKMaV6NJ
Ov1CtUMYp6nICWFMqumrDLHcZkp5rGlF2DVQy8l+o8A37gkioGQiwseIH5c/rt35
FVNXLkLwj2HfGC8PdVj8R9ZknVv20Tbhmd58pXshcPnSLetEM8pQbXDeiQHz7iEs
jU02FT9ac/EUHGAeKKoolo82/M4SYSV4H5xRaLHj6QMjuLfvXpU=
=2QRm
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.