Debian Bug report logs - #903446
diffoscope: libarchive.exception.ArchiveError: Unrecognized archive format (via comparators.debian) with lie/2.2.2+dfsg-3

Toggle useless messages

Message #3 received at submit@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: submit@bugs.debian.org

Subject: diffoscope: libarchive.exception.ArchiveError with lie/2.2.2+dfsg-3

Date: Tue, 10 Jul 2018 10:09:52 +0200

[Message part 1 (text/plain, inline)]

Package: diffoscope
Version 98
Severity: important

As seen with lie/2.2.2+dfsg-3 on all suites/architectures:

Mon Jul  9 00:26:25 UTC 2018  I: diffoscope 98 will be used to compare the two builds:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 327, in diff
    return run_diff(fifo1_path, fifo2_path, fifo1.end_nl_q, fifo2.end_nl_q)
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 220, in __exit__
    self.join()
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 253, in join
    raise self._exception
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 244, in run
    end_nl = self.feeder(fifo)
  File "/usr/lib/python3/dist-packages/diffoscope/feeders.py", line 45, in feeder
    for buf in in_file:
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/libarchive.py", line 91, in list_libarchive
    with libarchive.file_reader(path) as archive:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/libarchive/read.py", line 73, in file_reader
    ffi.read_open_filename_w(archive_p, path, block_size)
  File "/usr/lib/python3/dist-packages/libarchive/ffi.py", line 85, in check_int
    raise archive_error(args[0], retcode)
  File "/usr/lib/python3/dist-packages/libarchive/ffi.py", line 69, in archive_error
    raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
libarchive.exception.ArchiveError: Unrecognized archive format (errno=84, retcode=-30, archive_p=140510660415776)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 460, in main
    sys.exit(run_diffoscope(parsed_args))
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 432, in run_diffoscope
    difference = compare_root_paths(path1, path2)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 68, in compare_root_paths
    difference = compare_files(file1, file2)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 118, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/debian.py", line 184, in compare
    differences = super().compare(other, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 366, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 321, in _compare_using_details
    other.as_container, no_recurse=no_recurse))
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py", line 177, in compare_pair
    file1, file2, source=None, diff_content_only=no_recurse)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 118, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 366, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 321, in _compare_using_details
    other.as_container, no_recurse=no_recurse))
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py", line 177, in compare_pair
    file1, file2, source=None, diff_content_only=no_recurse)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 118, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 366, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 321, in _compare_using_details
    other.as_container, no_recurse=no_recurse))
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py", line 177, in compare_pair
    file1, file2, source=None, diff_content_only=no_recurse)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 118, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 366, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 321, in _compare_using_details
    other.as_container, no_recurse=no_recurse))
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py", line 177, in compare_pair
    file1, file2, source=None, diff_content_only=no_recurse)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 118, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 366, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 304, in _compare_using_details
    details.extend(self.compare_details(other, source))
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/elf.py", line 590, in compare_details
    source="file list",
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line 228, in from_text_readers
    **kwargs
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line 186, in from_feeder
    unified_diff = diff(feeder1, feeder2)
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 327, in diff
    return run_diff(fifo1_path, fifo2_path, fifo1.end_nl_q, fifo2.end_nl_q)
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 220, in __exit__
    self.join()
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 253, in join
    raise self._exception
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 244, in run
    end_nl = self.feeder(fifo)
  File "/usr/lib/python3/dist-packages/diffoscope/feeders.py", line 45, in feeder
    for buf in in_file:
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/libarchive.py", line 91, in list_libarchive
    with libarchive.file_reader(path) as archive:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/libarchive/read.py", line 73, in file_reader
    ffi.read_open_filename_w(archive_p, path, block_size)
  File "/usr/lib/python3/dist-packages/libarchive/ffi.py", line 85, in check_int
    raise archive_error(args[0], retcode)
  File "/usr/lib/python3/dist-packages/libarchive/ffi.py", line 69, in archive_error
    raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
libarchive.exception.ArchiveError: Unrecognized archive format (errno=84, retcode=-30, archive_p=140510526190864)


I can totally reproduce this locally with v99, artifacts are available
till tomorrow on
https://tests.reproducible-builds.org/debian/artifacts/r00t-me/lie_unstable_amd64_tmp-JkpRk/
and I also have a copy of them.


-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Message #12 received at 903446@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 903446@bugs.debian.org

Subject: Re: diffoscope: libarchive.exception.ArchiveError: Unrecognized archive format (via comparators.debian) with lie/2.2.2+dfsg-3

Date: Fri, 09 Nov 2018 12:00:30 -0500

tags 903446 + pending
thanks

On #debian-reproducible:

  < mapreri> lamby: one thing I'd love if you could help with is
             #903446 - I have some trouble following through the
             may indirections and I don't understand why it's not
             falling back to binary comparison as usual. I'd
             appreciate if you could stack that bug in your pile of
             todos.

Sure. Fixed in Git, pending upload:

  https://salsa.debian.org/reproducible-builds/diffoscope/commit/a1b57f8158ca0af72ed00e0a76ab6f50cc32f737

  diffoscope/comparators/elf.py | 1 -
  1 file changed, 1 deletion(-)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #19 received at 903446@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 903446@bugs.debian.org

Subject: Re: Bug#903446: diffoscope: libarchive.exception.ArchiveError: Unrecognized archive format (via comparators.debian) with lie/2.2.2+dfsg-3

Date: Fri, 09 Nov 2018 12:04:15 -0500

tags 903446 - pending
thanks

Chris Lamb wrote:

> Sure. Fixed in Git, pending upload:

I have reverted this as it causes regressions in the rlib tests:

    tests/comparators/test_rlib.py::test_identification FAILED               [ 81%]
    tests/comparators/test_rlib.py::test_no_differences PASSED               [ 82%]
    tests/comparators/test_rlib.py::test_num_items FAILED                    [ 82%]
    tests/comparators/test_rlib.py::test_item0_armap FAILED                  [ 82%]
    tests/comparators/test_rlib.py::test_item1_elf FAILED                    [ 82%]
    tests/comparators/test_rlib.py::test_item2_rust_metadata_bin FAILED      [ 83%]
    tests/comparators/test_rlib.py::test_item3_deflate_llvm_bitcode FAILED   [ 83%]
    tests/comparators/test_rlib.py::test_compare_non_existing FAILED         [ 83%]

I will get back to it soon.


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #24 received at 903446@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: Chris Lamb <lamby@debian.org>, 903446@bugs.debian.org

Subject: Re: Bug#903446: diffoscope: libarchive.exception.ArchiveError: Unrecognized archive format (via comparators.debian) with lie/2.2.2+dfsg-3

Date: Fri, 9 Nov 2018 18:07:26 +0100

On Fri, Nov 09, 2018 at 12:04:15PM -0500, Chris Lamb wrote:
> > Sure. Fixed in Git, pending upload:
> 
> I have reverted this as it causes regressions in the rlib tests:

I think the most proper thing to do would be to understand why it's not
falling back to hex comparison when libarchive raises its exception?

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

Message #29 received at 903446@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Mattia Rizzolo <mattia@debian.org>, 903446@bugs.debian.org

Subject: Re: Bug#903446: diffoscope: libarchive.exception.ArchiveError: Unrecognized archive format (via comparators.debian) with lie/2.2.2+dfsg-3

Date: Fri, 09 Nov 2018 12:11:31 -0500

Gentile Mattia,

> I think the most proper thing to do would be to understand why it's not
> falling back to hex comparison when libarchive raises its exception?

We could have such a "brute force" solution indeed but poking very
quickly I think either the code or the comment is incorrect here:

    @classmethod
    def recognizes(cls, file):
        """Check if a file's type matches the one represented by this class.

        The default test returns True if the file matches these tests:

        (cls.FILE_TYPE_RE OR
         cls.FILE_TYPE_HEADER_PREFIX) AND
        (cls.FILE_EXTENSION_SUFFIX)
        
    [..]

We don't match the string returned from file(1) (ie. "data") but we
match the FILE_EXTENSION_SUFFIX (".a"), yet we still return True
for StaticLibFile.recognizes.

From my reading of the above, we should return False.


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #34 received at 903446@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 903446@bugs.debian.org

Subject: Re: diffoscope: libarchive.exception.ArchiveError: Unrecognized archive format (via comparators.debian) with lie/2.2.2+dfsg-3

Date: Fri, 09 Nov 2018 12:25:36 -0500

[Message part 1 (text/plain, inline)]

Hi,

> diffoscope: libarchive.exception.ArchiveError: Unrecognized
> archive format (via comparators.debian) with lie/2.2.2+dfsg-3

Attaching 2 test files to make this easier to jump back into. This
is failing as of:

  https://salsa.debian.org/reproducible-builds/diffoscope/commit/16d519a9a1f77f45ea8ab5464f1b960db0d16983
  
(ie. in diffoscope 105)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

[a.INFO.a (application/x-archive, attachment)]

[b.INFO.a (application/x-archive, attachment)]

Message #39 received at 903446@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 903446@bugs.debian.org

Subject: Re: diffoscope: libarchive.exception.ArchiveError: Unrecognized archive format (via comparators.debian) with lie/2.2.2+dfsg-3

Date: Fri, 09 Nov 2018 17:17:35 -0500

tags 903446 + pending
thanks

This is now fixed in Git, pending upload. I can confirm that (at
least) the previously-regressing rlib tests do not fail now.

The commit/diffstat is at:

  https://salsa.debian.org/reproducible-builds/diffoscope/commit/cd4c64234a7eebe71fb6bdea21850622f3f5c4c8

  diffoscope/comparators/elf.py        |   2 ++
  diffoscope/comparators/utils/file.py |   4 ++++
  tests/comparators/test_elf.py        |   9 +++++++++
  tests/data/bug_903446.a              | Bin 0 -> 81069 bytes
  4 files changed, 15 insertions(+)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #46 received at 903446-close@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: 903446-close@bugs.debian.org

Subject: Bug#903446: fixed in diffoscope 106

Date: Fri, 23 Nov 2018 11:19:49 +0000

Source: diffoscope
Source-Version: 106

We believe that the bug you reported is fixed in the latest version of
diffoscope, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 903446@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Mattia Rizzolo <mattia@debian.org> (supplier of updated diffoscope package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Fri, 23 Nov 2018 11:09:28 +0100
Source: diffoscope
Binary: diffoscope
Architecture: source
Version: 106
Distribution: unstable
Urgency: medium
Maintainer: Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>
Changed-By: Mattia Rizzolo <mattia@debian.org>
Description:
 diffoscope - in-depth comparison of files, archives, and directories
Closes: 903446 913315
Changes:
 diffoscope (106) unstable; urgency=medium
 .
   [ Chris Lamb ]
   * comparators:
     + elf: Don't assume all files called ".a" are ELF binaries.  Closes: #903446
     + pdf:
       - Display the reason when cannot extract metadata from PDF files.
       - Prevent tracebacks when obtaining PDF metadata from files with multiple
         PDF dictionary definition entries.  Closes: #913315
 .
   [ Marius Gedminas ]
   * Add a python_requires to setup.py.
 .
   [ Mattia Rizzolo ]
   * debian:
     + Disable depends on gnumeric and procyon as they are RC buggy.
     + Override the new lintian's public-upstream-key-in-native-package.
Checksums-Sha1:
 4d1041a25ebaa4f4c9042926858187d51e8490af 4414 diffoscope_106.dsc
 27de1757c263157b8cf5505bfa0b64d7f89e7fd9 9277820 diffoscope_106.tar.xz
 d18d7505876284370d649ef9c5ea7290e42b8c21 20677 diffoscope_106_amd64.buildinfo
Checksums-Sha256:
 a1c920dbcfe5fbb10f291c9e0d8b47542c82b17f500405e97b7b875e30ce8128 4414 diffoscope_106.dsc
 3d0628088acc38d5edce1bab9b83ee409798ae4eb6a1c843655af740db32df94 9277820 diffoscope_106.tar.xz
 4d5216eb570da86858bb23d95d0cbc33c306043cfbea76c45169e649b755f553 20677 diffoscope_106_amd64.buildinfo
Files:
 cb915c4f0dde771df7dbb135ed1e1786 4414 devel optional diffoscope_106.dsc
 5fc534a9b4a8c89286e75a9fd0ef36c5 9277820 devel optional diffoscope_106.tar.xz
 7011db4ddb2b6c507f93980e171bd6fc 20677 devel optional diffoscope_106_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEi3hoeGwz5cZMTQpICBa54Yx2K60FAlv33H0ACgkQCBa54Yx2
K61HgxAAolWXdN3nSFXd0h0pkoOrGFmBq4oTOYYR8PoOLYnpnX88GnaQXy+ytCrl
yI7/16MzvOvXdtj5CjaJjB9lAH0pYF2l4R5Kg/7oMmI3s6b7Ky1snY/9X4jwfNcw
z6uCGicZ+eWlZQgFD7I5++MlNDvQZKCF5jMwzGf/ZouQJxwakXmEwvmhHGjS4E3U
+PTsHlaxpUIn3RRTK5kSynhSXW89ef6atqszmVOJu2pQaX6jXlft6iF4d9YskU2b
o/7syQDISyOEtCsdK/iG+LhZwaYjfinrKE/9rRAU1eZ8bZ/aF8IFFklkLgJTh1KO
RYIL/YXq5jov30tQcmJ8uJBVYrAYJqV+P9pjNKZXl1bUGk7aV7q8r/Gq35TJ8anE
y4zki88QMXwFV69skOHtCrnQ0DQA1S2I8wTG/n9OUuq6wKGvFER8XaqdS+7KicLG
OLdQph8wMNV+1F2Lwq1DMK/X5XzgB+hPH9j7XA3kDS4u5e5/E6I99xcP7XVYWw38
v5f8YlUY7sEe33xaX69074RhEJvr1koAm/Rc/zV975t1lp5wevJKAKGqSpYW7zqp
DXffugkfUdsQB8Ni4HG7FvqWLEd/+h3x93KEyuXDec1MHu333/Lh+X1wasxydmXU
KuPf72SCSeccCvPGrY1OQNikVoTJ2cqmzl9SBzMghHGem2RGtjc=
=IVWp
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.