Debian Bug report logs - #912756
diffoscope: Test failures with upcoming upstream file(1) version 5.35

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: diffoscope: Test failures with upcoming upstream file(1) version 5.35

Date: Sat, 3 Nov 2018 16:13:58 +0100

[Message part 1 (text/plain, inline)]

Package: diffoscope
Version: 104
Severity: important

Dear Maintainer,

A new upstream release of file/libmagic has been accepted into
experimental a few moments ago, version 1:5.35-1. As your package is
one of those that somewhat suffered from surprising feature changes of
libmagic in the past, I'd like to give you an opportunity to test and
to prepare for any changes, for better or for worse. In case of the
latter, use the BTS as usual to report detections that could see an
improvement.

Test-building diffoscope resulted in a regression, see log extract
below. I reckon this was introduced by an upstream change of pcap file
detection, e.g.

-diffoscope-104/tests/data/test2.pcap: tcpdump capture file (little-endian) - version 2.4 (Ethernet, capture length 262144)
+diffoscope-104/tests/data/test2.pcap: pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 262144)

My plan is to upload to unstable in a week from now. Let me know, if you
need more time.

Sorry for the hazzle,

    Christoph, bringer of the bad news

-- System Information:
Debian Release: buster/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.18.11 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: unable to detect

Versions of packages diffoscope depends on:
ii  libpython3.5-stdlib    3.5.6-1
ii  libpython3.6-stdlib    3.6.7-1
ii  python3                3.6.7-1
ii  python3-distro         1.3.0-1
ii  python3-distutils      3.7.1-1
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic          2:0.4.15-2
ii  python3-pkg-resources  40.4.3-1

Versions of packages diffoscope recommends:
pn  abootimg                                       <none>
pn  acl                                            <none>
pn  apktool                                        <none>
pn  binutils-multiarch                             <none>
ii  bzip2                                          1.0.6-9
pn  caca-utils                                     <none>
pn  colord                                         <none>
pn  db-util                                        <none>
pn  default-jdk-headless | default-jdk | java-sdk  <none>
pn  device-tree-compiler                           <none>
pn  docx2txt                                       <none>
ii  e2fsprogs                                      1.44.4-2
pn  enjarify                                       <none>
pn  fontforge-extras                               <none>
pn  fp-utils                                       <none>
pn  genisoimage                                    <none>
ii  gettext                                        0.19.8.1-8
pn  ghc                                            <none>
pn  ghostscript                                    <none>
pn  giflib-tools                                   <none>
pn  gnumeric                                       <none>
ii  gnupg                                          2.2.10-3
pn  imagemagick                                    <none>
pn  jsbeautifier                                   <none>
pn  libarchive-tools                               <none>
pn  llvm                                           <none>
pn  lz4 | liblz4-tool                              <none>
pn  mono-utils                                     <none>
pn  ocaml-nox                                      <none>
pn  odt2txt                                        <none>
pn  oggvideotools                                  <none>
ii  openssh-client                                 1:7.9p1-1
pn  pgpdump                                        <none>
pn  poppler-utils                                  <none>
pn  procyon-decompiler                             <none>
pn  python3-argcomplete                            <none>
pn  python3-binwalk                                <none>
ii  python3-debian                                 0.1.33
pn  python3-defusedxml                             <none>
pn  python3-guestfs                                <none>
pn  python3-jsondiff                               <none>
pn  python3-progressbar                            <none>
pn  python3-pypdf2                                 <none>
pn  python3-pyxattr                                <none>
pn  python3-tlsh                                   <none>
pn  r-base-core                                    <none>
pn  rpm2cpio                                       <none>
pn  sng                                            <none>
pn  sqlite3                                        <none>
pn  squashfs-tools                                 <none>
ii  tcpdump                                        4.9.2-3
ii  unzip                                          6.0-21
ii  vim-common                                     2:8.1.0320-1
pn  xmlbeans                                       <none>
ii  xxd                                            2:8.1.0320-1
ii  xz-utils                                       5.2.2-1.3

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.2.1-1

-- no debconf information


Build log:

(...)
tests/comparators/test_pcap.py::test_identification FAILED               [ 74%]
tests/comparators/test_pcap.py::test_no_differences PASSED               [ 74%]
tests/comparators/test_pcap.py::test_diff FAILED                         [ 74%]
tests/comparators/test_pcap.py::test_compare_non_existing FAILED         [ 74%]
(...)
=================================== FAILURES ===================================
_____________________________ test_identification ______________________________

pcap1 = <<class 'diffoscope.comparators.binary.FilesystemFile'> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.7/build/tests/data/test1.pcap>

    def test_identification(pcap1):
>       assert isinstance(pcap1, PcapFile)
E       AssertionError: assert False
E        +  where False = isinstance(<<class 'diffoscope.comparators.binary.FilesystemFile'> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.7/build/tests/data/test1.pcap>, PcapFile)

pcap1      = <<class 'diffoscope.comparators.binary.FilesystemFile'> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.7/build/tests/data/test1.pcap>

tests/comparators/test_pcap.py:33: AssertionError
__________________________________ test_diff ___________________________________

differences = []

    @skip_unless_tools_exist('tcpdump')
    def test_diff(differences):
        expected_diff = get_data('pcap_expected_diff')
>       assert differences[0].unified_diff == expected_diff
E       IndexError: list index out of range

differences = []
expected_diff = '@@ -1,299 +1,158 @@\n- 00:00:00.000000 IP 172.17.0.1.5353 > 224.0.0.251.5353: 0 PTR (QM)? _googlecast._tcp.local. (40...6c 2d6d 756c 7469 7363 7265  n:dial-multiscre\n \t0x0080:  656e 2d6f 7267 3a73 6572 7669 6365 3a64  en-org:service:d\n'

tests/comparators/test_pcap.py:49: IndexError
__________________________ test_compare_non_existing ___________________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fea5ecd2198>
pcap1 = <<class 'diffoscope.comparators.binary.FilesystemFile'> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.7/build/tests/data/test1.pcap>

    @skip_unless_tools_exist('tcpdump')
    def test_compare_non_existing(monkeypatch, pcap1):
>       assert_non_existing(monkeypatch, pcap1, has_null_source=False)

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fea5ecd2198>
pcap1      = <<class 'diffoscope.comparators.binary.FilesystemFile'> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.7/build/tests/data/test1.pcap>

tests/comparators/test_pcap.py:54: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fea5ecd2198>
fixture = <<class 'diffoscope.comparators.binary.FilesystemFile'> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.7/build/tests/data/test1.pcap>
has_null_source = False, has_details = True

    def assert_non_existing(monkeypatch, fixture, has_null_source=True, has_details=True):
        monkeypatch.setattr(Config(), 'new_file', True)
        assert Config().new_file, "didnt get patched"
    
        difference = fixture.compare(MissingFile('/nonexisting', fixture))
    
        assert difference.source2 == '/nonexisting'
>       assert not has_details or len(difference.details) > 0
E       AssertionError

difference = <Difference /<<PKGBUILDDIR>>/.pybuild/cpython3_3.7/build/tests/data/test1.pcap -- /nonexisting []>
fixture    = <<class 'diffoscope.comparators.binary.FilesystemFile'> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.7/build/tests/data/test1.pcap>
has_details = True
has_null_source = False
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fea5ecd2198>

tests/utils/nonexisting.py:32: AssertionError

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 912756@bugs.debian.org (full text, mbox, reply):

From: Daniel Shahaf <danielsh@apache.org>

To: Debian Bug Tracking System <912756@bugs.debian.org>

Subject: Re: diffoscope: Test failures with upcoming upstream file(1) version 5.35

Date: Sat, 3 Nov 2018 17:06:08 +0000

Control: tags -1 patch

Christoph Biedl wrote on Sat, Nov 03, 2018 at 16:13:58 +0100:
> -diffoscope-104/tests/data/test2.pcap: tcpdump capture file (little-endian) - version 2.4 (Ethernet, capture length 262144)
> +diffoscope-104/tests/data/test2.pcap: pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 262144)

Untested patch:

diff -ru diffoscope-104/diffoscope/comparators/pcap.py diffoscope-104.new/diffoscope/comparators/pcap.py
--- diffoscope-104/diffoscope/comparators/pcap.py	2018-10-15 10:09:05.000000000 +0000
+++ diffoscope-104.new/diffoscope/comparators/pcap.py	2018-11-03 17:04:03.749507655 +0000
@@ -40,7 +40,9 @@
 
 class PcapFile(File):
     DESCRIPTION = "tcpdump capture files (.pcap)"
-    FILE_TYPE_RE = re.compile(r'^tcpdump capture file\b')
+    # The regexp covers both libmagic<5.35 and libmagic>=5.35.
+    # See https://bugs.debian.org/912756
+    FILE_TYPE_RE = re.compile(r'^(tcpdump|pcap) capture file\b')
 
     def compare_details(self, other, source=None):
         return [Difference.from_command(

Cheers,

Daniel

Message #17 received at 912756@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 912756@bugs.debian.org

Cc: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>, Daniel Shahaf <danielsh@apache.org>

Subject: Re: diffoscope: Test failures with upcoming upstream file(1) version 5.35

Date: Sun, 04 Nov 2018 06:37:56 -0500

tags 912756 + pending
thanks

Thanks for this. I've tested it both sid and experimental (with
file 5.35) and it now passes in both so I've applied it in Git,
pending upload:

  https://salsa.debian.org/reproducible-builds/diffoscope/commit/0dfb8182fed6491296e1c1014351e00b29e59d8e

  diffoscope/comparators/pcap.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #24 received at 912756-close@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: 912756-close@bugs.debian.org

Subject: Bug#912756: fixed in diffoscope 105

Date: Thu, 08 Nov 2018 21:49:53 +0000

Source: diffoscope
Source-Version: 105

We believe that the bug you reported is fixed in the latest version of
diffoscope, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 912756@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Mattia Rizzolo <mattia@debian.org> (supplier of updated diffoscope package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Thu, 08 Nov 2018 22:27:23 +0100
Source: diffoscope
Binary: diffoscope
Architecture: source
Version: 105
Distribution: unstable
Urgency: medium
Maintainer: Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>
Changed-By: Mattia Rizzolo <mattia@debian.org>
Description:
 diffoscope - in-depth comparison of files, archives, and directories
Closes: 911846 912756
Changes:
 diffoscope (105) unstable; urgency=medium
 .
   [ Chris Lamb ]
   * tests:
     + Prevent test failures when running under stretch-backports by checking
       the ocaml version number.  Closes: #911846
 .
   [ Mattia Rizzolo ]
   * debian: Reinstate apktool Build-Depends and Test-Depends.
   * Fix some flake8 and deprecation warnings.
 .
   [ Daniel Shahaf ]
   * comparators/pcap:
     + Fix recognition with the upcoming file(1) 5.35.  Closes: #912756
       Thanks to Christoph Biedl for the heads-up in advance.
 .
   [ Will Thompson ]
   * Add a new command line flag --list-missing-tools.  MR: !14
Checksums-Sha1:
 3a5e81aa4031f3f2c88d80e26dfe1980c59c0851 4496 diffoscope_105.dsc
 53a9a46ee9e664944e76a3cf5081c8a8abd21209 9270120 diffoscope_105.tar.xz
 6401f8ad31db4f39d04131ea8a91c1bb6c1ee820 22160 diffoscope_105_amd64.buildinfo
Checksums-Sha256:
 c55cc108e898aaac8fdfd13709a7dbb6d701b010d8c37af1e90baa49eb8cb9ac 4496 diffoscope_105.dsc
 0dc82b8192c2ef34e3a0904fb20c0eed8f95dc795db3cbe23e3e288ea8dd362b 9270120 diffoscope_105.tar.xz
 eac552b113dbb761e3976e20e05f24c708c50691ae9d973dadcf226a4e333e22 22160 diffoscope_105_amd64.buildinfo
Files:
 b969d6c60a843a375b8a60753eb3d98a 4496 devel optional diffoscope_105.dsc
 2ad34a8ff1387b54f3e6a45fd207bff9 9270120 devel optional diffoscope_105.tar.xz
 9a26fbf254000622b6a45248e614cc90 22160 devel optional diffoscope_105_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEi3hoeGwz5cZMTQpICBa54Yx2K60FAlvkrcoACgkQCBa54Yx2
K60jsw//Qyf2WvK5XYnn9g7xc/jT1r5c3WyMHoND6HKU0V8hWiskn1Z28typcz5v
J8cTk4IUBuJfipfpWSeSxHfq6JY+n7lRMBdTl9ieeObWwGlFTu32TvoqipgzeCQ3
Dmj7B0anWgGNxWyVYG0jHlBhYOIrSnH++6qNF/+VJRPhDtyK9tJ1LbQXoIcpxPbR
7rR2fncv6e7fVMiMrD07rp5D+WquL/rUXw9ixF4a1JQ2v5nE6HMLzZ6nm6Uh79Cm
wjLVU8PjJXkbjrxSmEgE7CiNvJWJ2IYIrRx+WS6IrSfX+GD9uw8q+7rvH6H4L5/q
3cqDCjTP9dULRNipLIYZkWu0a/pOPrJ3nt/1ln+FJKu37GJaEAPYYKUAl8cyrKMm
wnnQzcJ4I6GcWyaGTyGQTPp3QSBCuKCLFNJRdgBpGESOqfzxv1A/WpsZfL5bnhe/
oiATRu4AhJq8tFIC5ggDKDMNaWrxID4RgyGDCMRjk7U6jDAq++2no/rDDnOWOUk8
AYOL9wmiDmD2MkE5R7D9B+5bZOQRMfW+5UdQyDw7jD0uQ4KRhGnpS910/TbjU1vI
zqoTIl0Fo9y9g+jbMk04sD9BLCKaAqyfhsmlt/sWbqKJ1Baxks9oHfqKtYJeAvn3
/YHIi6W7pVKqgmJQ/tuq6iQ+9uv+GyeQsl4z06BOgAsYl9y4IyY=
=zN1g
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.