Debian Bug report logs - #820631
diffoscope: does not see difference when comparing concatenated cpio archives (eg. initramfs images)

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>

To: submit@bugs.debian.org

Date: Sun, 10 Apr 2016 20:01:44 +0000

Subject: diffoscope: wrongly reports "No differences found inside, yet data differs"
Package: diffoscope
Version: 51
Severity: normal

Dear Maintainer,

when comparing two initramfs images [1,2], diffoscope reports:
[1] http://in.waw.pl/~zbyszek/diffoscope/initramfs-4.5.0-302.fc24.x86_64.img
[2] http://in.waw.pl/~zbyszek/diffoscope/initrd

$ diffoscope initrd initramfs-4.5.0-302.fc24.x86_64.img
--- initrd
+++ initramfs-4.5.0-302.fc24.x86_64.img
│┄ No differences found inside, yet data differs
│ @@ -99993,8 +99993,8 @@
│  00186980: 9f3b 9832 2c2d 928f 22a1 0518 4c44 9e04  .;.2,-.."...LD..
│  00186990: 6e15 ca38 8218 fbd3 f8ed 1150 021f 3d72  n..8.......P..=r
│  001869a0: 8043 a230 7d49 a55f a840 40d7 9242 4da1  .C.0}I._.@@..BM.
│  001869b0: dc08 8470 3e05 bdcc a6c4 980d d084 cf2e  ...p>...........
│  001869c0: e951 5086 f574 94c6 74c6 b2c8 5ffa 31d2  .QP..t..t..._.1.
│  001869d0: 941e 42e6 81a7 a945 aae3 e639 ac96 a81e  ..B....E...9....
│  001869e0: 8335 0705 238e cc20 11a0 219b 4300 d00a  .5..#.. ..!.C...
│ -[ Too much input for diff (SHA1: 3df03b52054657ed2022f06be8d8e0eedac8a4fa) ]
│ +[ Too much input for diff (SHA1: e691a25646c1427ec8471a0e73584399ac69e929) ]

but in fact there's a difference and it can be found manually:
$ cat initramfs-4.5.0-302.fc24.x86_64.img| (mkdir a; cd a; cpio -i; zcat|cpio -i)
$ cat initrd | (mkdir b; cd b; cpio -i; zcat|cpio -i)
$ diff -u -r a b
diff -u -r a/lib/dracut/build-parameter.txt b/lib/dracut/build-parameter.txt
--- a/lib/dracut/build-parameter.txt    2016-04-10 15:52:12.243191083 -0400
+++ b/lib/dracut/build-parameter.txt    2016-04-10 15:52:22.350662307 -0400
@@ -1 +1 @@
--f
+
diff -u -r a/usr/lib/dracut/build-parameter.txt b/usr/lib/dracut/build-parameter.txt
--- a/usr/lib/dracut/build-parameter.txt        2016-04-10 15:52:12.243191083 -0400
+++ b/usr/lib/dracut/build-parameter.txt        2016-04-10 15:52:22.350662307 -0400
@@ -1 +1 @@
--f
+

-- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'stable'), (500, 'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.3.3-303.fc23.x86_64 (SMP w/12 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  python3-libarchive-c   2.1-3
ii  python3-magic          1:5.25-2
ii  python3-pkg-resources  20.3.1-1
pn  python3:any            <none>

Versions of packages diffoscope recommends:
ii  acl                        2.2.52-3
ii  binutils-multiarch         2.26-8
ii  bzip2                      1.0.6-8
ii  caca-utils                 0.99.beta19-2+b1
ii  colord                     1.2.12-1
ii  cpio                       2.11+dfsg-5
ii  default-jdk [java-sdk]     2:1.8-57
ii  enjarify                   20151118-1
ii  fontforge-extras           0.3-4
ii  fp-utils                   3.0.0+dfsg-4
ii  fp-utils-3.0.0 [fp-utils]  3.0.0+dfsg-4
ii  genisoimage                9:1.1.11-3
ii  gettext                    0.19.7-2
ii  ghc                        7.10.3-7
ii  ghostscript                9.19~dfsg-1+b1
ii  gnupg                      1.4.20-5
ii  mono-utils                 4.2.1.102+dfsg2-6
ii  openjdk-8-jdk [java-sdk]   8u77-b03-3+b1
ii  pdftk                      2.02-3
ii  poppler-utils              0.38.0-2+b1
ii  python3-debian             0.1.27
ii  python3-guestfs            1:1.32.2-4+b1
ii  python3-rpm                4.12.0.1+dfsg1-3+b2
ii  python3-tlsh               3.4.4+20151206-1+b1
ii  rpm2cpio                   4.12.0.1+dfsg1-3+b2
ii  sng                        1.1.0-1+b1
ii  sqlite3                    3.12.1-1
ii  squashfs-tools             1:4.3-3
ii  unzip                      6.0-20
ii  vim-common                 2:7.4.1689-3
ii  xz-utils                   5.1.1alpha+20120614-2.1

Versions of packages diffoscope suggests:
ii  libjs-jquery  1.12.3-1

-- no debconf information

Message #10 received at 820631@bugs.debian.org (full text, mbox, reply):

From: Reiner Herrmann <reiner@reiner-h.de>

To: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>, 820631@bugs.debian.org, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: Re: [Reproducible-builds] Bug#820631: (no subject)

Date: Sun, 10 Apr 2016 22:34:59 +0200

[Message part 1 (text/plain, inline)]

Hi Zbigniew,

thanks for the report!
I just had a short look at this issue.
The problem why diffoscope doesn't see this change is that the initramfs
contains actually multiple cpio archives, but diffoscope treats
the files as single cpio archives:

> $ cpio -t < initrd 
> .
> early_cpio
> kernel
> kernel/x86
> kernel/x86/microcode
> kernel/x86/microcode/GenuineIntel.bin

And the files in this first/visible archive don't differ.

Right now it doesn't know that somewhere in the original file an additional
archive is embedded.
And so it falls back to a hexdump/binary diff, as the files clearly
differ, but it didn't find any more readable difference.

To solve this, diffoscope either needs to learn about initramfs files,
or (to have a more general solution) use binwalk for extracting data
embedded into other files (this would require #819018 first).

Regards,
  Reiner

[signature.asc (application/pgp-signature, inline)]

Message #15 received at 820631@bugs.debian.org (full text, mbox, reply):

From: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>

To: Reiner Herrmann <reiner@reiner-h.de>

Cc: 820631@bugs.debian.org, Reproducible Builds discussion list <reproducible-builds@lists.alioth.debian.org>

Subject: Re: [Reproducible-builds] Bug#820631: support for concatenated cpio archives

Date: Sun, 10 Apr 2016 21:11:49 +0000

On Sun, Apr 10, 2016 at 10:34:59PM +0200, Reiner Herrmann wrote:
> Hi Zbigniew,
> 
> thanks for the report!
> I just had a short look at this issue.
> The problem why diffoscope doesn't see this change is that the initramfs
> contains actually multiple cpio archives, but diffoscope treats
> the files as single cpio archives:
> 
> > $ cpio -t < initrd 
> > .
> > early_cpio
> > kernel
> > kernel/x86
> > kernel/x86/microcode
> > kernel/x86/microcode/GenuineIntel.bin
> 
> And the files in this first/visible archive don't differ.
> 
> Right now it doesn't know that somewhere in the original file an additional
> archive is embedded.
> And so it falls back to a hexdump/binary diff, as the files clearly
> differ, but it didn't find any more readable difference.
> 
> To solve this, diffoscope either needs to learn about initramfs files,
> or (to have a more general solution) use binwalk for extracting data
> embedded into other files (this would require #819018 first).

Thanks for the quick diagnosis.
I think that using binwalk as a general solution sounds better. 
I'll wait then for diffoscope to acquire that capability.
In Fedora we're lucky, binwalk already provides python3 modules (in fact
only python3 modules, without python2 support).

Zbyszek

Message #22 received at 820631@bugs.debian.org (full text, mbox, reply):

From: Gianfranco Costamagna <locutusofborg@debian.org>

To: "zbyszek@in.waw.pl" <zbyszek@in.waw.pl>

Cc: "reiner@reiner-h.de" <reiner@reiner-h.de>, "820631@bugs.debian.org" <820631@bugs.debian.org>, "reproducible-builds@lists.alioth.debian.org" <reproducible-builds@lists.alioth.debian.org>

Subject: Re: [Reproducible-builds] Bug#820631: support for concatenated cpio archives

Date: Tue, 31 May 2016 10:49:44 +0000 (UTC)

Hi all,

>Thanks for the quick diagnosis.
>I think that using binwalk as a general solution sounds better. 
>I'll wait then for diffoscope to acquire that capability.
>In Fedora we're lucky, binwalk already provides python3 modules (in fact
>only python3 modules, without python2 support).

Now I understand why some days later somebody (Reiner) asked me to enable the python3
package :)


Two things, the package is called binwalk, but provides a python3 binding.
I renamed it to python3-binwalk, just to be sure people are aware of the bindings
(I still provide the old binwalk binary package, even if now it is empty).

It will go in binary-new queue, so please wait for it to be accepted!

BTW I tried to install binwalk from unstable/testing and I did a:


$ python3 -c "import binwalk"

successfully.
(so even without waiting for the new python3-binwalk, you can already test your code
locally)

G.

Message #29 received at 820631@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 820631@bugs.debian.org

Cc: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>

Subject: Re: diffoscope: does not see difference when comparing concatenated cpio archives (eg. initramfs images)

Date: Sat, 09 Sep 2017 11:17:25 +0100

tags 820631 + pending
thanks

Fixed in Git:

  https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=1c26813019c09bf4b2e82da9668564cec58b504e


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #36 received at 820631-close@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: 820631-close@bugs.debian.org

Subject: Bug#820631: fixed in diffoscope 87

Date: Fri, 22 Sep 2017 18:34:49 +0000

Source: diffoscope
Source-Version: 87

We believe that the bug you reported is fixed in the latest version of
diffoscope, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 820631@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Mattia Rizzolo <mattia@debian.org> (supplier of updated diffoscope package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Fri, 22 Sep 2017 18:53:05 +0200
Source: diffoscope
Binary: diffoscope
Architecture: source
Version: 87
Distribution: unstable
Urgency: medium
Maintainer: Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>
Changed-By: Mattia Rizzolo <mattia@debian.org>
Description:
 diffoscope - in-depth comparison of files, archives, and directories
Closes: 820631 875281 875282 875451
Changes:
 diffoscope (87) unstable; urgency=medium
 .
   [ Juliana Oliveira Rodrigues ]
   * comparators:
     + utils:
       - compare: fix comparison between container types.
       - container: add get_path_name() method, aiming at preventing hitting
         OS' max filename length.
     + gzip: replace dest_path generation with self.get_path_name().
     + bzip2: replace dest_path generation with self.get_path_name().
   * tests:
     + presenters:
       - skip html_visuals test if 'sng' binary is not available.
     + comparators:
       - test_cointainers: new tests to test cross-container comparisons.
       - add test_containers.py.
 .
   [ Chris Lamb ]
   * comparators:
     + utils:
       - file: correct reference to path_apparent_size.
     + fonts: loosen matching of file(1)'s output to ensure we correctly also
       match TTF files under file 5.32.
     + binwalk:
       - new comparator to deal with (e.g.) concatenated CPIO archives using
         the new optional python module 'binwalk'.  Closes: #820631
   * diffoscope.diff: Correct reference to self.buf.
   * debian/rules: run flake8 during the tests, and fail the build in case the
     "underfined name" flake8's check fails.
   * tests/comparators:
     + test_cpio: check we identify all CPIO fixtures.
   * main:
     + print a debugging message if we are reading diff from stdin.
   * presenters:
     + html: use logging.py's lazy argument interpolation.
   * debian/control: bump Standards-Version to 4.1.0.
   * Code style improvements across all of the codebase.
 .
   [ Mattia Rizzolo ]
   * debian/changelog: retroactively close a bug in an old changelog entry.
   * setup.py: add a check to prevent people from installing diffoscope under
     python < 3.5 (i.e. python2) and give them a usable error message otherwise.
   * Code style improvements across all of the codebase.
 .
   [ Ximin Luo ]
   * comparators:
     +  add a fallback_recognizes() method to work around file(1)'s bug #876316.
        Closes: #875282
     + add a --force-details flag for debugging.
     + deb:
       - if --force-details then don't skip files with identical md5sums either.
   * presenters:
     + html:
       - prune all descendants properly.  Closes: #875281
       - don't show pointer-cursor when jquery is disabled.
   * config:
     + force-set a value if it must be less than another, and it was not set on
       purpose.  Closes: #875451
   * readers: convert bytes to str in the right place.
   * diff: use diff_split_lines everywhere.
   * difference:
     + in fmap/map_lines, don't forget about self._visuals.
     + also copy self._comment properly, compare self._visuals in equals().
   * tests/comparators:
     + test_deb: make test_md5sums less brittle.
Checksums-Sha1:
 c7ed45c20e00e3f895e97fb3653019fe7b8e30ba 3275 diffoscope_87.dsc
 a029a0296360c859d696f57ea0c6b0823ef52e11 679400 diffoscope_87.tar.xz
 2c1254ab760cefaa0d4575892a01db4d0b2da7e9 18695 diffoscope_87_amd64.buildinfo
Checksums-Sha256:
 c2d22689f48e07e310cf0c87710fce0f425c7c60f9f8f69af00605123381c8d0 3275 diffoscope_87.dsc
 1ab85e3bb7d16e3ef555ec622643cef5281189c5865d3268d573e446f586c001 679400 diffoscope_87.tar.xz
 aefa027c21243bdaae69ec3c0b4a4f05625683e45893df689275e2327106d39b 18695 diffoscope_87_amd64.buildinfo
Files:
 106da236bbc63e5f3bab33d690a761d7 3275 devel optional diffoscope_87.dsc
 3e3f080bb1e74378b762ddef61f2423a 679400 devel optional diffoscope_87.tar.xz
 44b0b6a532363ffaf89ce26bcb5c6183 18695 devel optional diffoscope_87_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEi3hoeGwz5cZMTQpICBa54Yx2K60FAlnFUrkACgkQCBa54Yx2
K61hsA/+MoGJGh2WY3O8bn/QrOQ4bTCdd/jPve6aNy3A0c/4obO/Kw8QXPofQdyG
P+pKGwH1ta6vjYX9r62rQ3snkbMoIC9Emx4uRaRK1VJXWYJvjpMX77xdAIdv4cxx
2P5eAA+4lBkL4GKgjRirLQnqyAJy/iLPjwadW4PlQrW7cbqG12P276VPoljzsseM
RtUU2z+MWDYlorOWUAedU6UdOBLkMzQPafHKfZdHN14z70cpcdHDLZ6rh0fOLsjg
tQFWg/zbdFFsA/kz2JPCRAWznCZ/iCufMHGwZSU6EagCWtZ6ZCkxN0/AKrSZA7RS
1et4J2KAVMUmlkyqXoVc9km5oK6Z3UIYF4ZVI7kASK26snXuiJMe7bdkBy31+lxi
lak+6K/GaLbXqN5FpWmZ9kj4As4+KVb0F4KuS9ulyp4rj5ueRB2wZiJTZoNQz7qF
MlvYRrnDAUk+OoWzUZZ0BL9aWUt5lCc1kRMnFkLLrP+cJ1UuYzyxYatj8IuLg47R
3zmOHBLXYbruiNrWJ2cxt6G0UDxWitLpGZn864ycAPvP4PeIswsB3CSjcQJm1deh
IziKItn497eJcEzDD/2P53KO0lIrVKMImgsdEWC/lPXig+QWfN3hRyZX4wRnD804
Mt6H1nTM8MAHS0y2F/EHSIJrXLS6d/yvqXSQ9XDwnwWcHZH54B0=
=YoCq
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.