Debian Bug report logs - #888112
diffoscope: compare JSON files using Python jsondiff module

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: diffoscope: compare JSON files using Python jsondiff module

Date: Tue, 23 Jan 2018 21:41:18 +0800

[Message part 1 (text/plain, inline)]

Package: diffoscope
Version: 90
Severity: wishlist

The Python jsondiff module allows comparing JSON files in a better way
than just comparing the plain text like diffoscope currently does.

The jsondiff command-line program outputs only the JSON attributes that
are different between the files, but the Python module probably allows
more advanced analyses.

If jsondiff doesn't prove useful or is hard to use, please pretty-print 
JSON files by loading the JSON file into a Python object and call
json.dump() with the indent=4 parameter and then diff the text.

$ curl -s 'https://salsa.debian.org/api/v4/users/1' > 1
$ curl -s 'https://salsa.debian.org/api/v4/users/2' > 2
$ diffoscope 1 2
--- 1
+++ 2
@@ -1 +1 @@
-{"id":1,"name":"Administrator","username":"root","state":"active","avatar_url":"https://seccdn.libravatar.org/avatar/9a5dac7fca5fa69fcb4fa36b948ff766?s=80&d=identicon","web_url":"https://salsa.debian.org/root","created_at":"2017-12-17T12:04:05.482Z","bio":"","location":"","skype":"","linkedin":"","twitter":"","website_url":"","organization":""}
+{"id":2,"name":"Paul Martin","username":"pm","state":"active","avatar_url":"https://seccdn.libravatar.org/avatar/a0177f306e580e136af48558989f2782?s=80&d=identicon","web_url":"https://salsa.debian.org/pm","created_at":"2017-12-17T12:46:16.992Z","bio":null,"location":null,"skype":"","linkedin":"","twitter":"","website_url":"","organization":null}
$ jsondiff 1 2 ; echo
{"id": 2, "name": "Paul Martin", "username": "pm", "avatar_url": "https://seccdn.libravatar.org/avatar/a0177f306e580e136af48558989f2782?s=80&d=identicon", "web_url": "https://salsa.debian.org/pm", "created_at": "2017-12-17T12:46:16.992Z", "bio": null, "location": null, "organization": null}

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 888112@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: Paul Wise <pabs@debian.org>, 888112@bugs.debian.org

Subject: Re: Bug#888112: diffoscope: compare JSON files using Python jsondiff module

Date: Tue, 23 Jan 2018 15:48:48 +0100

[Message part 1 (text/plain, inline)]

On Tue, Jan 23, 2018 at 09:41:18PM +0800, Paul Wise wrote:
> The Python jsondiff module allows comparing JSON files in a better way
> than just comparing the plain text like diffoscope currently does.

Actually, that's a problem.
Diffoscope has a json comparator, that works great, but it's failing for
your example:

> $ curl -s 'https://salsa.debian.org/api/v4/users/1' > 1
> $ curl -s 'https://salsa.debian.org/api/v4/users/2' > 2

mattia@warren /tmp % diffoscope --debug 1 2
2018-01-23 15:45:31 D: diffoscope.presenters.formats: Will generate the following formats: text
2018-01-23 15:45:31 D: diffoscope.main: Starting diffoscope 90
2018-01-23 15:45:31 D: diffoscope.locale: Normalising locale, timezone, etc.
2018-01-23 14:45:31 D: diffoscope.main: Starting comparison
2018-01-23 14:45:31 D: diffoscope.comparators: Loaded 63 comparator classes
2018-01-23 14:45:31 D: diffoscope.comparators.utils.specialize: Using TextFile for 1
2018-01-23 14:45:31 D: diffoscope.comparators.utils.specialize: Using TextFile for 2
2018-01-23 14:45:31 D: diffoscope.comparators.utils.compare: Comparing 1 (TextFile) and 2 (TextFile)
...

I.e. it's not recognizing them as json files.
If I simply rename them to 1.json and 2.json and run diffoscope on them:

mattia@warren /tmp % diffoscope --debug 1.json 2.json
2018-01-23 15:46:40 D: diffoscope.presenters.formats: Will generate the following formats: text
2018-01-23 15:46:40 D: diffoscope.main: Starting diffoscope 90
2018-01-23 15:46:40 D: diffoscope.locale: Normalising locale, timezone, etc.
2018-01-23 14:46:40 D: diffoscope.main: Starting comparison
2018-01-23 14:46:40 D: diffoscope.comparators: Loaded 63 comparator classes
2018-01-23 14:46:40 D: diffoscope.comparators.utils.specialize: Using JSONFile for 1.json
2018-01-23 14:46:40 D: diffoscope.comparators.utils.specialize: Using JSONFile for 2.json
2018-01-23 14:46:40 D: diffoscope.comparators.utils.compare: Comparing 1.json (JSONFile) and 2.json (JSONFile)
2018-01-23 14:46:40 D: diffoscope.comparators.utils.file: File.has_same_content: <<class 'abc.JSONFile'> 1.json> <<class 'abc.JSONFile'> 2.json>
2018-01-23 14:46:40 D: diffoscope.diff: Running diff -aU7 /tmp/tmpfpv5c6dm_diffoscope/fifo1 /tmp/tmpfpv5c6dm_diffoscope/fifo2
2018-01-23 14:46:40 D: diffoscope.diff: diff -aU7 /tmp/tmpfpv5c6dm_diffoscope/fifo1 /tmp/tmpfpv5c6dm_diffoscope/fifo2: returncode 1, parsed True
 |##################################################################################################################################|  100%                             Time: 0:00:00 
2018-01-23 14:46:40 D: diffoscope.presenters.formats: Generating 'text' output at '-'
--- 1.json
+++ 2.json
│   --- 1.json
├── +++ 2.json
│ @@ -1,16 +1,16 @@
│  {
│ -    "avatar_url": "https://seccdn.libravatar.org/avatar/9a5dac7fca5fa69fcb4fa36b948ff766?s=80&d=identicon",
│ -    "bio": "",
│ -    "created_at": "2017-12-17T12:04:05.482Z",
│ -    "id": 1,
│ +    "avatar_url": "https://seccdn.libravatar.org/avatar/a0177f306e580e136af48558989f2782?s=80&d=identicon",
│ +    "bio": null,
│ +    "created_at": "2017-12-17T12:46:16.992Z",
│ +    "id": 2,
│      "linkedin": "",
│ -    "location": "",
│ -    "name": "Administrator",
│ -    "organization": "",
│ +    "location": null,
│ +    "name": "Paul Martin",
│ +    "organization": null,
│      "skype": "",
│      "state": "active",
│      "twitter": "",
│ -    "username": "root",
│ -    "web_url": "https://salsa.debian.org/root",
│ +    "username": "pm",
│ +    "web_url": "https://salsa.debian.org/pm",
│      "website_url": ""
│  }
1 mattia@warren /tmp %                                                                                                                                                             :(


Which is arguably even much better than what you get from that jsondiff
command :)


FTR, it seems the only check used to decide whether a file is a json
file, is to look for a '.json' as a filename suffix (and I'm not sure we
can do much better without looking into the file, which is probably too
expensive a check).

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Message #13 received at 888112@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Mattia Rizzolo <mattia@debian.org>, 888112@bugs.debian.org, Paul Wise <pabs@debian.org>

Subject: Re: Bug#888112: diffoscope: compare JSON files using Python jsondiff module

Date: Wed, 24 Jan 2018 05:16:35 +0530

Hi Mattia & Paul,

> FTR, it seems the only check used to decide whether a file is a json
> file, is to look for a '.json' as a filename suffix (and I'm not sure we
> can do much better without looking into the file, which is probably too
> expensive a check).

I wonder how sensible it would be to, say, slurp the first 100 bytes and see
whether we start with a "{" (or some other heuristic) and, if so, try and
parse the file as JSON etc.


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #18 received at 888112@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: Chris Lamb <lamby@debian.org>, Mattia Rizzolo <mattia@debian.org>, 888112@bugs.debian.org

Subject: Re: Bug#888112: diffoscope: compare JSON files using Python jsondiff module

Date: Wed, 24 Jan 2018 08:54:45 +0800

[Message part 1 (text/plain, inline)]

On Wed, 2018-01-24 at 05:16 +0530, Chris Lamb wrote:

> I wonder how sensible it would be to, say, slurp the first 100 bytes and see
> whether we start with a "{" (or some other heuristic) and, if so, try and
> parse the file as JSON etc.

I think it would make diffoscope very slightly more useful if it could
fill in the gaps in MIME type detection for this and other cases.
If it turns out to be too expensive then it could have an option.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]

Message #23 received at 888112@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Paul Wise <pabs@debian.org>, Mattia Rizzolo <mattia@debian.org>, 888112@bugs.debian.org

Subject: Re: Bug#888112: diffoscope: compare JSON files using Python jsondiff module

Date: Wed, 24 Jan 2018 06:47:54 +0530

Hi Paul,

> > I wonder how sensible it would be to, say, slurp the first 100 bytes and see
> > whether we start with a "{" (or some other heuristic) and, if so, try and
> > parse the file as JSON etc.
> 
> I think it would make diffoscope very slightly more useful if it could
> fill in the gaps in MIME type detection for this and other cases.

https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=2a758d3d0205e934ed6dffebb5d6462b00fe590d

> If it turns out to be too expensive then it could have an option.

Mo' options, mo' problems :)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #28 received at 888112@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: Chris Lamb <lamby@debian.org>, Mattia Rizzolo <mattia@debian.org>, 888112@bugs.debian.org

Subject: Re: Bug#888112: diffoscope: compare JSON files using Python jsondiff module

Date: Wed, 24 Jan 2018 10:18:44 +0800

[Message part 1 (text/plain, inline)]

On Wed, 2018-01-24 at 06:47 +0530, Chris Lamb wrote:

> https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=2a758d3d0205e934ed6dffebb5d6462b00fe590d

That isn't going to work for UTF-8 JSON, I suggest using the MIME type
instead of the file type string.

$ curl -s 'https://salsa.debian.org/api/v4/users/1894' > user
$ file user
user: UTF-8 Unicode text, with very long lines, with no line terminators
$ file --mime-type user 
user: text/plain

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]

Message #33 received at 888112@bugs.debian.org (full text, mbox, reply):

From: Paul Wise <pabs@debian.org>

To: Chris Lamb <lamby@debian.org>, Mattia Rizzolo <mattia@debian.org>, 888112@bugs.debian.org

Subject: Re: Bug#888112: diffoscope: compare JSON files using Python jsondiff module

Date: Wed, 24 Jan 2018 10:30:11 +0800

[Message part 1 (text/plain, inline)]

On Wed, 2018-01-24 at 06:47 +0530, Chris Lamb wrote:

> https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=2a758d3d0205e934ed6dffebb5d6462b00fe590d

I would suggest also checking for '[' characters (from JSON arrays).

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]

Message #38 received at 888112@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Paul Wise <pabs@debian.org>, Mattia Rizzolo <mattia@debian.org>, 888112@bugs.debian.org

Subject: Re: Bug#888112: diffoscope: compare JSON files using Python jsondiff module

Date: Wed, 24 Jan 2018 08:15:57 +0530

Hi pabs,

> That isn't going to work for UTF-8 JSON, I suggest using the MIME type
> instead of the file type string.

  https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=f7ced27f52b974b8dac3354b3145637e3bc28f24

> I would suggest also checking for '[' characters (from JSON arrays).

  https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=de1abeae19067539c8d45758e657634762589451

:)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #43 received at 888112@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 888112@bugs.debian.org

Cc: Paul Wise <pabs@debian.org>

Subject: Re: diffoscope: compare JSON files using Python jsondiff module

Date: Thu, 25 Jan 2018 06:45:36 +0530

tags 888112 + pending
thanks

Fixed in Git:

  https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=1b121ea4efdb9fc614d32172b9749b5dd4afc550


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #48 received at 888112@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: Chris Lamb <lamby@debian.org>, 888112@bugs.debian.org

Cc: Paul Wise <pabs@debian.org>

Subject: Re: Bug#888112: diffoscope: compare JSON files using Python jsondiff module

Date: Thu, 25 Jan 2018 11:07:48 +0100

[Message part 1 (text/plain, inline)]

On Thu, Jan 25, 2018 at 06:45:36AM +0530, Chris Lamb wrote:
>   https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=1b121ea4efdb9fc614d32172b9749b5dd4afc550

So this added a new build dependency, etc.
I've built the required backports, and uploaded them to DELAYED/10 so
that they will land to stretch-backports' NEW after they migrate to
testing (as python-jsondiff was accepted into unstable for the first
time only two days ago).

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

[signature.asc (application/pgp-signature, inline)]

Message #53 received at 888112@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Mattia Rizzolo <mattia@debian.org>, 888112@bugs.debian.org

Cc: Paul Wise <pabs@debian.org>

Subject: Re: Bug#888112: diffoscope: compare JSON files using Python jsondiff module

Date: Thu, 25 Jan 2018 15:44:55 +0530

Hi Mattia,

> So this added a new build dependency, etc.
> I've built the required backports, and uploaded them to DELAYED/10 so
> that they will land to stretch-backports' NEW after they migrate to
> testing

Ooh, great thinking! Thanks :)


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #58 received at 888112-close@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: 888112-close@bugs.debian.org

Subject: Bug#888112: fixed in diffoscope 91

Date: Thu, 01 Mar 2018 13:05:09 +0000

Source: diffoscope
Source-Version: 91

We believe that the bug you reported is fixed in the latest version of
diffoscope, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 888112@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Mattia Rizzolo <mattia@debian.org> (supplier of updated diffoscope package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Thu, 01 Mar 2018 13:40:48 +0100
Source: diffoscope
Binary: diffoscope
Architecture: source
Version: 91
Distribution: unstable
Urgency: medium
Maintainer: Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>
Changed-By: Mattia Rizzolo <mattia@debian.org>
Description:
 diffoscope - in-depth comparison of files, archives, and directories
Closes: 849386 863879 863880 877726 886736 886963 887180 888112 888401 888402 890528
Changes:
 diffoscope (91) unstable; urgency=medium
 .
   [ Mattia Rizzolo ]
   * d/clean: Remove .cache/.
   * diff: Wrap long regular expression for improved readability.
   * comparators.json: Fix UnicodeDecodeError with a non-UTF8 locale.
   * d/copyright: Update for the new year.
   * d/control: Bump Standards-Version to 4.1.3, no changes needed.
   * tests/android:
     + Skip tests involving abootimg on BE archs (see #725729).
 .
   [ Daniel Shahaf ]
   * diffoscope.diff: Document feeders.  Closes: #863880
   * Optimize the common case of feeders.  Closes: #863879
 .
   [ Juliana Oliveira ]
   * {tempfiles, utils.archive}: catch possible FileNotFoundError.
   * comparators.java: add support for procyon-decompiler.  Closes: #849386
   * {command, feeders, diff}: replaces subprocess.Popen() by .run().
     This is done in preparation for work on multiprocessing (#842837).
     + {command, zip, feeders}: replaces .wait by the new .returncode property.
     + comparators.utils.command: replaces .stderr_content by .stderr.
   * tests:
     + test_progress: fix test case for some broken versions of
       python-progressbar.  Closes: #877726
     + utils.tools: add support for modules on skip_unless* annotations.
 .
   [ Chris Lamb ]
   * comparators:
     + utils/compare:
       - Show extended filesystem metadata even when directly comparing two
         files, not just when we specify two directories.  Closes: #888402
     + macho:
       - If the If the LLVM disassembler does not work, try the
         internal one.  Closes: #886736
       - Always strip the filename, not just when by itself.
     + json:
       - Do some cheap fuzzy parsing to detect JSON files not named .json.
       - Also match unicode JSON files.
       - Optionally compare JSONs with the jsondiff module.  Closes: #888112
     + directory:
       - Report differences in extended file attributes when comparing files.
         Closes: #888401
     + xsb:
       - Add support for comparing XMLBeans binary schemas.
     + barkeley_db:
       - Add support for comparing Berkeley DB files.  Closes: #890528
   * Misc code cleaup.
   * tests:
     + comparators.test_elf: Return '0' if we can't parse the readelf
       version number.  Closes: #886963
   * debian:
     + Explicitly build-depend and recommend e2fsprogs.  Closes: #887180
 .
   [ Ximin Luo ]
   * Partially revert the changes done for #888402 to maintain the current.
     behaviour of --exclude-directory-metadata.
   * Refactor how the configuration is loaded:
     + Move the defaults into a Config.reset() method.
     + reset() the configuration at the end of main(), to help the testsuite.
Checksums-Sha1:
 cf78d47c07e0cea4b0d667c02a0beff82b2cb28b 3429 diffoscope_91.dsc
 f10a233ff4469d6405eac3bc6c03da6d33962efb 8935512 diffoscope_91.tar.xz
 1413267eea08eb05b0664a4ba13b46abc50a9df7 20151 diffoscope_91_amd64.buildinfo
Checksums-Sha256:
 28e12196a5e461349d1cfdfa96bb3914337f6bd6327d13637de80a5bedd08169 3429 diffoscope_91.dsc
 12034827db747b831b0c80a340f607c8ac4b85936610d38985135941b8a4f0ee 8935512 diffoscope_91.tar.xz
 a456c23241bed0e2dc6230a03e3f7897ad9a715f54da1e662b4f7864c0804d5f 20151 diffoscope_91_amd64.buildinfo
Files:
 4a1c8bd026d379c91fec4db88eee3679 3429 devel optional diffoscope_91.dsc
 a72769fc6dc5f80c6a31b8730234444a 8935512 devel optional diffoscope_91.tar.xz
 728e5ffa6fff19b50c3baf3dfe0a2ee6 20151 devel optional diffoscope_91_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEi3hoeGwz5cZMTQpICBa54Yx2K60FAlqX9qYACgkQCBa54Yx2
K62jGg/+J2JLNEmq/nVoNa182wyzTzerZVy03V5Pc4WBt1pyZ4DPbvDrAhZvWZmP
qBXr54UxkbbNfXa+fYAph4xr7D5xL4olDEOFRcDf5UJfGwdjaravc6CeE1z915eW
IJkFIOd17ySKlxM5RAVQ1xtQqXiCsBBHa4W2XryleNivJOSOKotEbkEul8MrregE
HREtWF76l7t5Mas2bFxD0xjYAfIYgB8vDLCFnsoOHsqUKksf2PYr60ge2CmCLKUO
Lw/gTcXD+T8k6UBoC2HXmU/2Wgsd9KClAOHUWwdJK78akvvjSC+zBfcewW1vtM1P
lVui0SjSmd7K3suNZgNnrCboKpq7J8ULR8g1GikElxV6orkDsmhX5SfqgsiqWsqw
yZbm1VHRBtOawnQUD8U2bLqVZsTOsE5JMF7ehHjGehEgoCgg3oNTbY8cIAWyd8y/
erk43kNYKhYdHbEfpvGRJekSvJOS8BY4h94ThJ1B4dElUhMzESJ5Fa2S/Ik3fyDS
WKWY8rS3WSons4WL+yjvd7Y2hfuf/IWt6XaG2rge3ODJlwD3BxFrLg7NzkNhH7NJ
a1yITYx2JbMahQZQribpkdPiFcx38y0LcAT25br8AY+asa0fr7qPOfshQfytFiIs
bDfMRuWM5I6KsCwvE4Gtjrawrux03EdbB0piGQzWKpfAmO3qqdE=
=L47V
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.