Debian Bug report logs - #865660
diffoscope uses 35GB ram for comparing influxdb_1.1.1+dfsg1-4_arm64.changes

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Helmut Grohne <helmut@subdivi.de>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: diffoscope uses 35GB ram for comparing influxdb_1.1.1+dfsg1-4_arm64.changes

Date: Fri, 23 Jun 2017 16:42:53 +0200

Package: diffoscope

I noticed that a diffoscope running on jenkins.debian.net would consume
79G of virtual memory and 35G of resident memory while comparing
influxdb_1.1.1+dfsg1-4_arm64.changes after about 25 cpu minutes. Given
that the largest package from influxdb is ~10M, we're seeing a 3000 times
memory use. That's bad.

Unfortunately, I cannot provide any more details at this point. Sorry.
If this turns out to be undebuggable, please just close the report.

Helmut

Message #10 received at 865660@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: Helmut Grohne <helmut@subdivi.de>, 865660@bugs.debian.org

Subject: Re: Bug#865660: diffoscope uses 35GB ram for comparing influxdb_1.1.1+dfsg1-4_arm64.changes

Date: Fri, 23 Jun 2017 15:50:54 +0100

Helmut Grohne wrote:

> Unfortunately, I cannot provide any more details at this point. Sorry.

Can you at least provide the version of diffoscope you are using?


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb, Debian Project Leader
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #19 received at 865660@bugs.debian.org (full text, mbox, reply):

From: Helmut Grohne <helmut@subdivi.de>

To: Chris Lamb <lamby@debian.org>

Cc: 865660@bugs.debian.org

Subject: Re: Bug#865660: diffoscope uses 35GB ram for comparing influxdb_1.1.1+dfsg1-4_arm64.changes

Date: Fri, 23 Jun 2017 18:11:13 +0200

Hi Chris,

On Fri, Jun 23, 2017 at 03:50:54PM +0100, Chris Lamb wrote:
> > Unfortunately, I cannot provide any more details at this point. Sorry.
> 
> Can you at least provide the version of diffoscope you are using?

It wasn't me. It was jenkins.debian.net. So I actually tried looking up
the version, but I was too late and the process finished.

As far as I can tell, the reproducible chroots always use the latest
diffoscope. Most likely they were using 83.

Hope that helps. If it doesn't, close it.

Helmut

Message #24 received at 865660-close@bugs.debian.org (full text, mbox, reply):

From: Mattia Rizzolo <mattia@debian.org>

To: 865660-close@bugs.debian.org

Subject: Bug#865660: fixed in diffoscope 84

Date: Wed, 12 Jul 2017 11:04:43 +0000

Source: diffoscope
Source-Version: 84

We believe that the bug you reported is fixed in the latest version of
diffoscope, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 865660@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Mattia Rizzolo <mattia@debian.org> (supplier of updated diffoscope package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Wed, 12 Jul 2017 12:28:11 +0200
Source: diffoscope
Binary: diffoscope
Architecture: source
Version: 84
Distribution: unstable
Urgency: medium
Maintainer: Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>
Changed-By: Mattia Rizzolo <mattia@debian.org>
Description:
 diffoscope - in-depth comparison of files, archives, and directories
Closes: 865625 865660 866241 866577
Changes:
 diffoscope (84) unstable; urgency=medium
 .
   [ Ximin Luo ]
   * comparators:
     + directory:
       - raise warning for getfacl and remove a redundant try-clause.
       - add a --exclude-directory-metadata option.  Closes: #866241
   * diff:
     + Restore artificial limit when calculating linediff, to prevent memory
       blow up.  Closes: #865660
   * presenters:
     + text:
       - fix recursive header display in the text output, deeply-nested headers
         were not getting indented far enough.
     + add a PartialString class.
     + html:
       - in html-dir mode, put css/icon in separate files to avoid duplication.
       - split index pages up if they get too big.
       - add a size-hint to the diff headers and lazy-load buttons.
       - diffcontrol UI tweaks:
         * don't toggle buttons for diffs that haven't been loaded yet;
         * make the diffcontrol headers clickable.
       - more intuitive "limit" flags, with some backwards-incompatible changes:
         --max-report-size:
           Old: in --html-dir this limited only the parent page
           New: in --html-dir this applies across all pages
         --max-diff-block-lines:
           Old: in --html-dir 4 * this number applied across all pages
           New: in --html-dir this applies across all pages
         --max-page-size:
           New flag
           Applies to the sole --html page, or the top-level --html-dir page
         --max-report-child-size
           Renamed to
         --max-page-size-child:
           No behavioural changes
         --max-diff-block-lines-parent
           Renamed to
         --max-page-diff-block-lines:
           Old: Only applied to the top-level --html-dir page
           New: Applies to the sole --html page or the top-level --html-dir page
   * main:
     + temporarily add old flags back for backwards-compatibility.
     + warn if loading a diff but also giving diff-calculation flags.
     + implement proper boolean flags to fix test failure in previous commit.
   * tests:
     + handle existing but not importable modules.
       This fixes the tests during the Python 3.6 transition, where some
       modules (like rpm) exists but are not yet rebuilt for 3.6.
     + fix progressbar failure that was exposed by the previous importing fixes.
     + presenters:
       - skip visual test if tools aren't installed.  Closes: #865625
     + comparators
       - sqlite: test for sqlite 3.19.
       - fsimage: move the guestfs cache out of the default /var/tmp/ into a
         "more temporary" directory.
 .
   [ Mattia Rizzolo ]
   * debian/control: fix Vcs-Git field.
 .
   [ Chris Lamb ]
   * debian/control: bump Standards-Version to 4.0.0.
 .
   [ Daniel Shahaf ]
   * Fix markup in the man page synopsis.  Closes: #866577
    Thanks to Niels Thykier for the report.
Checksums-Sha1:
 d00b19b78bb0bc8a1d3ce27d95dc55e30f5020c6 3201 diffoscope_84.dsc
 8939f2cdeecf5ef16d4f2e181f737dd8dd285e95 650092 diffoscope_84.tar.xz
 0649579c84d961816f5f6ef81006be9425de3af2 18256 diffoscope_84_amd64.buildinfo
Checksums-Sha256:
 5c6ca47c6b6b8e8bdc208755ca3bc619cc4ef16c0ef662293658f00ce5829bab 3201 diffoscope_84.dsc
 a030a76c5fa472b4f28919c2d1a835a9d5a8c3fa45a5a765f03e00281c203e4e 650092 diffoscope_84.tar.xz
 8fdb954084d5adea838b3f7850e3837330f37fe59d679cb75a8d7de7b8827336 18256 diffoscope_84_amd64.buildinfo
Files:
 7a66858cdd8e4e7cb6c35a14a83352d2 3201 devel optional diffoscope_84.dsc
 b2580eff6f41294e4b1b9d8666aac28a 650092 devel optional diffoscope_84.tar.xz
 0c04128bb0cb2aed072a77abba202bc9 18256 devel optional diffoscope_84_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEi3hoeGwz5cZMTQpICBa54Yx2K60FAlll/q0ACgkQCBa54Yx2
K62odA/6Aw+RPI+bT5WCy05wHQPdU+1rGRJoKihJxh76xvKsK+dS+6aH2IKNoDfl
ICPqGkZ0mobJGj8BAvU18EjxPkZIa2jKSJnQIXo3xerBQy3dCWLoD5E5k9jqyPj8
Rvqa4fsH6QoNI2leUm+T9L+RVkCz7sG+AoRSg/fMHGcVNoRN1CfRopoxZmkn8kOY
1I/jdk0Uv3DqVUXPfK7gnhUnXdDZey2ZQGZn6CtSCdVBwRNSxdjeqMvc4nsXqGOX
J2Aa9DcQg5ZBeEDCSYVLXrlCtT4Tz0i94TE9tBzthWgPpTZZ7JiNIBOYEScsKcar
4Fe5oO96tftpEPCqyPkU33GlMxrsJK2VNq2pOdSe+pNy6r5V3plRmf3PU8V9lU1Y
dgDyTHbr3iCq+KwMnmM1vrmGQhZt98vkskd9usNHxjEFca1r2M+jN40Jke6QzE9H
zEiYizBNuNUXuJt/ObbqAA9Uw7w5dErVXH49QaHYExEBWvabm8iIjI+WN93OJ1q/
3h++qOUcTnxUuwluGLIAassqm669ad5AM0VtErqtCrZBn2D5Q29fdSuYuydqSIxh
3gh2RLmWvuDjRpMr2dpbhZ1U7NEyq8WBujOnpf/VvoTcn1ceZglJbD+g25c2csTe
sjXrTh5MPM8sHOwpHYjyuZcKCHzPpdzueaGkWe5hQb7n/NoLzBo=
=0j2Y
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.