Debian Bug report logs - #715035
lintian: Possible memory optimizations

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Niels Thykier <niels@thykier.net>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: lintian: Possible memory optimizations

Date: Fri, 05 Jul 2013 18:38:39 +0200

[Message part 1 (text/plain, inline)]

Package: lintian
Version: 2.5.13
Severity: minor

Hi,

I have spent a little time looking at the memory consumption of
Lintian.  I have already optimized a few trivial cases, but I have
also seen futher potential that may be worth deploying.

Method: I have only looked at the memory consumed by L::Collect at the
end of the run (i.e. the memory data you get with -dddd from the
master branch).  I suspect there is also a lot to be gained from not
slurping copyright files (etc.) without any regards to their size in
checks.

Observations so far:
 - (sorted_)index is(/are) the primary memory consumer(s).
   - fortunately, they share a large part of the memory so the output
     of "-dddd" looks worse than it is.
   - AFAICT I can tell, memory seems to be "leaked" by Perl allocating
     large buffers for the strings[1].  With my test case source:linux,
     I suspect we can save about 5MB if we can reduce the size of
     these buffers to problem.
 - changelog is also pretty expensive and could probably very often be
   shared between different L::Collect instances related to the same
   processable group.  For lintian, it is about 2Mb for both the .deb
   and the source.  The common "non-sharing" case would be binNMUs or
   a bug in the package[2].
   - If we can dedup this on a disk level, we are also recovering
     disk storage.  I haven't checked if this is worth it even for
     lintian.d.o.  The savings will be a lot less on the disk though,
     since Perl/Parse::DebianChangelog spends more memory on it than
     its actual size (about factor 4 for lintian's changelog)[3].

I have attached the memory usage information of running lintian with
-dddd (-X man) on some of the linux binaries and their source package,
which can be used as reference.

~Niels

[1] This makes sense if the strings are to be changed later, but
generally this strings will only be read.

[2] E.g. installing the wrong changelog in one of the binary
packages. :)

[3] If we are doing this disk de-duplication, we can probably trivally
apply it to copyright files as well.  But back to memory ...

[memory-usage-linux-Xman.log (text/plain, attachment)]

Message #12 received at 715035-close@bugs.debian.org (full text, mbox, reply):

From: Niels Thykier <niels@thykier.net>

To: 715035-close@bugs.debian.org

Subject: Bug#715035: fixed in lintian 2.5.34

Date: Wed, 22 Jul 2015 21:37:43 +0000

Source: lintian
Source-Version: 2.5.34

We believe that the bug you reported is fixed in the latest version of
lintian, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 715035@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Niels Thykier <niels@thykier.net> (supplier of updated lintian package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Wed, 22 Jul 2015 21:55:58 +0200
Source: lintian
Binary: lintian
Architecture: source all
Version: 2.5.34
Distribution: unstable
Urgency: medium
Maintainer: Debian Lintian Maintainers <lintian-maint@debian.org>
Changed-By: Niels Thykier <niels@thykier.net>
Description:
 lintian    - Debian package checker
Closes: 715035 792167 792237 792501 793068
Changes:
 lintian (2.5.34) unstable; urgency=medium
 .
   * Summary of tag changes:
     + Added:
       - unstripped-static-library
 .
   * checks/binaries.{desc,pm}:
     + [NT] Apply patch from Guillem Jover to improve the tag
       description of the LFS tag.  (Closes: #792167)
     + [NT] Detect unstripped static libraries (as an
       experimental tag).
   * checks/cruft.pm:
     + [NT] Avoid svn-conflict-file tag for ".r0" files, since
       there cannot be a conflict involving revision 0.
       Thanks to Andreas Beckmann for the report.
       (Closes: #793068)
     + [BR] Detect pdf, ps, and eps lenna file.
   * checks/files.{desc,pm}:
     + [BR] Fix detection of sources.list file.  (Closes: #792237)
   * checks/systemd.pm:
     + [NT] Apply patches from Felipe Sateler to ignore known
       masked init.d scripts.  (Closes: #792501)
 .
   * commands/info -> info.pm:
     + [NT] Renamed info to info.pm to remove some boilerplate
       code.
     + [NT] Forbid late --user-dirs and --include-dir options.
 .
   * frontend/lintian:
     + [NT] Forbid late --user-dirs and --include-dir options.
     + [NT] Remove the following obsolete command line
       options: --root, --all, --binary, --source, --udeb.
       They are no longer useful.
 .
   * lib/Lintian/Collect{,/*}.pm:
     + [NT] Add a shared memory cache that can be used to
       reduce memory comsumption a bit for sharable objects.
       (Closes: #715035)
   * lib/Lintian/Data.pm:
     + [NT] Remove references to LINTIAN_ROOT as Lintian::Data
       no longer uses it.
   * lib/Lintian/Path.pm:
     + [NT] Rework some implementation details to reduce memory
       consumption slightly.
     + [NT] The undocumented "type" and "time" methods have been
       removed from the API.
     + [NT] The "uid" and "gid" methods now return 0 rather than
       undef when numeric ownership is not collected.
 .
   * reporting/templates:
     + [NT] Remove (uses of) the "invisible-anchor" css class as
       invisible (display: none) elements cannot be used as
       anchors.
Checksums-Sha1:
 491fcb7d7bbd7f06e3e3979080fce97c24d770eb 2731 lintian_2.5.34.dsc
 b09752870cd488407f2e89ad3b50586cf8e8bd78 1220984 lintian_2.5.34.tar.xz
 896995e5227c750702ee96075ba36a3548211040 819008 lintian_2.5.34_all.deb
Checksums-Sha256:
 f5aee181b48017eefe5382b272eefa840cbbba617133ae8053b48d250c8e8ed7 2731 lintian_2.5.34.dsc
 590d7c8c4457e4c76e23daafdd5e518d00d56aab5efb6220e818f764462e2678 1220984 lintian_2.5.34.tar.xz
 40712986199ffc4a9e3d493cbfa6d00f74195bbee762c231191aaf46d2c10290 819008 lintian_2.5.34_all.deb
Files:
 b964bbcab97bb435b6b36263f34bc3c8 2731 devel optional lintian_2.5.34.dsc
 75301e71130477d2eb6b9caf8ab56a4b 1220984 devel optional lintian_2.5.34.tar.xz
 a1deabae89110bc2c31987cbcce44c42 819008 devel optional lintian_2.5.34_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJVr/m2AAoJEAVLu599gGRCPbkQAKS4vvGSsSs5cRajjrSugQyo
Eqf1088LhFW91fulZCZA04CHFZB6YQrsEMk/119sueaf76STbUovifF9Vx/TOgr1
8C+Ohub84Q9QGyGZP2uk8HyPwaSokoWdHkTTBhQNmbSKE/l3S1JEjd5pH3yOjqMg
+hMTjfWQySNQcnRD5FXZr5HGLNZY6TomCOWcQSZZrhFuCY85m8Z33VT5f5anKfx9
1FmHQQpSS75nv0aH52FukZQYrbiUpJi1LEK1wCD+0dVveCaDeQRGqm1mBT/BTGVJ
hkiaFBd2e3PtCZ7AKiuumn/eQjnw5iSQAtI5/Dvl7K1B/cb6v1HlHTsEnY0xg+JK
l0NdX3dfS6myT/L1mvJVupG6qJ6pzBvCkvtTVuSJiyQHYiIe9l9/ptumpnughzZj
rDqBeErAGE9iERE862zzwtIDxVaf6Fs7TSdOKz778QeNY0sTC1r5dRu/Y2Wxx1Bw
6ky/0VB7RQc03HcsajxNaAnalDl9c4mvpgRvlnWP2NqTwgtNT/yhtjeUHidUH3m6
mdRcONUtQ30BXrvPv5s+yMyE3Bj+l0APqlSdA5Kr9cgo6K5ykoBJF28c7SBdmyZU
F6ImaaeRSHWg4IGjNOZHCt/Z68/gnQXI5l1lnym8nnBkuffcBMpVxK1A/vLtMOLr
YkKF4sUL7jsKwuiMe+Ea
=IsKl
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.